I think it's fine actually. If the engine allowed it, you would push ladders instead of breaking them.
To make sieges more fun, I think the whole gamemode should be revamped to make a better use of siege equipment.
A way longer duration to encourage the effective use of catapults, towers and tents, smaller defender respawn timer to balance it up and centered spawns for defenders near the flag (not spawning all over the castle and dying immediatly because you are surrrounded by attackers).