If they are put in again, they must break if less inclined than 30 degrees or so, and maybe they should have a maximum load (let's say three men for a regular ladder), that also breaks the ladder.
This would probably fix the two biggest problems with ladders: ladderbridges and attackers flooding up the ladders as if they were escalators.
Maybe a new skill could be introduced that is necessary to be able to use the ladders in the first place. Could also be used with other siege equipment.