The other annoying thing about ladders in siege was people leaning them up against roofs where the defenders couldn't climb up to. Then suddenly it was raining men. If it was just smaller ladders that could only reach the curtain wall, that'd be ok. Just none of those that go six storeys above the curtainwall, then another that spans the courtyard to some tower, followed by one more that lands right above the flag.
Perhaps if the ladder is less that 45 degrees it breaks? Then all the devs have to do is redo the ladder mechanics and redesign all the castles and then we're laughing!