Something needs to be done. The main issue is that ladder exploits, both inside and outside of castles, can often ruin a map for defenders. Additionally, on some maps, a failure to exploit ladders means it's almost impossible to get a good ladder placement without it bugging and falling through the wall. I've spoken to Lucius about this and he's said that if, for example, we remove ladders altogether we'd need to seriously re-jig some maps or even remove them altogether. Alternatively, ladders could be made un-stackable and thus less exploitable. This would still leave, however, many ladders floating in the air and impossible to reach by the poor defenders.
Anyway, I thought a thread should be made to start discussion on the matter. Other suggestions that've been floating about have been making ladders quite expensive to repair, removing player ladders and replacing them with fixed ladders, and limiting it to one ladder per person.