This might solve the roof camping problem, or not.. but anyway here are my reasons why I definitely vote yes:
Now that I'm shielder, I use huscarl so I don't have space for ladders, but if huscarl would take only 1 slot I could bring ladder and because it would take only 1 slot, I could bring also even another ladder. And if I'm right the ladder's weight doesn't reduce your weapon speed or walking speed unless it's in your hand, and they don't require repairs.
I know that there will be haters who start saying "then the battle would be ruined cuz there're way too many ladders and so on" but we could even try this out.