Wouldn't this lead to the kind of "sniper tower" constructions like in older versions of cRPG?
Just place two ladders next to each others, not totally perfect on a line but looking a bit towards each other, so that they will connect somewhere in the air. As soon as they do they will stop and you can climb up. Especially handy on plain maps, protecting you from cavalry for a long time, and also from enemy fire while being allowed to shoot from an elevated position yourself.
Thank god this is not possible any more.