The texture bug? Dunno what your talking about there. Maybe that's what it seemed like for you the (VERY) few times a ladder was broken and due to it's position we were flinged up ON the wall instead of to our death. Happened to me once and was hilarious 
Castle is so big one cant see everything : its how rumors spread!
I tested it in other castles and it is there able too. In one castle easier in other more difficult, but able.
But is it supposed to be like that ?
Its already hard to man the walls where attackers are assembling for a ladder charge, if on every castle there is a preexistent route open for attackers to break in it could be too hard, even if in this siege defenders dealt with this without too much problems (you still almost managed to gather enough troops the first time, but I spotted your little trick and called for reinforcements
).
One way of dealing with the gate problem could be to open the gate and destroy ladders and come back but its still really dangerous for defenders.
