I find it extremely satisfying to kill footmen as cavalry (1h cav here), and I find it challenging to do so most of the time. Everybody seems to pack a spear these days.
Horse archers annoy me, but I either avoid them or team up with other cavalry.
I try to avoid complaining on how I die, as I know the vast majority of the time it was my fault.
The most fun for me comes from the ragdoll models, wish more games had them, and the plain fact that I'm playing a FPS-like action game but it's third-person and has swords.
As for the ladders, I think certain buildings in each map should have hollowed-out, "inside" models with staircases to the roof and windows, and ladders are removed. That way, archers can stay up high if they like, yet frustrated melee can at least die throwing themselves at a possible-to-kill enemy camping a staircase, rather than an unscalable building. I have no idea if there are models already made for some of the building interiors, so this may not work.