Add a chance to get knocked off your horse. 'The Deluge' had this feature and it always seemed to make Cavalry seem less like butt pirates and more like a tactical advantage if your team used them correctly.
Also, don't make the horse ride faster based on your riding skill, that's like your automatic car accelerating and moving faster because you know how to drive a stick.
And finally, I think there's generally more than just "WALK" "LIGHT RUN" and "FULL GALLOP" in terms of horse gaits, and they all have separate uses. Make turning extremely difficult at full gallop so the horse can't corner a 90 degree turn faster than a human can move out of the way if he's aware. Make the only speed value that allows the horse to be capable of turning efficiently a light run or a jog paced gait so that they actually have to slow down to corner that wall without running face first into it.
Also, if at all possible make horse bumping on a common horse (Steppe, Desert, Rouncey, etc) extremely difficult, as a horse is a living creature and not a motorcycle with a plow in the front designated to kill all humans, they have a tendency to turn away from people unless they've been trained for nothing but charging into groups of enemies.
I do however support the suggestion that the Riding skill should be used to decrease a delay in the response of movement keys for horses, and maybe increase the ability to turn with each point put into riding. Just please don't make the horse run faster because the rider has 8 riding.
Remove the insane blocking area you get from horseback. Arrows to horse's face being sucked up by the shield of the rider, and pikes to the horse's body sucked up by the rider's shield/down block is getting very old.
I have no issues with cav except for that one right there, which is just completely broken.
Also this ^