The problem are largely the "best" horses, which are just too common ingame. If most of cav was riding around on a palfrey or rouncey, some on destriers / sarranids / coursers and armoured horses were a rare sight, cavalry would be a hell of a lot more balanced.
And then perhaps we could play on more open maps. Right now, you're either the king or trash because it's some counterstrike map. The kicker is that light horses (ok, sarranids and such are so agile and small that they are a bit of an exception) can be killed in one go if they approach badly an infantry formation. Heirloomed armoured horses can make many mistakes which kind of breaks balance, in my view.