I played HX because it was fun, not because it was OP. But yeah, often it was completely OP. As long as horse ranged is in the game, they will be a trump card in some situations. It's part of the game...and I think the Flag Spawn changes have fixed it to a great degree as far as affecting the round goes. That HA only gets a minute to ride around like a tard now, not 5 min.
How about an inverse relationship between HA and effective ride speed as your speed penalty? It's something to be tested thoroughly, but my rough idea would be that a full 5 HA (11/30 HX specialist) should take your horse down "one tier" in speed. So an Arabian (40 speed) becomes like a Steppe Horse (37) and a Steppe Horse becomes slightly slower than a Sumpter (35). The less HA you have, the less deadly you are while running away at full speed, so you get more speed as part of the balance.
One problem with that is that it would be class crippling. So instead of a blanket speed nerf, the effective riding velocity could go down during the bow draw, so that HA have two choices: Run if they can, or Shoot at chaser horse (which allows chaser to quickly get to them, so better make that headshot). Further application of this idea would have the horses maneuver stat affected by HA when the bow is being drawn (or XBow shouldered).
I like choices, I don't like taking things away from players. The HX reload nerf is a good example, it took away the class viability as anti-HA, our only real competition, so it shriveled and died.
My idea to fix you, San, is to add a stamina bar. That way you have the choice between swinging your sword a lot/raising your shield or running at Terminator speeds. You don't get your AGI taken away, you instead have to think more about how to use your AGI. I'm sure highly skilled players like yourself would adapt and own.