The usual counter to HA/HX is archers and xbowmen. Give them more of an incentive to shoot at HX/HA, via upping the points you get for damaging enemy horses.
Also, I don't mind any ranged cav when I'm playing. Its waiting an extra few minutes for them to die like the rest of my team at the end of the round that I hate.
Its annoying to wait for melee to die too, but they at least usually die quickly and without dickless, inefficient kiting at the round's end.
Even considering removing a whole class cause ur mad is laughable and thus a funny joke, thanks guys I get it. Haha!