Well, after my playing HA for a few weeks, I couldn't pick any counter really. It depends highly on the way you play. Personally, I always stuck rather close to the fighting and got therefore killed by a lot of random weapons.
But if you choose to play like an asshole - which some of our HAs obviously do - then there basicly is no counter.
Xbow reload is fucking slow and when you spot one aiming at you, I rarely got hit but my warhorse soaked the bolt and before he could fire again I was already far far away or harassed him during his reloading.
Other archers were probably the worst cuz they can spam as much as you can with a steady aim. But again, seeing the archers pocket - they rarely spread out a lot - gives you the means to stay away from them. Not to mention that my unloomed warhorse easily takes 8 and more arrows before it dies.
Infantry is only dangerous when you make a mistake in riding. They are useless.
My conclusion is that there is no real counter but if something has to be named then I say Archers for the spamming capabilities.
This is particularly for Horse Archer. That class is pretty strong imho.
The problem with HA and HX is that they don't control the flow of battle. They can't capture flags. So if flags were to spawn earlier in the round, they would be even more useless to the actual winning of rounds. They soften up cavalry, and soften up ninjas who don't stick with their teammates, but they do jack shit for winning rounds (unless the enemy team is retarded).
The best counter to archers we don't really utilize in game, and that's MotF. If infantry are given an objective to fight over (territory) rather than "kill everyone" they can fight for ground. Instead what we have now, is infantry choosing between camping behind a building for 3 minutes, or pushing up a hill to fight archers who are in a superior position.