The point is that people bitch about cavalry and want it removed from the game, even though the title of the game "mount". Being on a horse has obvious advantages, but it also has obvious disadvantages as well. Just like every class in the game.
I don't have a clue about people wanting to remove Cavalry from the game.
I still think Cavalry is an integral part of the game and will remain to be. Just don't mistake "integral part of the game" with "Has to be well over other classes in term of fighting abilities".
It's mostly the fact that as a cavalry, the obvious disadvantages can easily be taken care of with little effort.
As once mentionned, as an infantry without a shield, outside of a few long weapons out there, cavalry will stab your shit and archer will shoot you and effectively be your hard counter and all you can really do is spaz around hoping the arrows doesn't hit you or very dearly hope that while you're busy with 3 guys, that some guy zipping in and out of the battlefield at ridiculous speed on uneven terrain won't one hit kill you.
Meanwhile, as a Cavalry, they have no clear hard counter, all they have are defensive counters or deterrent if you like that word better. Arrows to the face of a shieldless infantry is not a deterrent, long spears and pikes that can be dodged by just going a few degrees off-course are a deterrent as he can just continue his course somewhere else and impact with the next infantry that's already busy fighting one too many people.
Don't even get me started on archers, unless the archer has some serious foresight and can literally see in the future, unless the horse is extremely close to the archer, the horse will easily just change course and dodge the arrow without a single problem because at the speed they go to, they're impossible to lead unless they're going in a straight line, but with the current maneuvrability values of horses, this is not the case.
I just hate the sheer power that Cavalry has while remaining quick and agile. I played Cavalry myself, given a level 30 and above build with a heirloomed horse that can turn on a dime on the most uneven and shitty god damned terrain the game has to offer quicker than infantry can even walk on it, it pretty much feels like you're an infantry on crack that's using some sort of 50str/100agi build, you do the most damage of any weapon class in the entire game and yet you remain mobile enough to dodge any sort of "counter" (Read: deterrent in the Cavalry's situation) there might be against you and in the off chance that you do get brought down because you screwed up, you still have all the chances on your side by dint of shitty terrain collision making it so you're on your ass UNDER the ground, horses absorbing hits for you or the rider just outright being launched several meters away without taking a single digit of damage or getting conveniently teleported away from where he got brought down, at which point he can begin his new life as a perfectly viable 18/18, 21/18, 18/21 infantry or whatever god damned build your cavalry guy might be.
One last thing that seals the deal to me is how you're not safe from Cavalry anywhere, not on roofs, not on elevations, not on steep cliffs, fucking nowhere. Horses are god damned mountain goats and will climb the worse fucking terrain known to man and will trample your ass to death while simultaneously setting a new record for how fast he got to the top of the Mt. Everest.