It also isn't a battering ram
It currently hits you with about the strength of a wooden club in the hands of a near-peasant. I can stand in mail and headbutt the horse and an average horse won't knock me out until the game draws. Now as "highly annoying" it is to get bumped, it would also be highly annoying if people stopped horses by simply jumping into their path. People already do that to avoid getting hit. It's "highly annoying" that you can instakill a horse with a dart. (Imagine, all those fools which made pike and shot formations and shit, they should have just carried 8 darts each to massacre any cavarly charge).
If you want to argue from a realism standpoint, do you really think cavalry worked in such a way that the horses just fell when they hit infantry formations and that's it, cavalry charge done, now they're all face down on the floor together with their horses?
And if you want to argue from a game balance perspective, well, 1h/2h bump&slash cavalry is all but extinct, which is the type of cavalry which relies on bumping. Almost everyone is a lancer now, since it's way more effective.