Horse archers eat horses for a light snack.
Pedestrians need better situational awareness. Cav know bad situational awareness gets one killed (arrow from no where, lance from blind spot, hitting a hidden man or obstacle) but pedestrians can't seem to get this. Situational awareness, like feinting, is a essential part of the pedestrian skill set. It is nearly impossible to get an aware infantryman as cav.
Or take the easy way. Nearly 95% of maps have a campable roof, where you can throw situational awareness to the wind and tunnel vision all you like. Do you hear cav complaining about that? But you see a few cav killing unaware or afk people who don't bother to use tilda or turn around despite someone spamming "incoming cavalry-from the rear!" multiple times and suddenly horses don't have natural predators?
Horses are the means of natural selection. The unaware gets picked off, so the fittest can fight in the end.