Infantry should lose control of their characters and run away in panic when horses charge, unless they are standing in formation and facing the charge.
Horse corpses should fall on people with 90% chance of instakill. People spinning around should have a random chance to fall. Jump height should be decreased. Being bumped while jumping should have a chance to knock the infantryman out. Stabbing a horse going full speed should render your weapon useless as it is stuck in a horse.
Horses should have less HP, more speed, and more bump damage. Full-force horse collisions should occasionally knock down horses or be handled in some different way then silly bumper car collisions.
I mean, there should be more skill in dealing with horses then just holding out your pikesword and stabbing.