If you guys are looking for more realism then don't forget the massive fall damage when your mount gets killed while riding at speed.
I'm not looking for more realism, I'm looking for a horse not to be blocked by a person in the video game. "OOOOP I'm standing in front of your horse, you're stuck!"
Use whatever justification for that you want.