CrazyCracka, I can and do get glances for almost no damage in my ragged outfit and hide boots. A leg or maybe its a shoulder bump doesn't do much, especially if I manage to move in the direction the horse is moving, but if I take the full chest of the horse to my face, it does 25-80% health depending upon which horse it is. (12-15 strength, so not many HP's) If you guys are just standing there and being hit, or are moving towards the horse, the speed bonus kicks in making it more damaging. Move away from the horse, or don't get caught full on, (ie, get a glancing blow) it isn't that damaging.
I take horses on in the field all the time. In my opinion, the risk/reward/damage ratio is fine.
When I was testing out the HX, the only class that I found was really vulnerable was the shielder. You can bump charge them almost all day long. I suggested this before, but the shielder should get a damage reduction (or no damage), but still a knockdown when they are holding their shield up to block/absorb/deflect the impact.
Everyone else is fine.