No, the guy on the horse is not super weak to archers. It is the HORSE that is weak. If we don't factor in headshots, a cavalryman can easily wear around 40 body armour and afford it too. Your will only be able to afford light horse (you can thank chadz the horse hater
), so horse armour ranges from 5-20, usually under 10 off the top of my head. Decent armour does wonders in stopping shots from arrows. As a horse archer, I can take 5-7 hits to kill someone in mail, while the light horses only take about 3 max and often 1 or 2. And a horse is a bigger target with a bigger head to hit.
If you worry about archers that much, put more points to shield. Use the (force) field to draw arrows meant to hit the horse to your shield.
6 Athletics? That's good enough to be an infantryman. Not sure whether this is a cav build, or just a mounted infantry build.