It's normal, 100% of all time I die in any crossbow, from light to arbalest, just 1 shot because I have a very high riding skill. Even with heavier armor I still cannot survive an bolt from arbalest. Get a shield then you be fine, as infantry you should get as many str as possible, or some agi to dodge bolts.
