I'd say max out Athletics and riding instead of IF. That way you can steal every horse once yours is dead, and you can use armored horses => your horse will die less to ranged and you can deal damage to unaware lightly armored archers. And for extra damage, you can put the leftover IF into horse-archery to reduce the mounted ranged damage penalty (2% more damage per HA skill).
Edit:
You can also have a Dragoon build. A pure foot crossbowman build, but with 0, 2 or 4 riding skill, and use mule/sumpter/rouncey.
But it is more of an infantry support than "following cavalry" build because of your low speed.
Edit2: A masterwork light crossbow and steel bolts are a big advantage, +7 damage.