You should plan your build for level 30. Level 31 takes a long time, you want to be "complete" before that.
Here's my ideal speed build, currently running something like this with a steel pick (with less agi, not yet lvl 30).
Level 30
Strength: 12
Agility: 24
Skills to attributes: 2
Ironflesh: 1
Power Strike: 4
Shield: 8
Athletics: 8
Weapon Master: 8
One Handed: 173
With light armor (tunic over mail or other ~9 weight) you'll be very fast, hit hard enough with the pick and shield holds forever. I wouldn't recommend going with cloth only, a simple armor doesn't cost much and gives you a lot more survivability.
Remember though that this is mainly a support build intended to cover other people and/or strike at ranged units - heavily armored people still take quite a few hits. More strength would obviously give more killing power, but in my opinion speed > all.
I made the mistake of taking throwing as backup as I got annoyed by all the cav and not being able to do anything to them. With slot system huscarl+pick takes 3 slots, so there's not much room for throwing. Ofc if you drop the huscarl and get a 1 slot shield you can get 2 slots worth of throwing, but personally I never really used it much. Even before the slot change I had pick+huscarl+ 2 stacks of franciscas and they were more a distraction than benefit - I would get shot trying to throw them and die.