It really depends on the weapon you want to use, though. When I was 2h, 12-15 agi was enough, but with a weapon that can only do stabs and overheads, you have to be a bit more agile.
If you go for riding, you can always carry a shield skill 0 shield. It will at least catch a few arrows. Or you can go 1 shield skill and use a nordic, that will also catch a few blows before going bust. Although then you'd only have 3 riding, which means only tier 2 horses.