Yeah, if you want to focus on a Destrier, you don't really need more than 5 riding, and would be better with more PS and IF to play like medium cav, soaking up arrows and some melee hits to get kills. Its a tanky horse, but if you aren't a tanky character, you will get 1-2 shot by ranged and rarely outlive your horse. You often won't get full speed bonus either because of lower horse maneuver, so higher PS pays off. I usually did 21/15 hybrids to 1hand/lance with a Destrier.
With a low strength, no IF build, an Arabian would be better because the idea is to rarely get hit by anything. You'll do more damage with lower powerstrike overall. Its much less forgiving of mistakes though, and until you become a good horseman you will find yourself dying to ranged often. At least it'll make you get good fast!
I never went over 4 shield either, just to use the elite cav shield. Often did 3 WM as well, to do 110 1hand 80 polearms. Not ideal for melee, but just fine for mounted combat. The extra points should go in IF, because ranged attacks hurt really bad when you're riding towards them.
