If you do 4 PS and 0 shield, you can do 10 riding. I think that'd be a little too much though, but would be hilarious.
4 athletics I find is good enough to use my heavy lance as a long spear when I'm dismounted and run away when need be, but I usually just fight wherever I land. In strategus, its enough to get back into friendly lines. Need at least 3 shield to use most of the shields people buy in Strategus battles.
You don't need Weapon Master for any dedicated build except ranged, and you especially don't need it as a cavalry lancer.
I used to do 0 athletics cav builds, but I enjoy melee combat fighting enough that 0 athletics makes me feel too gimped and I'd get mad at the game. 4 athletics can make you twice as fast as some NA players anyways.
Just throw whatever points you can into Iron Flesh, because I can often survive a full speed light cav lance right to my chest and get away with barely any HP left. The only problem with the build is the relative weakness of having just 15 STR when it comes to melee fighting, but considering if I'm fighting on the ground I screwed up already, its not a big deal.