Characters without athletics are really unplayable, in my opinion. When you have tons of strength and ironflesh and use +3 plate armor, it might be okay to simply tank the hits, but otherwise, athletics gives you evasion that is much more important than additional damage from wpf, protection from a shield or a tiny bit of extra hp due to ironflesh. It was a good decision to go for athletics, and even as cav player better to prioritize it over WM for example.
If others tell you to respec and go for a build without athletics, don't listen to them
It is unrealistic to assume that your horse will survive every round, and you cannot even spawn a horse on every map and use it well. Also, when your horse is dead and another one is close, athletics can help you to run from close enemies and mount it. Melee footwork to maximize your speed bonus and minimize the enemy's speed bonus is super crucial, too.
I fully support your priority setting.
And as Macropus said, go for WM next.