WPF doesn't affect attack speed that much. Getting 150 WPF and you'll have 110 effective WPF in full ultra heavy plate. 100 wpf = using weapon as stats list it. 1 weaponspeed difference is equivalent to (roughly) 20 WPF [once you are over 100].
You won't be able to spam people with the difference of 130 wpf vs 170 wpf, the attack speed difference is that of a weapon 2 speed faster. WPF doesn't have a big impact, especially when you consider it has massively diminishing returns.
HOWEVER, increased athletics increases character movement speed, and stepping into your swing will make it hit in as much as HALF the time it would normally take. Of course, stepping in the right direction matters more than the speed you do it in.
So, what is the point of more WPF? You can't break the block-swing-block-swing cycle with it. You can with proper footwork and weapon stuns, though, but that is a different topic
You see where I am going with this right?
A full black plate guy with no wpf? Going to be a snail that isn't even using the weapon as its stats are listed. An agility build in light armor could spam him, yes.
But if Mr. Blackplate is using an innately fast weapon? Nope. Can't. Be. Done. Sure, it will be slown down to that of a slower weapon's normal speed, but uh, so what?
In fact, even if Mr. Blackplate had no WPF if he stepped into his swings and swung where you were stepping, He would still not be able to be spammed.
Anyways, My point stands. You get agility for maxing athletics. or riding/HA if you have a niche non foot-melee build.