I'm a dedicated shielder and I don't think shielders are imbalanced when it comes to speed. There are definitely things which need tweaking (stabs bouncing too much, for example) but speed isn't one of them. As a shielder, with only 5 athletics, I'm often the fastest running player on my team. Using a medium shield and footwork, I find I can also land some of the fastest blows in a fight. Against bowmen I orient myself to receive or avoid every incoming shot, and rarely get hit when an archer is in my line of sight. Against 2h and Polearms I use opponent-specific tactics and rarely find myself outgunned (though I'm often outmatched - but that's entirely different).
TL;DR: Shielders are fine and Fallen hax