Your speed is affected by many things other than just athletics and weight. Terrain can slow you down (e.g. the floors inside some castles), using your weapons to attack or block (e.g. raising your shield slows you down, and continues to do so for about two seconds after you do it), or even just going up hill.
Try this: Spend the first minute of a round without attacking or blocking. Run about, distract people, and avoid projectiles. It'll show more starkly the contrast between forward sprinting and folks who block and attack more often. It'll also give you a better sense of how different terrain can affect how you're moving.
