It should stay. I think your athletics gets set to 1 or 0 at those places.
It's hilarious seeing a ninja dodging arrows and suddenly come to a near full stop only to get shot.
it's quite annoying, i've been wondering why it is there, too... it should be changed.
if the ywant to nerf speed in castles, maybe set it to half athletics - but i'd prefer full speed