It would be nice if speed bonus had a slightly more pronounced effect (if possible), help giving you agility-based bulk compared to purely soaking more hits. Negative speed bonus based on the direction of the swing, not just simply moving backwards. If it's impossible to change, adding another layer on top might be possible.
Slow movement already helps you take less damage since it's less likely you'll be hit by a very effective attack. Even when stepping in, it's not that bad.
A larger speed bonus effect would offer extra damage with higher risk, and less when moving away (should be the same with horses proportional to their speed, not riding away and taking much less damage than they should).