That's due to the nature of cav being able to double your damage. Using the damage reporter, I saw 1h/2h cav dealing 100-130 raw damage, while a LHB dealt 50-70, 80 if I was lucky, all with 15-21 builds. Slashing when you're still tells you how much damage you're really doing without speed bonus. My final performance in an afternoon was 1.4:1. I was able to get top 3 a surprising amount, but the damage was so low that I rarely got kills. Instead, most of my success came from downing other horses to acquire points.
I might try playing on it some more to get a better understanding, probably with a pure cav build of 15-24 or 18-21 this time. Left swing when facing the right side also has a peculiar hitbox in front of you.