Difference is massive. Going too heavy usually results in all kinds of disadvantages. The game is a tradeoff. Unless you go full retard on your stats(i.e putting points into horsearchery while being 2h or not adding ridingskills etc) everybody is generally on the same ground. Being with light gear usually results in better overview of the battle, you can pick your fights, dodge cav, dodge ranged, swoop into some guys side and hit him, make the other guy miss his stab or overhead.