People sign up for the attackers, just not when the attackers have less troops than the village (as has been the case for every single fief battle so far except one, which the attackers won)
People don't want to sign up for a side that is going to 100% lose, when you attack a fief with 400+ troops people will sign up for you.
Of course though attackers have a merc limit considerably less than villages, but it evens out to both sides having roughly the same amount of mercs since lots of people who sign up for villages don't show up.