Even if you win fiefs by voting you still need to make the armies to defend it afterward.
What I mean is that the "grind" would still be there (the grind to survive), fiefs are just an incentive for people to get back in.
I remember that fiefs voting over-extension was a thing even when cRPG was mostly alive, so if people vote en masse and dont play the game (or just plain suck) they will just die away.
Without fiefs voting, I think it will be much harder to get your old pals to come back to fight AI fiefs battles a couple dozen times. 3 years ago it was already such a pain in the ass, today it would be completely surreal. Compare that to telling them day 1 that you have 2-3 fiefs to manage, you need lords and traders to do shit? Those people will of course fight the battles to keep what is theirs and even expand.
Not to add that fief voting is not just a 0;1 thing, you can modify how exactly the voting will work (minimum number of votes needed to acquire a village, a castle, a city; max number of votes per fiefs (prevent waste stacking)); voting transparency to see where you can grab unvoted lands; etc...).
Devil's advocate is how we move forward, you arent being a bad people doing this, more like you are doing good.