Make it so that leaving your fiefs unattended is dangerous, and ignoring them will gradually increase the chance of civil unrest and a revolt, that the fief owner must defeat as a battle in order to keep his fief.
This civil unrest factor could be tied to simply having enough faction members present, but preferably more complex than that, with a lot of non-faction players in the fief increasing the chance.