Any time a fief owner leaves a "regent" should be appointed who is on control of the fief. There should never be an instance where a fief owner is having an EU vacation and the fief is unmanageable (so long as you have other members of your faction).
If a lord leaves a castle, and the castle gets attacked in his absence, the people left behind are going to still organize, and someone is still going to take charge.
It seems like this would help solve a lot of the issues we currently face (with items becoming so numerous you can't access all of your equipment).