My idea to improve that, would be that you need "food" to support troops/population of a fief. You can also assign food from one fief, to be designated for another. That way villages would actually be extremely important, and you could essentially besiege castles/cities and force the people to come out (like in reality).
Also still not sure if Aldo was being serious or not (probably semi-serious at least) with your quote. Since he's always been my favorite couching target, I'll assume he loves me as well and was only jesting.