If you are not the owner, you can´t change it.
Both the current owner and the future owner have to be in the fief to transfer the ownership, there is a menu for this on the "info"-page.
The only way you can get it without him is attacking the village.
