Suggestion #1:
Village visiting fees should be visible on the village info screen (when you click on it from the world map)
Suggestion #2:
You should be able to input a "maximum visiting fee" that you are willing to pay. If ever you try to enter a village with a fee higher than this, you will stop outside the town instead of entering it (your "enter" order is canceled).
Presently, the game will kick you out of a village of the owner changes the fee, but nothing stops you from blundering into one. Since you are likely AFK when entering a village, the present system is just going to be a long string of abuses and complaints as people set the maximum fee (100/hr) and manage to catch some unaware/afk people.
The above suggestions should fix it. #1 lets you see it coming. #2 prevents you from entering if it changes while you are en route.