there is some kind of bug with fiefs
The guy who owned Caraf Castle is afk for a very long time, so we decided to re-attack castle, our guy exited faction, attacked caraf castle and after that caraf become neutral castle and we don't own it anymore.
So if anyone would attack our castle, why would it become neutral!?
So the guy who attacked the castle (that was still yours) was in no faction.
Thus the castle became factionless (as the new owner is factionless) BUT it is probably still him as the owner of the castle (the guy who left DRZ and then attacked).
I'm guessing you then accepted him into the DRZ strat faction? That's where the bug is. If an owner of a fief changes faction, the fief doesn't change with him. Same happened to me and some others when we changed from Risen to Merciless. We still had full control over tax, fee, equipment, gold and troops, but it remained "Risen" (in your case "neutral"). The only thing the AI controls is the BATTLE ROOSTER.
This is what you do:
1. Make the new owner remove EVERY SINGLE ITEM.
2. Make him REMOVE ALL THE GOLD.
3. Make him REMOVE ALL THE TROOPS.
4. Kick everyone out of the fief.
5. Attack the "neutral" castle with the guy that is in the DRZ faction now (can be the guy who already owns it).
6. AI signs up people for defense, you guys sign up people for attack.
7. Fight the battle. Shouldn't be hard as the defenders have
no weapons or armour. At least they shouldn't have since you removed all the equipment and gold. When I did this with Ilvia they all spawned with some pelt coats. Some randomers had payed the visiting fee and thus given the fief some gold. No weapons though.
8. ????
9. PROFIT!