Anyone whining about others doing a raid is a fucking nancy, man. Probably one that doesn't know how this shit works, anyway.
It doesn't even do anything spectacular. If you're keeping a bunch of goods and/or gear inside a fief with absolutely no garrison aside from population, you're being a jackass. If another party actually somehow FINDS THIS INFORMATION OUT (would need to be a spy, exploit, or amazing reconoitering), the fiefholders deserve every bit of shit they're keeping in there to be stolen.
If you don't have shit in the fief, but no garrison (only population), then the attackers will have wasted 24 hours and will receive nothing. This is ample time to jockey your armies (assuming you have them and they are contactable...again, anyone complaining about raids is probably woefully incompetent and doesn't have shit) to fuck on the raiders.
I'm not entirely sure, but reasonably confident that raiding is a non-issue for AI fiefs. I'm under the impression the AI buys weapons in a fashion preventing raiding and re-raiding troopless AI fiefs. I ain't seen anyone do it, anyway. This leads me to believe it is impossible, or maybe everyone in strat is being a good baby boy and you just tell me the chances of THAT.
If someone shit-talks you (or anyone) on the forums for raiding, you can be safe in the knowledge that I'll be writing up some real vehement shit-talk for them in return.