I don't see how someone could abuse, and ofc it has to be an army of their own, this is the meaning, having an army without controled village recruitment.
But i agree with you on the raiding part, bandits should not be used to own a fief. Bandits don't like fief, they like rape, pillage and burn!