Hmmm. My experience with small castles says that they are practically indefensible. You either have to funnel attackers through a handful of chokepoints or make sure that they need longer than 30 seconds to reach the flag, otherwise they will quickly swarm it. I had to update Holmet Castle many times because of this.