We didn't have that much trouble defending it tbh. All you really need is good archer/s to kill everyone trying to crush the gate to the keep, one guy ontop to break ladders and proper infantry to defend the backdoors. No biggy.
That sand castle on the other hand, which has that circular walled-off section within which the flag is, bah, that one is the problem. At least if you dont want to use the blocking ladders.
PS.