I spec-ed it for a while last night and though it was absolutely brutal, it was quite like a real medieval siege. I mean there's a reason that armies would SIEGE a castle instead of assaulting it. Castle walls can give a force multiplier in factors of ten if implemented correctly (such as they were last night). Albeit, that castle was more difficult than most, it did EXACTLY what a castle is supposed to do, be nigh-impregnable.
Now if you want to avoid the castle walls the attackers could use Medieval siege tactics and ignore the castle, cut it off from reinforcement and force the defenders to sally out when they begin to run out of gold. In addition, the attackers could take easier to capture fiefs and force field battles that way.
My overall point is that defense is SUPPOSED TO BE EASIER. In order to attack, the aggressor needs to MASSIVELY outweigh their opponent if they wish to directly take a castle. Otherwise they have to do what the medieval armies did and drain the blood from the enemy by taking their villages and forcing a battle on open ground.
