It is clearly enough. And I doubt you have less than 6Gb DDR3 ram.
So I guess we just have to blame TW's code like for everything else. Siege spawn script ftw. It is actually possible that the synchronisations in the code are done in such a way that the game will stutter under some conditions no matter the CPU and all that. Doesn't help much but heh.
i ever tought of packetloss/netcode related issue because fps are fine, cpu time is fine, no debug errors, but blocks/swings are slow/unregistered like playing under heavy bandwidth usage condition. as chadz stated, must be the siege code
EDIT:
yes my machine is ok for warband but maybe my old zyxel dsl router and the crappy italian dsl are to blame.