Make sure you have AT LEAST 2 GB free on your harddrive. This is space used by the HD for virtual memory, which is like poor mans ram. This is especially needed for systems with less ram, like you with your 4 GB.
Defrag if needed (use the analyze button), as it will help your load times dramatically and be less prone to stutters if you dont have 'load textures on demand' on. If you are playing via Steam, you can also have Steam defrag game files, which can be helpful.
You might also try turning off any security software, as they are notorious RAM hogs.