I have a decent rig as well and PoE freezes sometimes, especially when I open a strongbox and lots of enemies appear at the same time (also with visual effects), lasting 1-2 seconds, and, when I alt tab and go back into the game, fps drop a lot sometimes until I zone (like opening a portal and going to town and back; that fixes it).
That plus the desync make me not play HC either. I already lost a few HC characters due to desync.
The game is just not very optimised. I was reading about D3 vs PoE yesterday, and many people claimed that D3 has a solid engine but poor content, and PoE the other way around.