As a 2H cut/blunt...
HA/HX are just annoying. You don't really "fight" them, they just run from you and you can't catch them on foot. And they don't ever seem to be that effective other than being the last one alive in Battle. In Siege they don't amount to shit.
Other 2Hers who have the lolstab animation, just because they can always outrange me with the stab. But that's on me - I just have to be more aware of what I'm fighting. The spinstab is annoying as shit tho.
Anyone that abuses the jump glitch alot (where you swing right thru their body when they jump-spin).
Stealth-from-behind-Cav. Of course you don't really "fight" them either. They just couch you from behind and you never hear them coming. Other than that, Cav are not that annoying imo.
Spamitar/shielders are probably the most annoying for me. Either that or 1h hammer/shielders, but of the two spamitars are worse.
High-Agi polearms with either spears, awlpikes, becs, or halberd types are a pain 1v1. That godam 1-inch turn/spin stab is huge BS. Plus the polestun & hiltslashing. This coming from someone who has also been polearms for 2 Gens.
Pikers/lancers are a pain in a group, impossible to block other melee plus a piker who can hit you from a mile away through their own team. Also the fact that they can stab you through doors, gates and around corners = winsauce.
Archers, Xbow, Throwers are annoying mostly because I get hit by enemy as well as 'friendly' range usually right when I'm in the middle of a good melee fight. Other than that I can expect to get hit by them when I'm in the open or going up ladders/ramps. But then other times I sneak up on them from the side or behind - so we come out in a wash there.
Friendly cav, of course you don't fight them, but they're annoying due to trying to KS and end up bowling you over just before you make that last killing blow, or of course all the friendly horsebumps that take off any variety of %'s of life depending on how hard they hit you.