Cav is not the easiest class. Go ahead and respec to one. If you do super well, then congrats you are a good cav player. If you die early and often, you are average.
Cav is very fragile (cuz of speed bonus)...even heavier horses like destriers. This is true even more now with the cav nerf because leg damage got increased against horses. Also, you have to be VERY picky on who you attack, from what angle, and what terrain is surrounding your target. It is definitely a skill-based class, not just a money one. Infantry is easier to play because it is more forgiving of mistakes you may make. If you make a mistake with cav, it will kill you more often than not. It's a high risk/high reward class. You have to be skilled to maximize your killing power and minimize your risk and exposure.
Yes, some maps are easier for cav to do well (Large, open ones) just like some maps are better for infantry or archers. All-on-all, cav is well balanced. I think they should remove the leg-damage nerf actually because now that archers are back up to snuff cav is balanced against them.