I hate xbow cavs, but I respect them greatly (like most of the playstyles and people I "hate"). They serve a purpose, that purpose is keeping other cavalry off their friendly infantry. They are pretty damn good at destroying horses, not so good at killing 20 enemy infantry when they're the last one alive.
They serve a purpose, yes they can be annoying, yes it's probably not historically accurate, but they are just as much serving a vital role as anyone else, every time they hit an enemy horse they are helping their team win.