Crossbowmen (and archers too, really), are ranged troops which can do support, play tactically, score kills at distance, etc, and when melee closes in you're only somewhat less powerful then a normal infantryman.
What downsides you have - some less wpf (which isn't really a deciding factor in fights), some less skillpoints (none in case of crossbowman), and some less armour which is generally as protective as papermache in melee anyway, unless you wear plate in which case it's not too hard to outmaneuver you. I liked playing with both a hybrid archer and a hybrid crossbowman much better then with a melee with shield. You can't shoot someone on a roof with a shield, and they damn well can hit you in the flanks.
And using a crossbow promotes "safer" play - hang in the back rows and shoot a bit, try to down horses, friendly archers and other important targets, then when melee closes in, hopefully the 5+ monkeys spamming twoshot/oneshot throwing crap will be dead by then or at least out of ammo.