There is the time, like 1-2 seconds maybe, when you start aiming, when you can't shoot. If you try to shoot during that, you will shoot, but with the delay you're mentioning. Works much like bows in this regard.
Other than that though, it's probably just you ^^ I've spent my time with crossbows and never noticed that problem, if it's happening it's so rare that I just forget it.