Well, Throughout history it has been done many times where castles/fortifications are built into the side of mountains, where even a majority of the fortification itself has been underground.
It's convenient, really. I'd do that if I was building one. Cuts down how many sides you can be attack from dramatically.
It wouldn't be fantasy, so no problem?