There should definitely only be battles from at minimum 14:00 (2pm) and forward, IMO even later (e.g. from 16:00 (4pm) and up to midnight or 1 hour after maximum).
It would be OK to have it as it is now if people were actually hosting battles of any importance during the early/late hours, but no one does, and everyone who does is basically abusing the system to avoid playing the game - or at least minimize the amount of opponents for the battle, which shouldn't be the function nor desired in terms of game-design.