While I also think the defender prep time is a bit too long, that's mainly because the defenders, well, don't have anything to prepare right now.
Later when we get more siege equipment and the like to build, it will be useful. The last one wasn't long enough to completely make what you wanted to make.
So in stead of reducing the time, what about implementing some sort of "Ok, we are ready" button that a commander can press? If both defender and attackers signal that they are ready to fight now, the attackers spawn and the battle starts. This was not originally my idea, but it's a good one.
