I do agree that fixing the small pop defense spawn time is the thing that would have the biggest impact for sure.
As for small maps, they have similar balance issues but with a small pop people get to do more fighting - on the bigger maps its just running around by yourself most of the time.
Yes, these are two separate issues that conspire to kill NA2:
1. The huge maps are boring with <20 people, so players quit when they come up in the rotation
2. The long defender respawn means that attackers usually win when there are <30 people, so people rage when they spend a few rounds as a defender with x1 and quit
#1 and #2 are related, since a large map usually means defenders spawn miles from the flag (sometimes farther than even the attackers).
There are other annoyances, like maps where the defenders can't even open their own doors to the flag, so they have to watch helplessly through the doors while the flag is captured.