In battle there's no auto-switch, and it balances out. Many maps in siege are unbalanced, not really because of map design but because of player population. Some castles are freebies in 10 vs 10, but impossible to take when 20 vs 20.
So what, you lose some times, not a problem if you're having fun and not losing too much gold. If you get switched to a losing side every other round, your average multiplier is 1.5. You win 4 rounds in a row, you get 2.5. So ,according to my math, if you win half the sieges, and lose half, you will get a X2. Which is 25% more.