Just guessing here but it could be the same as the size of your army (the dot on strat map)
Under 100 men you have a tiny dot (very short battle time)
Under 500 but above 99 you have a 'normal' dot (about 30min)
Under (2000?) but above 499 you have a large dot (about 80min)
Above 2000 you have a massive dot but you're just wasting tickets battle time still about 80min (Devs way of stopping an endless battle between 10k men on each side)
As Haboe said it will be dependent on the largest side (attacker or defender)
Dependent on the smallest army (attacker or defender)
However, type (battle, siege, raid) probably takes effect.. so if you siege with under 500 men (presuming they have less or against under 500) you probably still get the 80min as it's a seige to allow time for the building of siege gear.