Agreed. Perhaps two could start pushing it in a very very very slow pace while six pushes in a very slow pace. Plus, those who are pushing should be at the side of the siege tower itself, let it be pushing towards a wall or away.
And perhaps - if that would be the case, Siege Towers should not be able to pushed UP a hill, no matter what - as it is as unrealistic and stupid as diesel engine towers.