I'd be fine with it if there were more viable counters against it.. Right now, the only choice is to ignore it. Even with a successful suicide charge, the opposing team can just destroy the ladder.
The (flat) roof provides cover against other ranged, allows another blind spot to shoot over shields, safety from cav, a controllable choke point for infantry to flow in, and melee don't know from where you'll begin to shoot from. Slanted roofs are easier to deal with since it leaves them susceptible to other ranged.
There are maps with built-in ladders that pose no large problems. It would be nice if the map provides a limited number of movable ladders. Player brought ladders from spawn, even if there are dedicated ladder users, balance/team-switching makes it difficult to ensure both teams get them.