I'm no expert, but it could be your ISP's traffic shaping, or something about the specific modem they provided- if they require you to use their modem. My ISP does this, and I had a problem similar to what you described. Only thing that solved it was transparent bridging through a router.