For an agility-heavy shielder, IF is usually the different between being 1-shotted or 2-shotted by a 2-hander. So if a few skill points can effectively DOUBLE your durability, that's a good deal. And don't think for a second that having a shield means the shield is your only defense; between teamhits, random projectiles, enemies backstabbing you, kicks, attacks that inexplicably go through your block, mis-timing your attacks....you still get hit A LOT. My shield outlives me at least 90% of the time, and I'm only moderately aggressive.
That said, I'm 18/21 at lvl 32 and no IF, but I chose to have throwing and a lot of shield skill and weapon master. If you are pure 1h, lvl 32+ builds like 21/21 or 18/24 with IF and foregoing some WM/shield are very effective.