I've been playing an xbow user and for the last three generations didn't put a single point in IF. Having 15-18 strength is enough extra HP for a ranged character. With light or medium armor, you won't die to a random arrow hitting your shoulder or a single horse bump. You don't need to be any more meaty than that since you ideally won't be in close combat anyway.
I can even throw on some mail and grab a decent 2h to get kills if it's a crappy map for ranged combat (or if I am bored and just want to mix things up).
That said, if you are going for a heavily armored infantry or cavalry build, points in IF change you from a tin can to a GUNDAM