Once you get past 64 body armor, the damage reduction is quite noticeable. Better than the few extra Hp from IF. I would use 6ath with 9.6-14 weight body armor, 7ath with ~ 12-16 weight, or 8ath with 14-20 weight, all with heavier loomed gloves, if you want good protection while remaining fast. Plate mittens and below are pretty affordable. Every point in athletics are as though you are moving in 7.5 less weight of gear.
With the lighter armors, people will hit you very late/early in their animations, ironically negating the use of ath. Arrows also deal more damage under 60 armor, even if you have good health. Higher armor + good ath allows those glances on bad swings to occur, and if they hit too early/late in their swings, the damage mitigation is pretty good.