If you plan on retiring again, get IF. If you don't, get riding. As others have said, yes, there is the issue of not being able to effectively use your PT while simultaneously using your riding, and in that sense Ironflesh is the better choice.
Though, as someone who just lost 133 MILLION EXP going from 35 and an 18/24 infantry build, to 34 and a 15/24 infantry/cavalry build, I'd endorse riding. Sure, I can't use my Athletics at the same time as my Riding skill, but it's really, really nice to have a new and different playstyle open to my character.