I'm pretty sure that chadz text works in the following way... When you get ATHW:
+3 smithing skill to all items you've got equipped at the time.
-1 smithing skill to all items you don't have equipped. (I'm not sure if it's -1 exactly).
So this means that your smithing skill is biased toward items you've been using recently, not necessarily items that you've used a lot over your entire career. If you've been running infantry or another horse recently, then your champion courser smith skill will have taken a hit.
As an example, since the Deli Cap came out, I eschewed my normal equipment load-out for a joke one, with a deli cap and a dark cavalry robe. As a result, I now 9 smith skill in my joke loadout, but only 1 or 2 smith skill in the serious loadout I had been using consistently before. I'm now level 31 and I hadn't planned on retiring this generation as my build is only really complete at level 32. However, this now means that my crafting skills are complete bogus.