If heirlooms make such little difference in fights, then 1) why did you get them in the first place, and 2) why do you care if you lose them?
They add a role-playing element thats been steadily disappearing with each patch and its nice sense of yes I've heirloomed my favorite equipment and its unique to me and my character and therefore different from every other guy using the same exact weapon or the same armor. My heirloomed body armor is far from the best (mameluke mail) but it allows me have armor geared toward middle-eastern themed character, yet to have closer to the same armor as all the unoriginal black armor/gothic plate and armet users min/maxing. It didn't give an advantage but put me on closer to even footing with people that are not role-playing with their armor and equipment. I used the Miadao and Daodao (chinese swords) because they are the closest looking to middle-eastern curved swords among the 2hed swords, I use the khergit/strong bows because they look more middle-eastern then warbow/longbow. Heirlooming adds just enough to not sacrifice role-playing for functionality.
However, xbows/bows is the one area where heirlooming makes a huge difference, not as OP, but with all the continual nerfs to ranged, to make them viable classes against the increasing assortment of polearm/2hers/1hersshiedlers.
I am 2h/archer right now with no plans to retire and many of my heirlooms I planned on trading away because they don't fit this class or my middle eastern style, but with a heirloom theft I would be only heirlooming a bow and a set of arrows (have to if want viability) and mail gauntlets (go with any armor) and much of the connection with my theme is lost where it doesn't really matter which armor or 2her I use. Its no longer my unique masterwork miadao with lordly mameluke mail, its just another piece of inferior armor and weapon I can choose to use or I can buy gothic plate with a winged helm and not gimp myself for the sake of role-playing.
Think of it this way, you saw so many people using side sword/military pick/elegant poleaxe/danish greatsword pre-patch, I never heirloomed those things even though they were the current OP theme, rather I heirloomed non-OP weapons that fit my theme so they could stand on closer to even footing with the unbalanced weapons. And to say just make all weapons balanced is not possible, already people are shifting toward Great Long Axes, Bec de Corbins, and Highland Claymores with the new patch. Also, one of the reasons I am against a respec too because people who heirloomed based on role playing are not going to change their triple-heirloomed stones, only people who heirloomed FlavorOfTheMonth weapons will switch heirlooms and it will be the new FOTM.