Just like to point out that giving a buff to matching sets does, in fact, penalize people not wearing matching sets. (They will be, relatively, weaker.)
Also, this would be impossible to implement for 95% of the armor, as there is no specific item to go with it.
Besides, why do you care what horrible outfit someone ELSE wears? Not like it's on your character. Sure, someone like Thulsa_Doom may look eye-bleedingly awful, but just let that make your character look better by comparison. You can't force RPing.