I don't think XP or gold gain should be changed. The way it's currently done rewards teamplay the most, which is absolutely necessary in a game like CRPG.
That said, I like the idea of penalizing TKs with gold. Say, 500 - 1000 gold per TK. That would make people take a lot more care with their swings and their aims.
A system such as this would have huge advantages over reflective damage:
1. Gameflow does not get diminished. With reflective damage people would get too careful, thus robbing a battle of its momentum.
2. Players get penalized, not teams. It often happens that people newer to the game don't mind their surroundings and walk into the line of fire of a ranged fighter just as he is about to fire. Result: damage gets reflected, ranged player might even die. Team is robbed of better player. Not good for team.
3. Game stays fun. Accidents happen, we all know that. Most of us have TK'ed someone accidentally even though we were being careful. This way, if a TK happens, the player get's penalized in a way he decidedly feels, but it allows him to continue playing.
1000 gold is enough for everyone to feel the pain without it being a crippling penalty. It's low enough that only repeat offenders really feel the burn and those who accidentally bludgeon or skewer a friend realize they have to be more careful next time.
