I feel really guilty when I teamkill someone, simply because I know that perhaps they would have helped win the match, and because I know the person behind the computer is very angry at me.
A penalty of some sort could help, I remember when you were really encouraged to watch your swings otherwise you'll lose a bunch of money. I even thought it would be funny if the cash you lost went to the person you TK'd for compensation, though I can see how that would be abused...
Maybe if you teamkill one of your items breaks? You'd have to pay money to repair it, which is pretty much the same thing. However, people with expensive gear would probably suffer a whole lot from that kind of system, and they already have to deal with the high repair cost from it breaking normally.