It's absolutely ridiculous to try to say I can't make that if I want to.
Unless I'm saying you suck at making it, so you'll fail.
It's really the same argument, in effect. You took a part of our culture? You're not able represent it properly, because you aren't us!
It's somewhat comparable to some shill homeo. healer, that didn't go through medschool and an actual professional calling him out on it.
Culture is not a service, though, so it kinda falls on its ass there. Maybe if taxes were dependent on the quality of (portrayal of) your culture then it'd make some sense, but all you really have is socialising. By taking a part of another culture and molding it, you are taking a social privilege from a group of people.
For example, if there is a hand-sign, used by a group of people when making a grocery store transaction - you get wise to it and use it, you've now unlocked an emotional response that is supposed to be only available to this ingroup.
So, if you own a drum used to invoke the spirits or whatever(i honestly have no idea), then you're now able to do magic that's only normally available to a Sami. But because you aren't a Sami, that means you'll just mess up the magic and probably for everyone else too.
So I think that the argument shouldn't be: It's just an instrument, a simple object, why care so much?
But maybe instead: Motherfucker, I'm a wizard too and you disrespect my magical abilities by saying I can't appropriate and properly use this drum, in fact, I'll transcend whatever magic you've created with it, just wait.
Then if they continue the harassment, just cast a curse on them.
That way you've either convinced them or you've reduced the whole argument to a complete and utter absurdity, which is imo better than trying to argue with common sense in this case.