Not a bannable offence. Annoying and I would probably ban him if he did this consistently because it would fall under the "arsehat" rule.
It's like a gentleman's agreement, unspoken but should always be honoured. You challenge someone to a duel and you expect it to continue on through until one is victorious.