dude, do it! I myself am a competitive beer drinker, and trust me, nothing blasts you away better than a hard hitting training session with the guys!
also on a serious note: do it. sword fighting is a lot of techinque and finesse, boxing is a LOT heavier on the strength/conditioning part. which is a bit lame from time to time, but just maybe you will find a few cool guys to workout with, which makes everything easier.
also next time you go hunting you can punch the dead moose into the face with a lot more effectiveness.