Does it perhaps have to do with the fact that the king was murdered by his own brother? And that so quickly afterwards the Queen was already married? Or maybe the King's corrupting influence upon Denmark.
It's not clear what definition you are looking for, you say the real life reference but what is that supposed to mean? Historically it would refer to the corruption of the royal house, how a Monarchy is the epitome of corrupted, violent, selfishness. The greed portrayed in the murder of King Hamlet, by his own brother. Who then took the throne and only two months later married the Queen.
Therefore, I can only conclude in this post that the phrase "There is something rotten in the state of Denmark" is meant to convey corruption.
tl;dr There is an injustice being perpetuated in the Danish Royal House.