While delaying is not permitted the attackers were sieging a castle no? In many games if the attackers take too much they have to retire cause they ate all food, losing moral, etc
I don't really see the point of kicking the defenders when they are just defending the castle, it should only apply to attackers that delay not vice verse.