This message in my logs has been irritating me for a while because I don't know if I should do anything about it. I spent some time analyzing smtp transactions that got the connection denied log entry with this reason code. Using both a hardware sniffer and Microsoft's Network Monitor , this is what I found.
First of all, the message time can, and often is, well after any event associated with the session or transaction. Some mail transactions completed in milliseconds while this message was written sveeral seconds after the last segment had been processed. The hardware trace verified that the MIcrosoft Network Monitor was correct and there was no traffic when the log was written.
What this message seems to indicate is that the reset bit (RST) occurs in a segment without an accompanying ACK. Sometimes these are found after the FIN-FIN segments indicating end of session, sometimes they occur before the transaction handshake, and sometime the mail server sending them seems to wqant to start over and does it by sending segments with RST set.
Since I can't correlate the time I can't be 100% on this, but sesions with 11 reset segments eventually resulted in 11 logged "denied connections." Sessions without any solo RST flags did not have any issue.
In no case was any mail lost so it seems to be mail servers that don't do things totally by the books but they do follow RFC 793 which allows, for example, sessions to be ended by reset flags.
I am going to ignore them unless any of you suggest further analysis.