Welcome to ISAserver.org

Forums | Register | Login | My Profile | Inbox | RSS RSS icon | My Subscription | My Forums | Address Book | Member List | Search | FAQ | Ticket List | Log Out

3GB of traffic over 2 days - SMTP denied - is ISA causing a mail loop?

Users viewing this topic: none

Logged in as: Guest
  Printable Version
All Forums >> [ISA Server 2004 Firewall] >> General >> 3GB of traffic over 2 days - SMTP denied - is ISA causing a mail loop? Page: [1] 2   next >   >>
Login
Message << Older Topic   Newer Topic >>
3GB of traffic over 2 days - SMTP denied - is ISA causi... - 22.Mar.2006 4:17:55 PM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		Hey everyone (Tom, I really, really hope you read this one!),

Over the past 6 weeks we've had major e-mail problems on our network.  This is going to be a long post so I'll split it into several parts:  To begin with I'll describe the symptoms we've been seeing, I'll then go over our network topology & the steps we've taken in troubleshooting this.  I'll then go over what we've discovered in our troubleshooting before finally giving my current theory on what's happening in the hope that somebody reading this can confirm or deny it.

Now we've been working with Microsoft's Exchange Server support team for about 6 weeks on this, we've just recently passed the case onto their ISA team, but they're already at the stage that they want netmon traces and that's going to be incredibly difficult.  The problem I suspect takes about an hour to replicate (if I can) and it's so intermittent it happens about once every week to ten days.  Short of installing a dedicated server for netmon traces this is going to be difficult, so I'm coming here for a 2nd opinion (and then I'll build that server).

Symptoms of the Problem
  • Our mail server starts generating a backlog of messages.
  • Under Exchange server message delivery is failing with "The connection was dropped by the remote host."
  • On a MIMEsweeper appliance message delivery fails with "I/O error".
  • (Yes, we swopped mail servers while troubleshooting this)
  • Small messages can send ok, but any large message will jam up.
  • Once you've sent a large message to a domain, further messages to the same domain just get stuck in the queue.
  • The firewall server starts logging denied SMTP packets from multiple servers with error code:
    0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED
  • We have no other problems on the network.  Internet access works find and I can establish a telnet session to the affected mail servers.
  • The problem occurs every week to ten days, other than this we have no problems with mail, we routinely send hundreds of MB's of mail a day and receive similar amounts.


Our Network - route taken by E-mails

1. SMTP Mail Server (Exchange or MIMEsweeper)
        |
2. ISA Server 2004 Firewall
        |
3. Q-Balancer (QoS and Load Balancer)
   Restricts outbound SMTP to 128kb/s
        |
4. DSL Router
        |
5. Remote SMTP server (recipients server or smarthost)

We tried rebooting all the devices, starting & stopping services, pretty much everything we could think of.

In the end, we had to eliminate items one at a time, leaving ISA for last:
1.  Two different SMTP servers exhibit the exact same symptoms.
2.  ....
3.  Bypassing the Q-Balancer entirely has no effect
4.  We've tried two different DSL routers, both known to work perfectly.
5.  Multiple domains are affected, including big names like hotmail.com, yahoo.com.
    Changing delivery to a smart host also has no effect.

However:
2.  Moving the mail appliance outside the ISA server immediately solves the problem (we plugged it directly into 4. the DSL router).

ISA has been working for 2 years without ever causing any kind of problems.

ISA Configuration
Windows 2000 Server, SP4.
ISA 2004, fully patched with the exception of SP2.  (Some of the problems reported here with SP2 will affect us directly so I have not even considered installing this yet.)

What we have found so far
  • Moving the MIMEsweeper appliance outside the firewall solves the problem immediately (we moved it to plug directly into the DSL router).
  • Running windows update on the firewall server didn't seem to have any effect.
  • Since installing the MIMEsweeper appliance we were able to access reports on mail traffic, we discovered that when this problem hits we are sending 2.9GB of traffic to a single e-mail address over a 24 hour period.  This is our first clear indication of the cause of the problem.
  • We checked our logs and discovered that the problem starts immediately we send large mails to this address.
  • (In this case we were sending fifteen 3MB e-mails.)
  • Reading up on mail loops, I found two possible causes:
    • a forwarder sending the same messages back and forth between two servers
    • the sending server not receiving the final 'OK' response from the receipient server

  • Checking our mail logs, we are not receiving large amounts of mail, so I suspect the 2nd option.
  • This could tie in with the "TCP_NOT_SYN" message on the ISA server - if the final OK is being blocked our mail server will simply keep trying to send messages.
  • We have found out that this particular recipient had an infinite mail loop, their e-mail filtering company has reported that they saw over 200,000 messages flowing from their server on that day.
  • Once delivery has failed to this domain, all large e-mails to any domain fail.
  • Some mail does appear to flow, our message counts and queue size will vary by up to 3-4 messages and 1-2MB throughout the day but the general trend is always upwards.  (I believe this is small messages being transmitted ok).

Some Figures
  • Our mail server allows up to 50 concurrent connections.
  • At the peak of the problem we have upwards of 200 messages quueed, totalling over 150MB.
  • Those 15 original messages reduce the outbound transfer rate to 8.5kb/s.
  • This means it will take 48 minutes to transmit these messages assuming no errors occur.
  • During those 48 minutes, many other messages will arrive and I believe we quickly hit the 50 connection limit.
  • 50 concurrent connections drops the speed per message to 2.56kb/s.
  • That means a 1MB message will take 53 minutes to send.  The original 3MB messages will now need 2hrs 40 mins.

As far as we can tell, these kinds of delays are pretty normal for our mail servers and they cope fine normally.
Even with 200 messages in the queue, once we move the server outside the ISA box the queue drops fairly rapidly, matching the peak transfer rates we would expect on this connection).

My Theory
  • The load on the original recipient mail server could conceivably delay the time it took for their server to send that final 'OK'.
  • Standard SMTP servers appear to handle this fine (we have no problem when mail server placed outside ISA)
  • I think ISA may however have dropped the connection due to this long delay, causing it to block that final 'OK'.  That would cause the mail server to retry these messages (and repeated attempts would add up to 3GB over 24 hours).
  • After this problem has happened once, I think ISA also begins dropping any other connections that have been open for a while.  At this point that will include large e-mails to any domain - exchange will have between 15 and 50 connections open, and will will be wanting them kept open for a minimum of 30 minutes, probably nearer 60.

What I would like to know
  • Could ISA be blocking the packet in this way?  Microsoft are adamant that ISA will not cause this, however after the chaos caused by their Exchange team I am not inclined to simply take their word for it.
  • Many, many people have reported the TCP_NOT_SYN error but nobody seems to have a definitive answer.
  • I have seen one report of almost exactly the same symptoms - SMTP connections dropping with TCP_NOT_SYN errors from multiple domains:
    (see http://www.mcse.ms/archive99-2004-11-1007098.html) 
    If you have seen or are affected by this problem, please let me know.
  • If ISA is blocking this packet, what then causes the problem with the other domains?  When this problem occurs we see that same TCP_NOT_SYN error for multiple domains, usually we have no problem with any of them.  Could there be a problem with ISA causing it to drop all these other connections once this error has occured?


If anybody has any ideas or theories on this, please let me know.

Ross
Post #: 1
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 23.Mar.2006 4:19:11 PM   
tshinder

 

Posts: 47490
Joined: 10.Jan.2001
From: Texas
Status: offline 		Hi Ross,

First, let me compliment you on an excellent description of the problem!

Its interesting that you only see the problem when the MIMEsweeper is behind the ISA firewall, since I assume that when the Exchange Server forwards the connections, it must go through the ISA firewall to reach the MIMEsweeper, when the MIMEsweeper is in front of the ISA firewall. Is that right?

Of course, if the ISA firewall is timing out state table entries because the destination SMTP server is slow in repsonding, that would make sense.

A packet trace taken on the ISA firewall would really help a LOT. It should be standard practice to install Network Monitor on the ISA firewall. Is it possible to start a trace when you notice this is happening?

Thanks!
Tom

_____________________________

Thomas W Shinder, M.D.
Sr. Consultant/Technical Writer
Prowess Consulting http://www.prowessconsulting.com/
Blog: http://blogs.isaserver.org/shinder/

GET THE NEW ISA 2006 Book!: http://tinyurl.com/2gpoo8

(in reply to myxiplx)
Post #: 2
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 23.Mar.2006 4:32:33 PM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		Heh, if you'd been working on it for as long as I have, you'd be able to describe it well too ;-)

I should maybe point out that when we moved the MIMEsweeper appliance in front of the firewall it had no link to Exchange :)  We simply changed it's IP address & plugged it into a different point (we were actually thinking that in the worst case scenario we could stick MIMEsweeper behind the firewall, pick up a load of mail & then move it back outside the firewall to get it going - a very manual way of routing mail, but it would have worked ;)).  By that point we'd actually already stopped the SMTP service on Exchange since the constant flow of new mails was hampering our analysis of the traffic that was actually flowing through the MIMEsweeper queue.

Also, this happened exactly the same when we just had the Exchange server.  The benefit of the MIMEsweeper appliance was that we felt secure enough to try moving this in front of the firewall.

From what you're saying about ISA timing out state table entries it sounds like my guess could be spot on.  Do you have any idea how long this timeout would take, or any details on the exact process taken?  I've suggested this to Microsoft but they flat out denied that ISA would ever drop a connection.  They're telling me that one of the two servers would have to have dropped the connection first.

I'm trying to get netmon traces right now but the problem is that since we've notified the original recipients of the problems with their server they have taken steps to resolve the problem.  That means there's nowhere near the backlog on that server as they had before and we're able to send mails to them now without the problem happening.

I'm still very keen to get to the bottom of this however.  It's had a huge impact on our mail traffic to dozens of destinations over the past 6 weeks, and the time spent on it has been horrendous.

Ross

< Message edited by myxiplx -- 23.Mar.2006 4:34:01 PM >

(in reply to tshinder)
Post #: 3
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 23.Mar.2006 4:38:13 PM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		Hmm... found this article on the FW-1 state table:
http://www.windowsecurity.com/whitepaper/Understanding_the_FW1_State_Table.html

It seems that FW-1 has a 60 minute timeout on it's state table so I'd guess ISA has similar.  Now, is this timeout updated with every packet, or just when the connection is established?

There's a very strong possibility that some of these mails were taking over 60 minutes to send...

(in reply to myxiplx)
Post #: 4
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 23.Mar.2006 5:40:46 PM   
tshinder

 

Posts: 47490
Joined: 10.Jan.2001
From: Texas
Status: offline 		Hi Ross,

You could check the ISA firewall's state table using fwengmon. Have you heard of that utility?

Thanks!
Tom

_____________________________

Thomas W Shinder, M.D.
Sr. Consultant/Technical Writer
Prowess Consulting http://www.prowessconsulting.com/
Blog: http://blogs.isaserver.org/shinder/

GET THE NEW ISA 2006 Book!: http://tinyurl.com/2gpoo8

(in reply to myxiplx)
Post #: 5
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 23.Mar.2006 5:55:25 PM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		No, haven't heard of that, but am downloading it now :).

The problem I have is that I don't believe I have any way to replicate this problem any more so I don't think any kind of monitoring tool will help at this stage, I really have to try to get an answer to this with the information currently to hand.

My guess is that we need a slow outbound mail connection to a remote server where that server is under enough load that there's a noticable delay before it can send the final 'OK' message back to our server.  I believe that happened here with a remote server under considerable strain, with us sending large attachments, and with their server running anti-virus scans on all incoming messages.

Without that situation to start the problem I don't think I'm going to be able to get any specific traffic logs.  All I can do is go by what I've seen so far & come up with a guess as to what happened.  So far the best theory is that ISA timed out the connection before the final 'OK' packet arrived.  It fits all the symptoms of the problem, I just need to know a bit more about how ISA could drop a connection to know if this is a feasible answer.

I believe the default delay for both Exchange and MIMEsweeper to drop an SMTP connection is 10 mins.  If ISA at any point can time out connections in a shorter time it would explain why e-mails worked outside but not inside the firewall.

Ross


(in reply to tshinder)
Post #: 6
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 24.Mar.2006 11:12:50 AM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		Just sent the following to the MS engineers I'm speaking to:


Just thinking about this, if ISA works in a similar way to other firewalls it's possible that this connection simply timed out in the internal state table.


This is based on the assumption that ISA behaves in a similar way to other firewalls but I cannot find any documentation to clarify this.  Do you have any details on how ISA manages the state table, and how long a connection would need to be inactive before it would be removed?


This would seem to fit the evidence of what's happened so far.

 - We have a long slow connection to a remote mail server.

 - At the end of that connection it's likely we have a considerable delay with no traffic flowing before the remote server responds, giving an opportunity for ISA to flag the connection as idle and close it.

 - We know that the remote response is then being blocked with a 0xC0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED


The error library on msdn states that the 0xC0040017 error means a packet is dropped because the source does not have an established connection.  We know from the mail server logs that we are sending traffic to these destinations and that we have no problems with the mail server outside the firewall.  That all seems to fit with the theory of the connection timing out in ISA's state table.


I've been told that the TCP_NOT_SYN error also crops up if ISA thinks it's under attack.  I assume it counters attacks by reducing the timeouts in the state table.  We were sending 15 messages to this slow server, would 15 of these errors be enough to trigger this?  Could this explain the knock on effect on the other mail servers?


If there are any flaws in this argument, please let me know.  If this does indeed seem to fit, I would appreciate somebody letting me know.  Either way, I would appreciate some details on ISA's state table, how it times out connections, and how it responds to perceived attacks.


Ross

< Message edited by myxiplx -- 24.Mar.2006 11:14:30 AM >

(in reply to myxiplx)
Post #: 7
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 25.Mar.2006 7:48:22 PM   
tshinder

 

Posts: 47490
Joined: 10.Jan.2001
From: Texas
Status: offline 		Hi Ross,

Its certainly possible that the connections are timing out of the state table -- that's why it would be very useful to check the entries in the table using the fwengmon tool when this problem is taking place. Without being able to replicate it the issue, I think it's impossible to figure out exactly what caused the problem.

Tom

_____________________________

Thomas W Shinder, M.D.
Sr. Consultant/Technical Writer
Prowess Consulting http://www.prowessconsulting.com/
Blog: http://blogs.isaserver.org/shinder/

GET THE NEW ISA 2006 Book!: http://tinyurl.com/2gpoo8

(in reply to myxiplx)
Post #: 8
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 25.Mar.2006 10:07:27 PM   
wbplomp

 

Posts: 138
Joined: 18.Nov.2004
From: Netherlands, The
Status: offline 		Hi Ross,

I have almost exactly the same problem. I don't have problems with the mailflow as you have, but I see a SYN_PACKET_DROPPED after every Initiated and Closed Connection between our internal (Exchange) and external (MessageLabs) mailservers. I have not opened a case yet; we are planning a network change, if the problem then still exists I will open a case at Microsoft. I'm looking for a solution for months now... but still no result.

One question... do you use Enterprise Edition with NLB too?

I will check this post every week...

Good luck!

Boudewijn

< Message edited by wbplomp -- 25.Mar.2006 10:09:52 PM >

(in reply to tshinder)
Post #: 9
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 26.Mar.2006 7:08:03 PM   
tshinder

 

Posts: 47490
Joined: 10.Jan.2001
From: Texas
Status: offline 		Hi Boudewijn,

Do you have any packet traces of this problem?

Thanks!
Tom

_____________________________

Thomas W Shinder, M.D.
Sr. Consultant/Technical Writer
Prowess Consulting http://www.prowessconsulting.com/
Blog: http://blogs.isaserver.org/shinder/

GET THE NEW ISA 2006 Book!: http://tinyurl.com/2gpoo8

(in reply to wbplomp)
Post #: 10
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 26.Mar.2006 8:03:59 PM   
wbplomp

 

Posts: 138
Joined: 18.Nov.2004
From: Netherlands, The
Status: offline 		Hi Tom,

When I'm back in the office I will try to create a packet trace with netmon. But I first have to figure out how to create what you are asking. I get these Denied Connections for a lot of traffic, but I'll focus on SMTP traffic for now.

Gr. Boudewijn



(in reply to tshinder)
Post #: 11
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 23.May2006 2:11:30 PM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		Ok, we had a repeat of this problem last Friday (May 19th).  This time I was able to get some network traces and I managed to get a few more details from Microsoft about ISA's timeout values for it's state table.

Yet again very little mail was flowing (any large mails were being denied and subsequent mails to that server would also fail).  Many errors in the ISA log of:
0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED

One curious thing I noticed (and pointed out to the MS engineer), the errors in the event log were not restricted to inbound traffic.  We were getting the packet drop message for *Internal* sources.  I asked the MS engineer why ISA would need to check the session table for outbound traffic - surely internal clients have every right to create a connection and the state table is purely to identify which internal client inbound traffic is destined for?

I did find out that the state table has a default timeout of 60 seconds.  If no traffic is sent from either party for 60 sessions the entry will be dropped from the table.  If ISA believes it's under attack it will reduce this value but I wasn't able to find out by how much.

Now just before the problem we had 2 unusual events on the network:
- router problems with our ISP preventing connectivity to many US sites (we don't usually send mails to the US however)
- a very large (probably 250MB) e-mail sent by one of our users

They may be unrelated, but it may be that e-mail (or an NDR as the mail bounced back to us) that started this off.

My suspicion is this:
- A large e-mail session has a processing delay somewhere along the line (quite possible with a virus scan of a 250MB message...)
- ISA drops the entry from the state table
- The mail servers then continue their transmission (and I believe this could be either our server or theirs)
- ISA now denies all packets since the session has timed out
- The mail servers continue attempting to send the mail (they're in the middle of SMTP transmission, I don't know how long they keep trying before giving up, but I do know that SMTP is designed to be robust and work over unreliable or intermittent connections)
- ISA keeps blocking these attempts, eventually enough are detected to trigger it's attack rules
- ISA drops the timeout on the state table to cope with the 'attack'
- Now *any* large mail with a slight delay will have its session dropped and mail transfer will fail
- We now have numerous servers repeatedly trying to send their messages, re-enforcing ISA's belief that it's under attack.

I believe the unique events to trigger this are:
- A mail system which regularly sends many large messages
  (we routinely have 150MB in our outbound queue with 10MB messages to half a dozen people)
- A relatively slow internet connection
  (256kb outbound, mail is restricted to only use half of that)
- SMTP configured to send multiple messages at once
  (our server was configured for 50)
- A single large message or slow server to trigger the initial response
  (I believe triggers were 250MB message this time, last time a remote server with a mail loop last time receiving a 10MB mail from us.)

This would explain:
- why we're affected when other users are not (we send a lot of large messages)
- why the problem starts with one message and affects most SMTP traffic
- why the problem goes away if we move the mail server outside ISA (the mail server is working fine and just clears it's backlog, ISA no longer percieves a threat and resets it's triggers, and the mail server then works as normal when back inside the ISA's defences).

To solve the problem this time I reduced the number of outbound SMTP connections to 3 and watched the mail queue slowly clear itself.  I cannot be 100% sure this was the solution but I'm probably 99% sure.  Microsoft were looking at the ISA server at the same time but their only change was to tweak the default external IP address (our server uses 2) which I don't believe would make a difference.

I'd be very interested if anyone can suggest whether this would seem to be a reasonable explanation for this problem.  I do have a copy of the ISA server's logfile and 10 minute packet traces while the problem was in progress if anybody wants to check any particular details.  The files are quite large though and don't compress much with winzip:

ISA log:  34MB
Internal NIC capture:  33MB
External NIC capture:  17MB

Ross

(in reply to wbplomp)
Post #: 12
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 8.Jul.2006 9:52:46 PM   
adenhaan

 

Posts: 35
Joined: 15.Jul.2005
Status: offline 		Ross, I have different yet similar issue and want to share some of my observations in hope they will trigger a brilliant mind into providing a solution, and apologise if i'm just confusing matters. 

Using Isa 2004 EE SP2 on 2 node cluster. Published SMTP server using "SMPT server" publishing rule.

Now all mailflow was running flawless *untill* I enabled TLS on the SMTP server... at that point inbound mail would still be received (including mail headers showing that secure TLS was used for delivery), however the sender would never get positive confirmation that the messages has been delivered, and as a result would go into re-try / re-send mode, causing mailboxes to fill up with duplicates of the same messages.

With regard to the denied connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED messages: they started showing up *after* I enabled TLS (none of them before that). Closer examination of the denied connection messages is that they would show the external IP of the published server as destination (instead of the internal IP) and SMTP as protocol (instead of SMTP server) and local host as the destination net (instead of internal net).

so from my pov, it is TLS that is somehow messing up the SMTP server publishing on ISA

Still desparately looking for a solution !

Andre.

(in reply to myxiplx)
Post #: 13
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 9.Jul.2006 2:29:59 AM   
adenhaan

 

Posts: 35
Joined: 15.Jul.2005
Status: offline 		Update on my situation:

On a hunch I disabled the SMTP filter on the SMTP publishing rule (what else running on ISA would cause connections to drop prematurely)  and... voila ! The number of denied connections while using TLS dropped to a minimum again, and senders of email are correctly notified of succesfull delivery so they will not go into retry mode.

Note of caution: I can live without the SMTP filter because we route all our inbound email through a hosted filtering solution and only allow access to our SMTP server from these datacenters... I wouldn't neccesarily recommend it if you do not have other defenses in place.

My personal conclusion: There is some bug in the SMTP application filter on ISA 2004 EE SP2.

(in reply to adenhaan)
Post #: 14
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 9.Jul.2006 5:47:44 PM   
tshinder

 

Posts: 47490
Joined: 10.Jan.2001
From: Texas
Status: offline 		Hi Aden,

That's interesting, because TLS is fully supported by the ISA 2004 firewall and I use it often.

Are the AUTH and STARTTLS verbs enabled?

Thanks!
Tom

_____________________________

Thomas W Shinder, M.D.
Sr. Consultant/Technical Writer
Prowess Consulting http://www.prowessconsulting.com/
Blog: http://blogs.isaserver.org/shinder/

GET THE NEW ISA 2006 Book!: http://tinyurl.com/2gpoo8

(in reply to adenhaan)
Post #: 15
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 10.Jul.2006 2:41:56 AM   
adenhaan

 

Posts: 35
Joined: 15.Jul.2005
Status: offline 		Tom,

quote:

That's interesting, because TLS is fully supported by the ISA 2004 firewall and I use it often.

I would have thought that TLS would be fully supported as well, so I'm curious to find out what is causing the issue in my situation.

quote:

Are the AUTH and STARTTLS verbs enabled?

Yes all 'default' verbs where enabled (including AUTH and STARTTLS). I did only enable anonymous access for the SMTP server authentication and I also configured smtp filter event alerts in ISA, but no alerts where reported.
I even turned filter back on, and can reproduce the situation quite easily.... as soon as the filter is turned on, the denied connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED (for those hosts that we allow to communicate with us = the 3rd party filtering service) go skyrocket, and after a while duplicate messages (re-sends) are starting to show up in mailboxes.

Do you have a recommendation ? Is there anything specific in a network trace that should show up regarding the acknowledgement of receipt of a message ?

Thanks ! Andre.

(in reply to tshinder)
Post #: 16
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 10.Jul.2006 9:50:03 AM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		Interesting since we don't use TLS at all here, but we do use the SMTP filter.  I assume you only get the problem using both TLS *and* the SMTP filter, and that either on their own work fine?

It's curious that you've got a very different situation but with some very similar symptoms.  For us the problem was with outbound, not inbound mail, was nowhere near as easy to reproduce and we had no reports of duplicate messages.  However we did send a *lot* of duplicate traffic to remote mail servers.  I also still have the network traces from the last occurance so if someone with more knowledge than I were interested in looking into this they might be worth comparing with Andre's traces. 

I've tried reporting this to MS but unfortunately they kept insisting that ISA was performing as designed, and would not accept this as abnormal behaviour.

My own theories are very much guesswork, but I'm going to ask a few quick questions based on them nontheless: How much load is there on your ISA & Mail servers?  Does TLS significantly increase the overheads for receiving mail?  Does the problem for you start immediately you turn on the filter, or do you get a few messages through before the problem starts?  (I'm 90% sure our problems started with large messages being delivered over a slow connection).

Ross

(in reply to adenhaan)
Post #: 17
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 11.Jul.2006 4:57:06 AM   
adenhaan

 

Posts: 35
Joined: 15.Jul.2005
Status: offline
Ross,

Curious indeed that both situations seem related to the final confirmation of receipt of message getting lost (because ISA SMTP filter drops connection prematurely ?) - in your case for your outbound messages, in my case for my inbound messages.

Here are the answers to your questions;

Yes, the problem shows up using both TLS and SMTP filter. (Had no issues without TLS while using SMTP filter)
The problems occured during weekend, with verry little load on server (50 messages / hour).
I have not measured the overhead of TLS - Both ISA and Exchange servers aren't really breaking a sweat.
The problems start almost immediately (even though some messages did not seem to go in re-try)... within an hour or so the rate went up to 2000 messages per hour due to re-sends.
My connection is 6Mbit to 3rd party datacenter.. so I'm not suspecting slow connection or slow SMTP server on other end to be the trigger, I did not track message sizes.

Another thing I observed is that the number of denied connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED roughly correlates with number of messages going though system. It might be a 'standard' thing that happens regardless.

I might just take the time to log a case with MS, see if I have more luck than you did...

Andre.

(in reply to myxiplx)
Post #: 18
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 11.Jul.2006 11:52:39 AM   
myxiplx

 

Posts: 132
Joined: 16.Mar.2001
Status: offline 		If you do raise the case with MS, it may be worth referencing my original case and letting them know I have network traces for a very similar situation.  Original case ID was: SRQ 060 209 600 707.

Your comment about this error being a standard thing got me curious, I just had a look at my logs and found you may be right.  Whether or not the SMTP filter is active I get numerous errors in the log, apparently for incoming *and* outgoing mail.

So, is this error is a red herring and something that occurs normally, with the true cause yet to be identified?  Or do both of our systems have some kind of problem causing this error to occur?  Is there a fundamental clash between the way ISA interprets SMTP communications and the way a standard SMTP server uses them?

Here's a selection of the log file, I've edited it to group the various smtp conversations, but otherwise it's just a random sample from our logs with the SMTP filter turned off.

11/07/2006 10:15 213.239.246.149 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Closed Connection 0x80074e21  Exchange Server SMTP Out (ROB-026) 192.168.1.24
11/07/2006 10:14 213.239.246.149 25 SMTP Initiated Connection 0x0  Exchange Server SMTP Out (ROB-026) 192.168.1.24
 
11/07/2006 10:14 213.239.246.148 25 SMTP Closed Connection 0x80074e21  Exchange Server SMTP Out (ROB-026) 192.168.1.24
11/07/2006 10:14 213.239.246.148 25 SMTP Initiated Connection 0x0  Exchange Server SMTP Out (ROB-026) 192.168.1.24
 
11/07/2006 10:14 217.204.246.117 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 194.152.74.225
11/07/2006 10:14 217.204.246.117 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 194.152.74.225
11/07/2006 10:14 217.204.246.117 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 194.152.74.225
11/07/2006 10:14 192.168.1.24 25 SMTP Server Closed Connection 0x80074e20  ISA22: SMTP Mail Server 194.152.74.225
11/07/2006 10:14 192.168.1.24 25 SMTP Server Initiated Connection 0x0  ISA22: SMTP Mail Server 194.152.74.225
 
11/07/2006 10:14 38.119.91.41 25 SMTP Initiated Connection 0x0  Exchange Server SMTP Out (ROB-026) 192.168.1.24
 
11/07/2006 10:14 217.204.246.117 25 SMTP Denied Connection 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED - 212.135.6.13
11/07/2006 10:14 192.168.1.24 25 SMTP Server Closed Connection 0x80074e20  ISA22: SMTP Mail Server 212.135.6.13
11/07/2006 10:14 192.168.1.24 25 SMTP Server Initiated Connection 0x0  ISA22: SMTP Mail Server 212.135.6.13
 
11/07/2006 10:13 213.161.84.126 25 SMTP Closed Connection 0x80074e20  Exchange Server SMTP Out (ROB-026) 192.168.1.24
 
11/07/2006 10:13 192.168.1.24 25 SMTP Server Closed Connection 0x80074e20  ISA22: SMTP Mail Server 66.127.32.50

There's obviously a lot of packets being dropped here.  I can't see this many SMTP servers sending random packets for no good reason so I guess the questions to ask are now:
- What are the packets that ISA is blocking here?
- Why is ISA closing the session when multiple brands of SMTP server obviously expect to continue talking to each other.
- Under what circumstances do the loss of these packets affect the sending & receiving of mail between these SMTP servers?

Ross

(in reply to adenhaan)
Post #: 19
RE: 3GB of traffic over 2 days - SMTP denied - is ISA c... - 11.Jul.2006 7:34:17 PM   
simonhill

 

Posts: 21
Joined: 1.Mar.2005
From: UK
Status: offline 		Did anyone ever get an answer to this problem from Microsoft?  I have a very similar issue, and have not been able to troubleshoot myself.

Simon.

(in reply to myxiplx)
Post #: 20

Page:   [1] 2   next >   >> << Older Topic    Newer Topic >>
All Forums >> [ISA Server 2004 Firewall] >> General >> 3GB of traffic over 2 days - SMTP denied - is ISA causing a mail loop? Page: [1] 2   next >   >>
Jump to:

New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts