I have noticed some strange behaviour in TMG SP1 with the "Web Proxy Filter" and wanted to try and find an explanation for it.
I have unbound the "Web Proxy Filter" from the HTTP protocol in TMG and then created an Access Rule which allows my machine out to the Internet using the HTTP protocol. My machine is a SecureNAT client and not a Web Proxy client.
I can access the Internet fine - however, if I look at the TMG logs I can see that all my HTTP traffic is still being processed by the WPF. The logs come from the WPF with all the information that WPF logs normally show (including URL etc).
I've also found that the Malware Inspection process still blocks things (eg if I try and download the Eicar virus it is blocked). This is despite the fact that the Malware Inspection section in the Access Rule is greyed out (which happens when the WPF is unbound from the HTTP protocol). The only way to turn off Malware Inspection is to re-bind the WPF to the HTTP protocol and then switch off the Malware Inspection option in the Access Rule (which is no longer greyed out after binding the WPF).
Given that I have unbound the WFP from the HTTP protocol, how can these things still be happening? My understanding of the WPF is that it has to be bound to a protocol definition before it will do anything. This is clearly not what I am seeing. What is the point of unbinding the WPF in TMG? It seems to make no difference. The "HTTP Filter" options disappear (just like they did in ISA) and the Malware Inspection page becomes "greyed out" like you would expect it to (as the Malware Inspection is a Web Filter which requires the WPF), but despite this the WPF is still intercepting and processing all HTTP traffic.
I have disabled NIS and Malware Inspection (at the higher level) to see if one of these being on was causing this issue but none of it helps. All HTTP traffic still passes through the WPF despite the fact that it is not bound to the protocol. Is this a bug?
I can work around this easily enough by creating a custom HTTP protocol on port 80 which I also do not bind to the WPF and this works fine. It just seems strange that I have to do this given that both protocols end up looking exactly the same (outbound, port 80, no filters) yet behaving totally differently. The issue I have described is not preventing me from doing anything - I am posting this just out of interest to see if anyone has an explanation or can confirm that they see this too. I am trying to understand how TMG works at a reasonably deep level and this behaviour is really confusing me. If the WPF intercepts HTTP traffic regardless of whether it is bound to the default HTTP protocol definition or not then why have the option to unbind it? What is going on "under the hood" which causes this? There must be more to the default HTTP protocol definition than what can be seen in the GUI, given that I get totally different results using what looks like an identically configured protocol (my custom HTTP definition).
I don't think this happened with ISA but I don't have a machine handy to test it. Something weird is going on. :)
< Message edited by Optic -- 11.Jul.2010 7:50:44 PM >
From: Southern California
I think you may be confusing the Web Proxy filter with the HTTP filter. They provide two distinct services for HTTP traffic. If you look on the 'Web Filters' tab under the System node in the TMG management console you can disable the filter there. Why you would want to do that, I don't know...but you can.
The HTTP Filter is a "Web Filter", which is an extension or plugin (not sure of the right term) of the "Web Proxy Filter" (which is an Application Filter, not a Web Filter).
I am not trying to disable the Web Proxy Filter in the System node (it cannot be disabled).
If you edit the HTTP protocol you'll see that it is bound to the "Web Proxy Filter". This means that all HTTP traffic is routed through the WPF even if the client is SecureNAT - I guess you could call it a transparent proxy. If you unbind the WPF from the HTTP protocol, TMG should go back to treating the HTTP traffic as pure port 80 rather than running it through the proxy engine (and all the Web Filters).
Microsoft even recommends doing this in one of the TMG technet articles if the server is not used as a proxy in any way. For example, you may have a separate TMG server which is used as a proxy and another TMG server used as an edge firewall - you may want to unbind the WPF from HTTP on the edge firewall as the HTTP traffic coming from your internal TMG proxy server has already been through a WPF and there is no need to have it processed a second time.
Anyway - regardless of all this, I am much more interested in what is happening and why it is happening. This isn't about whether it is something that should or shouldn't be done. I am merely trying to understand TMG at some kind of architectural level and this behaviour goes against everything I know about how TMG works and it would be great to get an explanation. If I unbind the WPF from the HTTP protocol then an Access Rule which allows HTTP should *not* be processed by the WPF - yet with TMG it still does get processed by the WPF.
This creates all sorts of unanswered questions (as per my previous post). :)
< Message edited by Optic -- 11.Jul.2010 7:52:37 PM >
I have not tried it on TMG, but I agree that by everything I understand, SNAT tcp/80 traffic should not be intercepted by the web proxy if that app filter is unhooked from the HTTP protocol. My first suspicion would be, can you *confirm* it's coming in as SNAT traffic and not proxy traffic with a wireshark/netmon.
Anyway, this is intriguing and I think I shall test it in my lab to see if I can repro your results.
To answer your question - I have not confirmed it is SNAT traffic by using wireshark/netmon but I am still 100% sure that it is SNAT traffic. There are a few reasons that I know this for certain.
1. Proxy is not set in browser, nor is automatic detection set and the firewall client is not installed. The PC I am using has no knowledge of the TMG server at all. 2. I actually have the web proxy listener (port 8080) totally disabled on all networks on the TMG server. Even if I did tell the browser to use TMG as a web proxy it wouldn't work. 3. Lastly, and most importantly - in my original message I said that if I create a new HTTP protocol definition (Custom HTTP) and use this instead of the built in HTTP definition then it works fine. The SNAT traffic is not routed through the WPF. It is only when I use the built in HTTP protocol definition that I see this strange behaviour (even though the built in HTTP protocol is *not* bound to the WPF so in theory it should be exactly the same as my custom definition).
I can't seem to repro this issue. My methodology was the following:
Entered the script:
Set the Alerts behavior for the IsaScript-Information alert to "immediate" so I'd see each one instead of only one per minute
Wait for configuration to sync
Browse from a remote SNAT system
Confirmed SNAT traffic with wireshark
Refresh alerts, see "ding!"s
Confirm web proxy by noting "Via" header in outbound traffic (also with wireshark)
Remove "Web Proxy Filter" from the HTTP protocol
Wait for configuration to sync
Reset all alerts in TMG monitoring
Reload browser on client machine
Refresh alerts, no new "ding!"s were found, indicating no proxy inspection
Inspect outbound traffic, no "Via" header found, indicating no proxy inspection
During your test did you have a config sync error perhaps, or maybe didn't wait long enough for configuration to sync? Lame, I know, but honestly it's the only thing I can think of. I've seen that happen often enough to at least ask the question.
Apart from that, if your system demands to inspect HTTP even when it's unhooked, it sounds like a PSS case is in order. At least you have a workaround, but if the system is not doing what you tell it, that would be a cause for concern to me.
Since I don't work at MS that's as far as I can take it. If I had access to your TMG server I'd try the above procedure to see if anything illuminating could be found. IsaScript has a free evaluation, that would enable you to do exactly what I did.