I have been given the task to setup OWA and RPC over HTTP from my Exchange 2003 SP2 server to the internet. After choosing ISA Server 2006 to publish my single Exchange server and spending a few days trying to learn how to get this working internally, I have made some strides but am now stuck. I am able to get RPC/HTTPS working directly to the Exchange server on the internal network, but not via the ISA 2006 Server. OWA has been working internally for some time, but after I login and click submit via ISA, I get “Error Code: 500 Internal Server Error. The request was rejected by the HTTP filter. Contact the server administrator. (12217)”. I have my listener network setup for internal and external so I am able to test from the internal network. Since this is still in the testing phase, I am not actually testing this from the internet and there is no DNS out there for the certificate I am using. What I am doing to make DNS match up with the certificate is to edit my local hosts file to point to either the Exchange or ISA server depending on how I’m testing. If I simply change my hosts file for the certificate hostname to point to my Exchange server, I get HTTPS showing up in my Exchange Server Connection Status window in Outlook. If I change it back to point to my ISA server, it shows TCP/IP meaning it does not connect via RPC/HTTPS. I have also been testing this by placing my PC directly connected to the external interface of the ISA server and I get the same message trying to connect to OWA and I cannot connect at all to Exchange via Outlook.
I believe I have followed the instructions in Mr. Shinder’s articles “Publishing Outlook Web Access and Outlook RPC/HTTP with ISA Server 2006 Enterprise Edition (RC) Firewalls using Forms-based Authentication (Single Member Array without NLB)” and I will continue to step through them to try and find a misconfiguration. I thought I would go ahead and post this message to see if anyone had any suggestions with the symptoms I have. The only discrepancy between the article and my experience is when you run the Publish Exchange Web Client Access wizard and choose both OWA and RPC/HTTP(s), it does not create two rules for this. Since the HTTP filter settings should be different for each of these, I created separate rules for each of these, applying the appropriate web filter settings for each rule.
I am currently testing with the 180-day trial version of ISA 2006 Standard with the intention of purchasing a license after the testing is successful. Unfortunately, I do not have access to a dedicated test server, so the 2003 SP1 Enterprise machine I am using has some other services running on it such as Virtual Server 2005 R2, McAfee VirusScan Enterprise + Anti-Spyware Module 8.0.0, MS Automated Deployment Services (ADS), MS VSMT 2005. I thought I’d mention this in case there are some other services that could conflict with ISA.
You know, I skipped article 3/4 in that series. I went ahead and started reading it and found my answer. Since I don't have DNS setup for my Exchange domain name (not easy in the corporate AD structure I'm in), I mentioned I put a hosts file entry in my PC. Running the BPA tool it told me right there that I needed to put a hosts file entry in my ISA server as well. As soon as I do that, voila! RPC/HTTPS works through the ISA server!
Note to others reading this: read ALL 4 articles on this start to finish!
I still have an OWA problem. The BPA tool tells me "ISA Server was unable to resolve the DNS name [insert FQDN here]. Requests that use the Web publishing rule OWA may be denied, or the response time may be slower than expected." The hosts file didn't fix this, so I wonder if OWA will simply not work without a successful non-hosts file DNS query. Time to finish reading article 3/4 and researching the BPA error...
I used the article on configuring the HTTP filter for OWA that Mr. Shinder referred to in article 4/4. In the Extensions tab, .exe is listed, but I noticed it had a note by it that you may not want to include it. It says
Note 1 Blocking .exe file extensions and enabling Block responses containing Windows executable content for Outlook Web Access will block access to the S/MIME control. If the S/MIME control is required for Outlook Web Access on Exchange Server 2003, do not include .exe in the blocked extensions list or enable Block responses containing Windows executable content.
I removed .exe from the list, applied my changes, and voila! OWA works!
So, for reference to others going through the same process, here were my main hangups (experts, if there are better ways to do what I did, PLEASE feel free to chime in, you WILL NOT hurt any feelings):
1. Being brand new to any version of ISA, I didn't realize I needed to create a Network Rule along with my Firewall Policy Rule to allow traffic to pass from the External to Internal networks. Previous firewall products I've worked on didn't have this type of step, so this stumped me for a day. Make sure you create a Network Rule with the Source Network as External and Destination Network as Internal along with the Rule.
2. The rest of my problems existed because of my lack of a DNS structure. Being only a Domain Admin and not an Enterprise Admin and being unsure of what DNS authority I had, I feared creating a internal zone file for this test domain certificate I had available to me. After realizing there was a problem here, I created an entry in my PC's hosts file to take the place of a true DNS entry. This allowed me to properly send requests to either the Exchange server directly (if I was on the internal network) or to the ISA Server (internal or external).
This is where I was at when I made my first post above. The last two problems were that I didn't read part 3/4 in this series by Mr. Shinder and didn't add a hosts entry on the ISA Server to point to the/an Exchange server on the internal network and that you shouldn't put .exe in your blocked extensions list for your HTTP Filter on the OWA Rule.
You ever notice that after you post something to a forum, the answers just start coming to you quickly and you get it figured out shortly? Maybe I should just start doing that from now on and I'll get these projects done more quickly.