issue with ISA 2006 EE CSS servers and NLB (Full Version)

All Forums >> [ISA 2006 Firewall] >> Network Infrastructure



Message

kas -> issue with ISA 2006 EE CSS servers and NLB (10.Jun.2008 10:19:22 PM)

although we are not new to ISA, we are new to ISA 2006 EE.    we are installing  ISA 2006 EE as a new install, with primary and alternate CSS servers,  and one ISA array (so far), with two ISA servers in this first ISA array.  

we have configured IntraArray communcations between the array members to use a non-nlb network segment separate from the internal network segment.  IntraArray communications are working successfully.

prior to our having enabled ISA integrated network load balancing on the Internal adapters of the ISA array,  the CSS servers showed both array members as active in the ISA adminstrator console.  

ISA logs show the entries listed at the end of this post after the statement ***** before nlb isa log ******* for this successful connection


After ISA Integrated Network Load Balancing is enabled on the Internal network adapters of the ISA array, only one of the array servers shows active in the ISA administrator tool on the CSS servers. The other shows the error message "unable to retrieve data from 'servername'.

ISA logs show the entries listed at the end of this post after the statement **** after nlb isa log **** for this unsuccessful connection

HOWEVER
 

1. ISA configuration changes made on the CSS server are still replicated to both array members


2. all functions of both members of the ISA array that have been tested to-date are still working successfully..including network load balancing


3. in the ISA administrator tool of the array members themselves, both array members show active


and perhaps the most interesting symptom of all:

4. the problem of the one of the array member servers not showing active in the ISA administrator tool of the CSS server will flip from one array member to the other depending on the order in which ISA firewall services are started on the array members


Note that based on the error in the isa logs, an ISA network route rule was added between the internal and intra-array network segment in an attempt to work around this error.

This resulted in the application event log error 21215 from the microsoft firewall service (short description: route rules cannot be added for networks where one is network-load-balanced and the other is not). This network route rule has since been revoked.


A high-level description of these ISA servers,  followed by a detailed description of the network topology/IP addressing,  is listed at the end of this post after the statement  ***** isa server description *******.    the external ip addresses are obfusciated, all other ip addressing info is accurate.  

Microsoft support has so far been unable to provide any assistance in resolving this issue.   Thank you in advance to anyone who responds to this post with any suggestions/recomendations. 

*** before nlb isa log ***

Original Client IP Client Agent Authenticated Client Service Referring Server Destination Host Name Transport HTTP Method MIME Type Object Source Source Proxy Destination Proxy Bidirectional Client Host Name Filter Information Network Interface Raw IP Header Raw Payload GMT Log Time Source Port Processing Time Bytes Sent Bytes Received Cache Information Error Information Authentication Server Log Time Client IP Destination IP Destination Port Protocol Action Rule Result Code HTTP Status Code Client Username Source Network Destination Network URL Server Name Log Record Type

10.100.10.213    -  TCP - -      -    6/10/2008 8:53:39 PM 1610 0 0 0 0x0 0x0 - 6/10/2008 1:53:39 PM 10.100.10.213 172.31.10.2 3847 MS Firewall Control Initiated Connection [System] Allow remote management from selected computers using MMC 0x0 ERROR_SUCCESS   Internal Local Host - ISA02 Firewall

10.100.10.213    -  TCP - -      -    6/10/2008 8:53:41 PM 1626 0 0 0 0x0 0x0 - 6/10/2008 1:53:41 PM 10.100.10.213 172.31.10.1 3847 MS Firewall Control Initiated Connection [System] Allow remote management from selected computers using MMC 0x0 ERROR_SUCCESS   Internal Local Host - ISA01 Firewall

**** after nlb isa logs ****

Original Client IP Client Agent Authenticated Client Service Referring Server Destination Host Name Transport HTTP Method MIME Type Object Source Source Proxy Destination Proxy Bidirectional Client Host Name Filter Information Network Interface Raw IP Header Raw Payload GMT Log Time Source Port Processing Time Bytes Sent Bytes Received Cache Information Error Information Authentication Server Log Time Client IP Destination IP Destination Port Protocol Action Rule Result Code HTTP Status Code Client Username Source Network Destination Network URL Server Name Log Record Type


10.100.10.213 -    - TCP    - -  -  - - - 6/9/2008 7:10:03 PM 1941 0 0 0 0x0 0x0  6/9/2008 12:10:03 PM 10.100.10.213 172.31.10.1 3847 MS Firewall Control Initiated Connection [System] Allow remote management from selected computers using MMC 0x0 ERROR_SUCCESS  - Internal Local Host  ISA01 Firewall

10.100.10.213 -    - TCP    - -  -  - - - 6/9/2008 7:09:57 PM 1936 0 0 0 0x0 0x0  6/9/2008 12:09:57 PM 10.100.10.213 172.31.10.2 3847 MS Firewall Control Denied Connection - 0xc0040012 FWX_E_NETWORK_RULES_DENIED  - Internal IntraArray  ISA01 Firewall

****** isa server description ****




1. All servers are W2k3 R2 Standard SP2 with current security patches plus patch 948496
2. All servers have ISA 2k6 EE with patch 939455
3. All servers have Receive Side Scaling disabled on the NICs
4. all servers are domain members
5. two servers are installed as ISA Configuration Storage Services on Internal network segment
6. two servers are installed as ISA Array member servers
7. ISA IntraArray communication between the array member servers are configured to use a separate non-NLB network segment rather than the internal network segment per microsoft best practice recommendations
8. internal dns entries added for both intra-array adapters and the internal nlb address of the isa array (as well as of course the dynamically-added dns entries of the internal adapters)
9. dns entries for the intra-array adapters registered to active directory



A. ISA Array members "ISA01", "ISA02" in array "ISA"

1 . Internal Network segment (defined to ISA by default as Internal Network) 10.100.10.0/24

ISA01 - 10.100.10.239
ISA02 - 10.100.10.240
ISA NLB - 10.100.10.241
manual static route to internal network via internal network router
internal dns servers

2. IntraArray Network segment (defined to ISA as Internal Network, with firewall client disabled and web proxy client enabled) 172.31.10.0/30

ISA01 - 172.31.10.1
ISA02 - 172.31.10.2

ISA IntraArray communication configured to use these adapters on this non-nlb network segment instead of the default nlb Internal network segment

3. "Authenticated"DMZ Network segment (defined to ISA as Perimeter Network) 172.31.11.0/26

ISA01 - 172.31.11.60
ISA02 - 172.31.11.61
ISA NLB - 172.31.11.62

4. "Anonymous"DMZ Network segment (defined to ISA as Perimeter Network) 172.31.12.0/26

ISA01 - 172.31.12.60
ISA02 - 172.31.12.61
ISA NLB - 172.31.12.62

5. External Network segment (defined to ISA by default as External Network) xxx.xxx.xxx.xxx/28

ISA01 - xxx.xxx.xxx.1
ISA02 - xxx.xxx.xxx.2
ISA NLB - xxx.xxx.xxx.3
default gateway=internet router

6. Non-default Network rules added

(a) Route rule: "Authenticated"DMZ->Internal
(b) NAT rule: Internal->"Anonymous"DMZ

7.  default "Internet Access" Network rule modified

(a) "Authenticated"DMZ and "Anonymous"DMZ added as source networks  '

B. Configuration Storage Servers on internal network segment

Primary ISA Configuration Storage Server - "ISACSS1" - 10.100.10.213
Alternate ISA Configuration Storage Server - "ISACSS2" - 10.100.10.212





IanC -> RE: issue with ISA 2006 EE CSS servers and NLB (11.Jun.2008 10:48:26 AM)

Hi Kas,

I think the most likely cause is DNS.  Specifically, the CSS is resolving ISA01 and/or ISA02 to the wrong IP address, perhaps one of the Intra-Array addresses or one from a DMZ adapter.

Check your internal DNS zone.  I would remove the DNS records for the intra-array adapters for ISA01 and ISA02.  Also, it's not a good idea to let the array members dynamically register their IP addresses, particularly as your ISA servers have several adapters.  Make sure you're getting 10.100.10.239 and 10.100.10.240 returned for ISA01 and ISA02 respectively when running nslookup from the CCS(s).

Ian



kas -> RE: issue with ISA 2006 EE CSS servers and NLB (11.Jun.2008 3:50:08 PM)

Thank you, IanC, for your reply! 

I apologize for not being more clear with regard to the dns configuration in my original post.   To clarify:

1.  the host names in our internal dns for the intra-array adapters were different host names than those for the internal adapters.     i.e.  internal adapters are isa01.domain.com,  isa02.domain.com mapped to the internal ip addresses,  while the intra-array adapters are isa01a.domain.com,  isa02a.domain.com, mapped to the intra-array ip addresses.

dns entries for the intra-array adapters were created only in an attempt to work around this issue,  and was done in accordance with the instructions in the article <<http://www.microsoft.com/technet/isa/2006/nlb.mspx>> in the paragraph titled "Multiple Network Adapters and NLB",  where this action is documented as being necessary when the CSS is installed on the array member.   (the only deviation from the instructions in this paragraph was correcting the syntax of the setspn command).   

2. the only adapter on the array members that was configured to automatically register the dns entries was the internal adapter

3.  still, i have gone ahead and done as you suggest, specifically:

a.  removed the internal dns entries for the intra-array adapters

b.  reconfigured ISA intra-array communication back to using the ip addresses rather than the dns names of the intra-array adapters

c.  unregistered the dns names for the intra-array adapters from active directory (setspn -d etc.) 

d.  ipconfig /flushdns on the primary and alternate CSS servers

items a-d  puts the array members and css servers back to the state in which this problem was first observed

e.  configured the internal adapters of the array members to not do automatic dns registration,   ensured that the dns entries for the internal adapters were static and not dynamic,  confirmed from the css servers via nslookup that the names for the internal adapters of the array members  were resovling correctly. 

4.  after having done all of the above,  the problem persists

We still haven't heard anything useful from Microsoft support on this.    Thank you  in advance to anyone who may have suggestions/recomendations for a solution.   



IanC -> RE: issue with ISA 2006 EE CSS servers and NLB (12.Jun.2008 12:44:28 PM)

Thanks for the additional info Kas.  This is an interesting one.

I can replicate your issue exactly but only when my CSS resolves the array members to their intra-array Ip addresses. 

Now that you have confirmed that your CSS resolves ISA01 and ISA02 to 10.100.10.239 and 10.100.10.240 respectively, how do the log entries look?  Do they now show the correct destinations (10.100.10.239 / 10.100.10.240) rather than 172.31.10.1 and 172.31.10.2 as per your original logs?  If so, is the FWX_NETWORK_RULES_DENIED log entry still there?

Cheers

Ian



kas -> RE: issue with ISA 2006 EE CSS servers and NLB (15.Jun.2008 10:24:05 PM)

Thank you once again, IanC, for your reply. 

Log entries are as they were before, and as shown in my original post. 

a.   When network load balancing is disabled on the Internal network segment,  the CSS servers residing on the Internal network segment, successfully connect to both array members using the IntraArray IP addresses of the array members via the Internal->LocalHost network rule.

b.  When network load balancing is enabled on the Internal network segment, the CSS servers continue to attempt to connect to the array members using the IntraArray IP addresses of the array members.       However the CSS servers connect to only one array member using the Internal->LocalHost network rule.   The other attempts to connect using an Internal->IntraArray network rule.  Since no such rule exists, this connnection is denied.    Also as previously noted,  this symptom will flip from one array member to the other depending on the order in which firewall services are started on the array members.  

Since we cannot define a network route rule from Internal->IntraArray due to the fact that one network segment is nlb and the other is not,  and since we intend for IntraArray communcations to be performed on the non-nlb IntraArray network segment and not on the Internal network segment as per Microsoft's published documentation (e.g. <<http://www.microsoft.com/technet/isa/2006/evaluate/sysreqs.mspx>>,   <<http://technet.microsoft.com/en-us/library/bb794814.aspx>>  among others),   we are stiill looking a solution that will allow the CSS servers to properly communicate with both ISA array members when network load balancing is enabled on the Internal network segment.

Microsoft support has still not provided us with any useful information regarding this issue.   Therefore, thank you in advance to anyone who may have suggestions/alternatives/recommendations for resolving it.     



IanC -> RE: issue with ISA 2006 EE CSS servers and NLB (16.Jun.2008 10:08:45 AM)

Hi Kas,

The term intra-array communication referenced in the MS documentation means communication between members of the same array, not between the CSS and members of the array.

Configuring the CSS to communicate with the internal adapters of the array members has always served us well in the past and should solve your problem.

Ian




MyITGuy -> RE: issue with ISA 2006 EE CSS servers and NLB (14.Aug.2008 8:47:51 AM)

Hi Kas, what's the current status of this issue?

I'm having the same exact symptom in a new production enviroment we are currently standing up for one of our client.

While I'm currently working to resolve this issue, I can tell you this much base on my understanding of what you have outline.

The NLB is not the cause of the problem more so is the fact that CSS seems to be making some random RPC TPC call (using different port #)once it established communication with an Array member on 2171/2172; and in our situation, this is a problem because there is a third party firewall manage hardware between my protected enclave (which currently house my CSS) and the Array member (sitting on the other side of the firewall in a private dmz).

The word is from a previous engagement, this was somehow resolved by modifing a registry attribute that force CSS to make this call on static port number.

I will keep you posted; and if you happen to find a resolution please let me know.

Thanks




Page: [1]