I haven't had any luck so far so. I'll be re-imaging the servers with ISA 2006 Fresh install with no NLB. We're going to use Wireshark utility to monitor the packets leaving which MAC address and destination MAC address. Once we have a full connectivity between clients and ISA servers, we'll then use enable NLB and monitor where it stops at.
Yes we're still on 8600. I just spoke to the network team and they said, the NLB is now enabled on the two ISA servers. They have modified something on the network gear to accept the ARP cache on the 8600. I will find out what it is and will post the action tasks.
NLB does not work with Layer 3 Switches. It just doesn't. I tried everything and so as Microsoft team and Nortel Team. We even installed a dummy hub to make sure the ISA servers are plugged into the hubs and then from the hub to the switch, and no go. We contacted Microsoft and we addedd and modified couple of reg keys but still no luck. We monitored the packets from workstation, Server to the switch and back and still no luck. We could track the packets from source MAC address to the switch but at the switch side, everything gets dropped. As long as there is a layer 2 switch it would work without any issue but layer 3 does not cache and update the ARP. We cleared the ARP table, reloaded the NLB by using nlb reload command still no luck. NLB stops responding immediately.
Long hours were spent over this and finally decided to leave the NLB out of the equation and move on with simple ISA. Unless otherwise anyone else has any idea, please let us know.
I have NLB working fine on a L3 switch in my ISA array. I have Extreme Networks core and edge switches. The problem I have is the fact that Integrated NLB runs in Unicast only. The switch gets bogged down with cpu traffic because of the Unicast port flooding operation of NLB, that I have to manually clear the forwarding database on the switch from time to time. Hopefully this will be resolved when we apply the update to enable multicast operations for ISA Server integrated NLB.