NIC Config - Intra-Array (Full Version)

All Forums >> [ISA 2006 Firewall] >> General



Message

ldoodle -> NIC Config - Intra-Array (19.Jun.2008 10:49:03 AM)

Hiya,

Our ISA boxes have 2 NICs at the moment - one for External and one for Internal.

We are running a 2 server array at the main office (Enterprise), and a single firewall at the remote office (Standard), and my understanding is that dedicated Intra-Array NICs are only needed if you have more than one array (correct if wrong), which we don't currently have (just the one array for the main site).

I am planning to upgrade the remote office to the same config (1 CSS + 2 members) within 12 months, and have a second array.

Would this then bring up the requirement for dedicated Intra-Array NICs?

Thanks



Jason Jones -> RE: NIC Config - Intra-Array (19.Jun.2008 11:16:01 AM)

A dedicated intra-array NIC is best practice and was also required when using NLB as the default unicast mode of ISA prevented intra-host communications unless you had a dedicated NIC.

With the advent of Windows 2003 SP1, it became possible for unicast NLB nodes to communicate without a dedicated NIC, but it still "feels right" to use an intra-array for all EE deployements IMHO.

There is also an issue of security in that all intra-array traffic is much better protected by using a server-server network as opposed to passing it across the internal network, which could be sniffed.

For the sake of an extra NIC per server, I always include it in my designs...

Have a look here for recommended intra-array NIC config.

Cheers

JJ



ldoodle -> RE: NIC Config - Intra-Array (19.Jun.2008 11:47:57 AM)

OK, so i'm guessing the dedicated intra-array NICs are on their own address range, so for example:

LAN: 1.1.1.1/23
WAN: 2.2.2.2/29
DMZ: 3.3.3.3/29
Intra-Array: 4.4.4.4/29 or whatever mask is suitable

Just to clarify, does Intra-Array mean communication between x number of different arrays, or the communication between members of the same array?

If the latter (which I think is correct, or it would be Inter-Array), does the CSS server need to have a dedicated NIC?

Thanks



Jason Jones -> RE: NIC Config - Intra-Array (19.Jun.2008 12:16:17 PM)

Yep, it needs a unique subnet/network. A class C is normally fine, but just make sure it is unique within you environment. If the network is used elsewhere ISA will never be able to route the packets back out of the correct interface as it will send it down the intra-array NIC by mistake.

You also need to create an ISA network to represent the subnet and enabled the web proxy listener for this network.

Yep, Intra-array is the name for communications between array members in the SAME array. There is no communication between arrays.

Nope the CSS doesn't need a dedicated NIC as it hosts the array configuration and does not form part of the array (assuming you have a dedicated CSS of course).

Cheers

JJ



gbarnas -> RE: NIC Config - Intra-Array (19.Jun.2008 2:20:56 PM)

quote:

ORIGINAL: Jason Jones
Yep, it needs a unique subnet/network. A class C is normally fine, but just make sure it is unique within you environment. If the network is used elsewhere ISA will never be able to route the packets back out of the correct interface as it will send it down the intra-array NIC by mistake.

In our enterprise, every ISA array or server cluster uses the same network for Intra-Array communications. We designated 172.31.255.0 as the intra-server network. Since our enterprise network is 10.0.0.0, it's pretty obvious that the 172.31.255.0 network is "special". Also, since that address range is captive to each set of clustered servers, there's no chance of mis-routed replies.

Glenn



Jason Jones -> RE: NIC Config - Intra-Array (19.Jun.2008 6:30:06 PM)

I don't see a problem with using the same intra-array network definition across multiple arrays, but the key issue is to make sure you do not choose an address that overlaps with any part of the WAN/LAN that may need to communicate with ISA. Using a complete separate IP address class (as per Glenns example) is a good way to try avoid any potential problems.

IMHO I would use rather different intra-array networks per array, not that it really matters though...

Cheers

JJ



ldoodle -> RE: NIC Config - Intra-Array (20.Jun.2008 5:42:17 AM)

quote:

ORIGINAL: Jason Jones

Yep, it needs a unique subnet/network. A class C is normally fine, but just make sure it is unique within you environment. If the network is used elsewhere ISA will never be able to route the packets back out of the correct interface as it will send it down the intra-array NIC by mistake.

You also need to create an ISA network to represent the subnet and enabled the web proxy listener for this network.

Yep, Intra-array is the name for communications between array members in the SAME array. There is no communication between arrays.

Nope the CSS doesn't need a dedicated NIC as it hosts the array configuration and does not form part of the array (assuming you have a dedicated CSS of course).

Cheers

JJ


Just thinking out loud here - technically could I use a 10.x.x.x address for LAN, DMZ and Intra-Array. As long as they have a different subnet, it should work OK shouldn't it?

So for example:

Main Office:
LAN 10.0.0.0/23 (10.0.0.0 - 10.0.1.255)
WAN: ISP
DMZ: 10.0.100.0/24 (10.0.100.0 - 10.0.100.255)
Intra-Array: 10.0.101.0/24 (10.0.101.0 - 10.0.101.255)
VPN: 10.0.201.0/24 (10.0.201.0 - 10.0.201.255)

Branch Office:
LAN 10.0.2.0/23 (10.0.2.0 - 10.0.3.255)
WAN: ISP
DMZ: 10.0.102.0/24 (10.0.102.0 - 10.0.102.255)
Intra-Array: 10.0.103.0/24 (10.0.103.0 - 10.0.103.255)
VPN: 10.0.203.0/24 (10.0.203.0 - 10.0.203.255)

Or is that a complete nose-turn to best practices? :)



Jason Jones -> RE: NIC Config - Intra-Array (20.Jun.2008 7:11:08 AM)

Personally I would use a different class. There is a lot of value in making DMZ or hearbeat networks "look different" and this also avoid potetial problems if someone gets a subnet mask wrong.

If you are worried about big overlapps, just choose a private range but subnet it down to just a few hosts - maybe use a /28 or something?

What you have proposed will work technically, but I wouldn't do it that way. Maybe put the DMZ on a class B and Intra-array on a class C.

Cheers

JJ



ldoodle -> RE: NIC Config - Intra-Array (20.Jun.2008 7:24:28 AM)

Yeah I wouldn't have actually used a setup like that, and can see the pros of different addresses.

I just like to try to understand as much about a product as I can!

Thanks



gbarnas -> RE: NIC Config - Intra-Array (20.Jun.2008 7:26:47 AM)

That's exactly the point of using a "special" intra-array network... it's never going to be part of your enterprise LAN or VPN environment. Most cluster systems I've seen employ a /24 subnet, even if connecting just two systems. If you use a different network for each cluster (or ISA array - same difference), you or the network group need to track each network.

We have three multi-server ISA arrays here, 7 Windows clusters, and about a dozen *nix clusters. It now comes down to keeping track of 20 subnets or 1.

In IDoodle's example, he's using the same core network for intra-array, DMZ, and Internal. This opens the possibility for misconfiguration by using an overlapping network. By using a different range entirely (ie - 10 for your enterprise, and 172.31 for your clusters) it's very unlikely to have problems.

Also - for IDoodle - Assigning ip addresses on decimal boundaries makes it easy for us analog life forms to recognize network divisions, but binary life(less) forms prefer binary divisions. As your network becomes more complex, you'll find that starting a network address range with "50"  or  "100" will cause more problems than if you start with, say, "64" or "128". Using these powers of two will allow simple routing (summary routes) instead of having to specify several or even dozens of route statements in the future.

Glenn



ldoodle -> RE: NIC Config - Intra-Array (20.Jun.2008 8:17:56 AM)

As there is no communication between arrays, I therefore assume multiple arrays can have the same NIC config.

Another example (Intra-Array NIC only):

Main Office:
Server1: 172.31.255.1/29
Server2: 172.31.255.2/29

Branch Office:
Server1: 172.31.255.1/29
Server2: 172.31.255.2/29

Or does the same apply for remote-site VPN networks (must be unique network ID), and therefore each Intra-Array NIC needs a unique address (so .3 and .4 for Branch Office servers), even if in different arrays?

Last question I promise!



gbarnas -> RE: NIC Config - Intra-Array (20.Jun.2008 9:42:50 AM)

There is no routing of that address range. Only computers directly connected to an array or cluster will be aware of those addresses, and since each cluster is directly connected, they will never try to route to them via the other interfaces.

That is why it's important to use a unique address for intra-cluster communications.

There's no need to use different addresses, although some people feel that it adds an extra level of identification. For me, I start with 1 and go up for each array/cluster.

Glenn



Jason Jones -> RE: NIC Config - Intra-Array (20.Jun.2008 9:51:50 AM)

The compromise I have used with a few customers is to use a /24 class C which can be learnt by the network team as a hearbeat or intra-array network. Each array can then have a supernet, say using /28, to define the an individual network for each intra-array network. At the end of the day, it is just personal choice, as it is pretty much cosmetic anyhow [:)]

I tend to use a lot of multi-NIC setups and like the 4th octet to match on all interfaces as much as possible. Hence my intra-array NIC would have the same 4th octet as other interfaces. But again, this is maybe me just being a bit too anal [8D] 



ldoodle -> RE: NIC Config - Intra-Array (20.Jun.2008 10:20:59 AM)

quote:

ORIGINAL: Jason Jones
But again, this is maybe me just being a bit too anal [8D] 


No, I am exactly the same, hence this 'round the sun to meet the moon' thread - could've been closed after post #4!

OK, so this is my planned config:

Main Office:
Server1 LAN: 10.0.0.1/23
Server1 WAN: ISP
Server1 DMZ: 192.168.0.1/29
Server1 IA: 172.31.0.1/29
VPN Pool: 10.0.101.1 - 10.0.101.127/24

Server2 LAN: 10.0.0.2/23
Server2 WAN: ISP
Server2 DMZ: 192.168.0.2/29
Server2 IA: 172.31.0.2/29
VPN Pool: 10.0.101.128 - 10.0.101.254/24

Branch Office (single fw at the moment but when upgraded):
Server1 LAN: 10.0.2.1/23
Server1 WAN: ISP
Server1 DMZ: 192.168.2.1/29
Server1 IA: 172.31.2.1/29
VPN Pool: 10.0.103.1 - 10.0.103.127/24

Server2 LAN: 10.0.2.2/23
Server2 WAN: ISP
Server2 DMZ: 192.168.2.2/29
Server2 IA: 172.31.2.2/29
VPN Pool: 10.0.103.128 - 10.0.103.254/24



gbarnas -> RE: NIC Config - Intra-Array (21.Jun.2008 7:55:53 AM)

Anal?  with 24 pages & 2 visio's to document my home network config - certainly not me! ;)

One more comment about network setup - "Organization"

You'll need a VIP address for each array to implement NLB. I might use the .1 address for the VIP, and use 2-7 for the physical addresses (start with 2 & 3, the rest are for possible growth). Allocate another block of addresses for your servers, and another for other hosts. I usually allow at least 50% for growth in each address block, often more.

Like my earlier comment - arrange hosts on a power of 2 boundary whenever possible. If you expect 12 servers, round up to 16 & double - 32. Place the server address block on a multiple of 32 boundary, 32-63, 64-95, etc... This way, should you ever want to islolate them via a firewall, you can easily move them as a logical subnet.

Glenn



ldoodle -> RE: NIC Config - Intra-Array (26.Jun.2008 9:56:07 AM)

quote:

ORIGINAL: gbarnas
You'll need a VIP address for each array to implement NLB. I might use the .1 address for the VIP, and use 2-7 for the physical addresses (start with 2 & 3, the rest are for possible growth)

I do the opposite - start with .1 for physical and the last for the virtual.

quote:

ORIGINAL: gbarnas
Like my earlier comment - arrange hosts on a power of 2 boundary whenever possible. If you expect 12 servers, round up to 16 & double - 32. Place the server address block on a multiple of 32 boundary, 32-63, 64-95, etc... This way, should you ever want to islolate them via a firewall, you can easily move them as a logical subnet.

I see the benefit of this, but can I ask why you have started with 32-63 and not 0-31?

Thanks



Jason Jones -> RE: NIC Config - Intra-Array (26.Jun.2008 11:35:02 AM)

quote:

ORIGINAL: ldoodle

quote:

ORIGINAL: gbarnas
You'll need a VIP address for each array to implement NLB. I might use the .1 address for the VIP, and use 2-7 for the physical addresses (start with 2 & 3, the rest are for possible growth)

I do the opposite - start with .1 for physical and the last for the virtual.

quote:

ORIGINAL: gbarnas
Like my earlier comment - arrange hosts on a power of 2 boundary whenever possible. If you expect 12 servers, round up to 16 & double - 32. Place the server address block on a multiple of 32 boundary, 32-63, 64-95, etc... This way, should you ever want to islolate them via a firewall, you can easily move them as a logical subnet.

I see the benefit of this, but can I ask why you have started with 32-63 and not 0-31?

Thanks


I disagree with both [8D]

In my experience, it is more likely to add more virtual IPs (VIPs) than dedicated IPs (DIPs) expecisally on the external interface. Hence it makes sense to set aside a block of addresse for array members (DIPs) then leave a block of addresses above this for the growing VIPs.

This way you avoid the chance of having:

DIP Server 1 = .1
DIP Server 2 = .2
VIP1 =.3
DIP Server 3= .4
VIP2 = .5
VIP3 = .6
DIP Server 4 = .7

[:'(]

and hopfully have this:

DIP Server 1 = .1
DIP Server 2 = .2
...
DIP Server 10 = .10

VIP1 = .20
VIP2 = .21
...
VIP10 = .30

[:D]

Alternatively if you like the default gateway to be .1 flip them around as follows:

DIP Server 1 = .20
DIP Server 2 = .21
...
DIP Server 10 = .30

VIP1 = .1
VIP2 = .2
...
VIP10 = .10

By allowing growth in both the VIP and DIP addresses, you will always have a contiguous flow of addressing that makes sense...I also never publish anything on the primary VIP as I leave this as a special gateway only VIP.

All just IMHO of course [;)]

Cheers

JJ



gbarnas -> RE: NIC Config - Intra-Array (26.Jun.2008 3:33:02 PM)

Yeah - hadn't had my coffee yet.. Most of the time I had added an additional server - DIP - so that was on my mind.  The point was, and Jason clarified it - that you define a block for VIP and another for DIP, otherwise you wind up with a mess of interspersed DIP & VIP addresses.

As for starting with 32-63 - just habit. I usually stick my communications gear at the lowest addresses, infrastructure servers above that, and app servers above that. Based on my example, we wanted 32 addresses for servers. Since I mentally assign routers and such to low addresses, I moved to the next full block. Grouping similar servers & network equipment into logical address blocks allows you to apply subnet rules, especially to devices that don't authenticate to ISA.

Finally, if you do want the flexibility of being able to subnet those devices, you'll probably want to treat it like a subnet from day-1 and avoid the first 2 & last address in the block. ;)

Glenn



ldoodle -> RE: NIC Config - Intra-Array (30.Jun.2008 11:26:29 AM)

quote:

ORIGINAL: Jason Jones
I disagree with both [8D]

In my experience, it is more likely to add more virtual IPs (VIPs) than dedicated IPs (DIPs) expecisally on the external interface. Hence it makes sense to set aside a block of addresse for array members (DIPs) then leave a block of addresses above this for the growing VIPs.


Well that is kind of what I have done.

Internal server range: 10.0.0.1 - 10.0.0.100/23
FW's: 10.0.0.1 - 10.0.0.5/23
CSS's: 10.0.0.6 - 10.0.0.10/23
DC's: 10.0.0.11 - 10.0.0.15/23
MAIL: 10.0.0.16 - 10.0.0.20/23
FILE: 10.0.0.21 - 10.0.0.25/23
etc
etc

External interfaces: 100.100.100.128 - 100.100.100.143/28
FW1: 100.100.100.131/28
..
FW5: 100.100.100.135/28
VIP1: 100.100.100.136/28
..
VIP5: 100.100.100.140/28

We will never, ever have more than 5 firewalls, dc's, mail servers etc at each site.



gbarnas -> RE: NIC Config - Intra-Array (30.Jun.2008 12:06:45 PM)

You're still thinking decimal though.. 1-5 is not a binary divisible range, 0-3 and 0-7 are. Even though you won't be subnetting, you can easily summarize a group of systems for applying network or firewall rules if they are on binary boundaries - multiples of 2, 4, 8, 16, etc.

It's easier for the computer, which means there's less chance of misconfiguration (or misinterpretation of your expectations). That's not to say 1-5,6-10, etc won't work, it's just "unnatural" in terms of networking.

Glenn



Page: [1] 2   next >   >>