Welcome to ISAserver.org

Site-to-Site VPN partially working (PING only)

Users viewing this topic: none

Logged in as: Guest

Printable Version

All Forums >> [ISA 2006 Firewall] >> VPN >> Site-to-Site VPN partially working (PING only)

Page: [1]

Message

<< Older Topic Newer Topic >>

Site-to-Site VPN partially working (PING only) - 13.Sep.2006 8:21:23 PM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

I have set up a site-to-site VPN between two ISA 2006 servers. For simplicity sake, I'm using straight IPSec, and not L2TP or PPTP. My reason is, I don't want to mess with RRAS, static pools, DHCP, etc. I just want a simple tunnel between my a remote site and the main office. It seems simple enough.

After setting up the Network, the Network Rules, and the access rule on both firewalls, the VPN tunnel is created and I can ping the other side. From a machine behind the firewall in the branch office, I can ping all the servers behind the firewall at the main office. However, this is all I can do. Once I attempt to open a RDP connection to a server behind the firewall at the main office from the branch office, I get the initial connection, but then the connection is dropped. This happens with every protocol I've tried (RDP, SSH, FTP). Again, I can initiate the connection, but it is immediately dropped.

The access rules are configured for all protocols, so that should not be the issue, but something is obviously blocking this. What would allow ping to work, but nothing else? Also why would it start to work, (bring up the remote console for a brief sec), then drop the connection?

I'm at a loss. I've been pulling my hair for 2 days, so I've come to the only place I know that will help, and I'm at your mercy. Please be gentle.

< Message edited by RedSunshine -- 13.Sep.2006 8:28:16 PM >

Post #: 1

Featured Links*

RE: Site-to-Site VPN partially working (PING only) - 13.Sep.2006 8:54:08 PM

spouseele

Posts: 12782
Joined: 1.Jun.2001
From: Belgium
Status: offline

Hi RedSunshine,

I didn't play yet with ISA 2006 but I think that using ESP Null Encryption as described in my article http://www.isaserver.org/tutorials/enable-ESP-Null-Encryption-ISA-2004-site-to-site-VPN-scenario.html could be very useful to debug your scenario.

HTH,
Stefaan

(in reply to RedSunshine)

Post #: 2

RE: Site-to-Site VPN partially working (PING only) - 13.Sep.2006 9:54:43 PM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

Update:

From the Main office, I can RDP to any machine into the branch office. However, from the branch office, I cannot RDP to anything in the Main office.

Another addition:

I said previously that both servers were 2006. This is incorrect. The Branch office is 2004 Version: 4.0.2165.594, and the Main office is 2006 RTM. Also, the Main office is Server 2003 R2, where the Branch office is 2003 SP1. Not sure that any of this makes a difference. It's puzzling. I wonder if the 2006 Server failing to create a proper dynamic IPSec Policy. Any way to compare the policies between the 2 servers?

(in reply to spouseele)

Post #: 3

RE: Site-to-Site VPN partially working (PING only) - 13.Sep.2006 10:02:32 PM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

Update 2:

I've started logging on the ISA 2006 server in the Main office for all connections from the Branch Office. It showed the connection initiated, but then soon after, a "denied connection". And what rule is denying the connection you ask? None. It's blank.

(in reply to RedSunshine)

Post #: 4

RE: Site-to-Site VPN partially working (PING only) - 13.Sep.2006 11:39:05 PM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

Update 3:

When monitoring sessions, I noticed that only the VPN tunnel is created on the Main side, and not on the Branch side. Is it normal to have only 1 side show the tunnel?

(in reply to RedSunshine)

Post #: 5

RE: Site-to-Site VPN partially working (PING only) - 14.Sep.2006 7:31:03 AM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

I have done extensive testing today, and I'm still stumped. I even went as far as setting up a new ISA server in the branch office to see if something what weird with the current server. The new server was ISA 2006 (2004 was previously used in the branch office). No matter what I seem to do, it always gives the same result. Why would everything work from one side of the tunnel, but not the other? Another puzzling thing is why does PING work from both sides, but once real traffic is passed, it chokes one way but not the other? Okay, I have a theory, but may need to help from the community to tweak the registry...

After looking at all I changed, I found one constant. The branch office uses a PPPoE connection, where the main office does not. The main office is sitting in a datacenter with a 100mb connection. The branch office uses PPPoE and has a fiber line provided by Verizon FiOS. No matter what version of ISA I used in the branch office, the PPPoE was not changed. This got me thinking, "Isn't the MTU a different size when using PPPoE"? Would this be the reason I can communicate from the main office side, but not the branch?

Any thoughts? Any idea what my next step should be? My head hurts now. :) Time for the delicious nectar from the St. James Gate Brewery.

(in reply to RedSunshine)

Post #: 6

RE: Site-to-Site VPN partially working (PING only) - 16.Sep.2006 11:57:44 AM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

After many, many hours of troubleshooting, I've finally figured out what was causing the problems. It was indeed an MTU problem.

The Main Office is on a 100mb connection using straight Ethernet. (MTU 1500)
The Branch Office is using a PPPoE connection provided by Verizon FiOS. (MTU 1480)

Here is the rub.

This is a simple diagram of the setup:

Desktop -- ISA (Branch) -- Internet -- ISA (Main) -- Server

Both the Desktop and the Server have a standard MTU of 1500. The ISA (Main) Server also has an MTU of 1500. The ISA (Branch) has an MTU of 1480, the standard MTU for PPPoE. Normally the MTU on the PPPoE connection being different should not matter. If I were to connect to a web server on the Internet, the MTU being different would not matter because the routers would send each other ICMP packets telling the sender to fragment the packet and life would be fine. However, when I attempt to connect to the server using RDP, it fails. The reason is (until proven otherwise), the two ISA Servers that are connected via the IPSec tunnel become black hole routers because they do NOT inform the client the MTU is too large. It's either this, or the ISA Servers do not inform each other about the different MTU.

I proved this by setting the MTU on the ISA (Main) side to 1480 for the External Connection. After a reboot, connectivity between both side worked as expected. Also, if I set the MTU on the desktop to 576, and the Main Office with the default 1500, I would be able to connect without issue as well.

To make sure a firewall policy was not blocking the ICMP between sites, I created a rule that allows all protocols from the Internal Network, the Local Host, and the Remote Site Network (VPN). This rule is bi-directional and was created on both sides.

I'm not sure if I found a bug in either ISA Server or in the Server 2003 TCP/IP Stack, but I think this is big none-the-less. This issue is found in both ISA Server 2004 and 2006.

Any thoughts from Tom or Stefaan? And others, feel free to join in with the discussion as well.

(in reply to RedSunshine)

Post #: 7

RE: Site-to-Site VPN partially working (PING only) - 17.Sep.2006 6:49:31 AM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

I just noticed one thing that is also very odd.

1. The MTU on the Main side is 1500. I tested this by pinging google.com from the main office and the max MTU was verified to be 1500.
2. The MTU on the Branch side is 1480. I tested this by pinging google.com from the branch office and the max MTU was verified to be 1480.
3. The MTU from the Main side to the Branch side and vice-versa is 1422. This was verified by pinging the other side of the tunnel until finding the max MTU.

It appears that the IPsec tunnel is causing MTU havoc between the sites.

(in reply to RedSunshine)

Post #: 8

RE: Site-to-Site VPN partially working (PING only) - 20.Sep.2006 5:56:27 AM

colinpbriggs

Posts: 1
Joined: 20.Sep.2006
Status: offline

I have a similair problem with a VPN between our ISA 2004 server and a Cisco VPN 3000 Concentrator at a third party site.

The VPN connects OK. We can Ping in both directions OK. FTP over the VPN from my site to the Cisco site works fine.
FTP of any file bigger than about 1k from the Cisco site to the ISA site fails.

Monitoring shows-
Protocol: IPSec ESP, Action: Denied Connection, Result Code: 0xc0040013 FWX_E_FRAGMENT_PACKET_DROPPED

If I ping with a packet size of 1418 bytes it all works, 1419 bytes and over fails.

I am trying reducing the MTU to 1400 bytes on VPNs. I can't re-boot till tonight. I will confirm if it works around the problem.

(in reply to RedSunshine)

Post #: 9

RE: Site-to-Site VPN partially working (PING only) - 20.Sep.2006 10:21:40 AM

RedSunshine

Posts: 35
Joined: 14.Apr.2003
From: Dallas, TX
Status: offline

I found a workaround that I am willing to accept for now. Things remain to be seen in the long run, but I'm happy for now.

The MTU on the Main side was 1500. This is the main office, and I did not want to dork with the MTU on this side if it were at all possible. So then I started looking for ways to "fix" the branch side. As stated earlier, the MTU on the branch side was 1480 (PPPoE). However, when I would attempt to find the max MTU within the tunnel, I noticed it was 1422. Then, I for the heck of it, I decided to set the branch ISA server to 1422 by following this guide.
http://support.microsoft.com/default.aspx?scid=kb;en-us;283165

This seemed to work.

Just in case this does not work for everyone, I did get some feedback from a MS contact from PSS. The guys worked a case for a customer in Germany that was experiencing a similar issue. Here is the summary of the ticket. If my solution above does not work for you, try the following. Thanks to "T.D." for the help.

PROBLEM:
After you moved your Remote Location from a Leased Line to a Broadband DSL
Connection you notice various Problems with your IPSec Site to Site Tunnel like:
DC Replication is not working, Exchange Email Sync does not work, and some HTTP
pages do not work

CAUSE:
Large IP Packets got dropped on the way to the remote side. The most likely reason
for this are Black hole Routers on the Internet that dropped ESP Packets with the
DF Flag set.

RESOLUTION:
1. Create Rules and Protocol Definitions on both sides of the Tunnel that allow
ICMP PMTU Messages to be delivered to the ISA Server. Follow the instructions from
KB: 902347 CPU use may be more than 50 percent when an ISA Server 2004 computer is
operating under heavy load conditions
http://support.microsoft.com/default.aspx?scid=kb;EN-US;902347

2. Set the Registry Key IPSecDFEncap with Type DWORD to 1 on both ISA Servers,
please set this Key on the Internal AND the External Interface of both ISA Servers.
Use the following Registry Path to set it:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{Interface GUID}
After this change you need to reboot both ISA Servers

3. Added the external IP Address of the remote IPSec Tunnel Endpoint to the Tunnel
Address Range as outlined at:
http://www.isaserver.org/tutorials/2004ipsectunnelmode.html

(in reply to colinpbriggs)

Post #: 10

RE: Site-to-Site VPN partially working (PING only) - 19.Oct.2006 11:20:31 AM

ErikM

Posts: 9
Joined: 4.Feb.2004
From: Norway
Status: offline

I have a similar problem.
The configuration are ISA server 2006 at one site, and Cisco 3005 at the other site.
The tunnel are going down and up again about every 5-6 min.
The traffic over the tunnel are mostly RDP, and the response are sluggish, and sometimes stop responding when the tunnel drop.

In the event log on the ISA server, I get:
"IKE security association ended.
Mode: Data Protection (Quick mode) Filter:........."

In the Cisco 3005 log, it says that the remote peer has ended the session.

In a few days we will try to put a Cisco pix in paralell with the ISA server to see if the problem are on the ISP side or in th ISA server/OS.

I have also tried to reduce MTU size on client computers
Anyone that have a solution on this?

(in reply to colinpbriggs)

Post #: 11

RE: Site-to-Site VPN partially working (PING only) - 19.Oct.2006 5:10:35 PM