l2tp/ipsec vpn fails with error 789 (Full Version)

All Forums >> [ISA Server 2004 Firewall] >> VPN



Message


rportch -> l2tp/ipsec vpn fails with error 789 (24.Sep.2004 5:27:00 PM)

Looking for some direction. We have a 2004 isa server (just testing it), and want to move away from pptp vpn's. The idea being that we want to control who can use what equipment to vpn into the network. First I got vpn to work using PPTP, then following one of the articles/guides I set up a enterprise CA and issued certificates for both the ISA server and a couple clinet machines. Both client are XP.

The first test was with one of client connected to a different local network, which sends traffic out of a isa 2000 firewall. PPTP works, but setting the client to use only l2tp (also did this on the isa 2004 server - disabled pptp), the 789 error comes up. Looking at the ISA logs shows the client initiating a IKE connection, then the ISA server initiates a IKE connection back to the client. Hangs there for a while then (reports the 789).

Although that network has others using vpn to connect to remote sites, to eliminate the isa2000 server as a fault, I took a laptop home where I have verizon dsl. Same end results, however, the ISA 2004 server doesn't attempt a IKE connection back to the client. The client does however send tcp port 1 traffic to the ISA server.

I tried turning on the oakley loggin, but the log file is empty.

Seems like this should be simple huh based on the many articles.

The VPN server is the ISA 2004 server, it is a member of the domain, and is running on a 2000 server platform (which I think is what prevents the use of a pre-shared key as an alternative to the certificates). The clients are XP (not sp2 yet).




tshinder -> RE: l2tp/ipsec vpn fails with error 789 (25.Sep.2004 11:21:00 PM)

Hi R,

Is there a NAT device in the path? If so, you need to use the updated L2TP/IPSec VPN client.

Also, make sure that fragment filtering is disabled.

HTH,
Tom




ClintD -> RE: l2tp/ipsec vpn fails with error 789 (26.Sep.2004 2:12:00 AM)

Are we using certs? Usually an empty client oakley means that since we don't hae a cert, RAS can't push the IPSec policy dynamically into IPSec (this is how L2TP works - by pushing filters for UDP 1701 into IPSec).

If we're using a Pre-Shared key, post the contents of the oakley log from the server and we'll see what's up.

Edit - Whatta dork I am - i just re-read your post. I see you're using certs on the client - does it have the Trusted Root Cert installed from the CA? Go into the MMC-Add/Remove Snap-In and add the Certificates console and focus it on the "Computer Account" context. Go into Personal\Certificate and open up the cert that is installed and make sure there are no errors on the 3 tabs. If it reports that it can't verify the cert, you'll need to go to the CertSrv website and choose the "Download a CA certificate, certificate chain or CRL" link.

From the next page, hit the "Download the CA certificate" link and choose to Save the certnew.cer file - do not open it from within IE or it will not get installed correctly for IPSec. Once it is saved, go back to the Certificates snap-in and go to the Trusted Root Certification Authority node, right click\All Tasks\Import and browse to the certnew.cer file.

Once you've done this, check the certificate up in Personal\Certificates again and the cert should have no errors.

Try to connect again.

[ September 26, 2004, 02:23 AM: Message edited by: ClintD ]




rportch -> RE: l2tp/ipsec vpn fails with error 789 (27.Sep.2004 3:48:00 PM)

Thanks for the replies. I have verified the certs on both the ISA server and the clients. The internal CA that I created is shown in the trusted root authorities and the certs do show an OK status.

One thing I found after I had created the enterprise CA and issued the certs was that I was supposed to have installed the CA ont he domain controller. It works properly, and the cert for the CA got into the systems by domain membership, I did not have to do anything manually there.

I will try the updated client, or verify that the client files are already correct. In both cases I used to test there is a NAT device htta the client transmits to. Thanks again for the feedback and suggestions.




rportch -> RE: l2tp/ipsec vpn fails with error 789 (27.Sep.2004 6:09:00 PM)

Same results after updating the l2tp/ipsec client files (and packet fragmenting is disabled)




ClintD -> RE: l2tp/ipsec vpn fails with error 789 (28.Sep.2004 1:45:00 AM)

One quick note - if the XP client is beind a NAT device, this is not going to work since you have ISA 2004 instlaled on Win2000 - only Win2003 has the updated VPN Server side component to allow NAT-T L2TP connections - you can install the NAT-T patch on Win2000, but it doesn't update Routing and Remote Access.

With that out of the way...

Can you install the WinXP SP2 support tools and run the RASDIAG utilty on the client?

WinXP SP2 Support Tools Link. You do not need SP2 installed in order to run RASDIAG, nor does the Support Tools package update any core operating system files.

It's named RASDIAG.EXE and when you run it, it will open a command shell and echo "Preparing Windows XP for RAS Diagnostics". After a brief delay, it will echo that it is ready and waiting on you to reproduce your problem.

Attempt to connect to the server and after you receive the 789 error, go back to the RASDIAG command shell and press the space bar. It will generate a .RDG file which is a CAB file of network traces from all interfaces and a RASDIAG.TXT file that contains some logging information.

Could you send the <timestamp>.RDG file to me at work and I'll try to figure out where this is broke.

My email is <My ISA Login>@microsoft.com.

[ September 28, 2004, 01:50 AM: Message edited by: ClintD ]




ToddMan -> RE: l2tp/ipsec vpn fails with error 789 (28.Sep.2004 6:27:00 AM)

I've been loosin hair over this 'bug' for a week! PTPP works rock solid every time. L2TP works when all the moons align.

Group Policy: Under network security: Lan Manager authentication level - you have to have the DC, ISA, AND the client all able to send LM&NTLM

If you set to Send LM & NTLM - use NTLMv2 session security if negotiated - It will work sometimes, not others????!




ClintD -> RE: l2tp/ipsec vpn fails with error 789 (28.Sep.2004 9:47:00 AM)

ToddMan - are you getting Error 789 or 691?

The issue you describe is similar to the issue covered in Microsoft KB Article 826157 - "Error 691" Error Message When You Log On to a Windows Server 2003-Based Computer or a Windows 2000-Based Computer That Is Running Routing and Remote Access or Internet Authentication Service.

[ September 28, 2004, 09:49 AM: Message edited by: ClintD ]




tshinder -> RE: l2tp/ipsec vpn fails with error 789 (28.Sep.2004 12:31:00 PM)

quote:
Originally posted by ToddMan:
I've been loosin hair over this 'bug' for a week! PTPP works rock solid every time. L2TP works when all the moons align.

Group Policy: Under network security: Lan Manager authentication level - you have to have the DC, ISA, AND the client all able to send LM&NTLM

If you set to Send LM & NTLM - use NTLMv2 session security if negotiated - It will work sometimes, not others????!

Hi Todd,

What bug?

Is there a NAT device in the path?

Do you have a network diagram?

Where is the ISA firewall?

Where is the VPN client?

What are the interposed network devices?

If you're in Texas you better tell me right, because both Clint and I are in Texas and we'll find ya [Wink]

Thanks!
Tom




rportch -> RE: l2tp/ipsec vpn fails with error 789 (28.Sep.2004 3:43:00 PM)

ClintD, thanks so much. In the 2 test scenarios the clients are behind NAT devices, and I am thinking that for the most part our remotes users will always be in environments that use nat for outbound traffic. Since the case is that server 2000 won't support the updated nat-t, I will move the isa 2004 over to a 2003 server. Prior to doing that I will do xp sp2 on the client that is behind the isa 2000 server and capture the ras traffic with rasdiag just for the heck of it, and will wait till I try the 2003 server before posting anything. Thanks again so much, the help in these forums in invaluable.




ToddMan -> RE: l2tp/ipsec vpn fails with error 789 (28.Sep.2004 8:19:00 PM)

Sorry that I jumped in the middle of this one.

ISA 2004 on Windows Server 2003.
All Server 2003 - XP SP2 network.
Edge firewall running Rainconnect.
Clients are on/off network clients.

Initially received 691 error - fixed unfortunately by changing network security under group policy to: Send LM & NTLM - use NTLMv2 session security if negotiated.

PTPP works rock solid under this setup.

I use only L2TP and it will work once. Then if you lose connection, and go to log back in, it hangs and will give me the 789 error.

Question: Why do I have to 'downgrade'policy to allow VPN access.

Question:Is RASDIAG the best way to troubleshoot.

I'm three and a half good days south by wagon.
Those were the days!




rportch -> RE: l2tp/ipsec vpn fails with error 789 (29.Sep.2004 8:25:00 PM)

It works with server 2003 as the o/s, not perfect yet as I can't use the domain to authenticate (have to use local accounts on the isa server). Thanks again for the assistance.




Page: [1]