The setup is ISA 2004 on a Server 2003 box. We are able to connect with PPTP with no issues. But, when we try via L2TP we receive the Error 768: The connection attempt failed because of failure to encrypt data. Then I uncheck the option of "Require data encryption" on the Security tab in the VPN Connection Properties and receive Error: 789 The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer. Any assistance would be great!
We tested only one at a time and are currently testing different types of certs. We have created IPSec, Computer, Administrator, User certs with no success in connecting. ItĘs an instantaneous error that we receive while we are trying to connect.
From: Albuquerque NM USA
quote:L2TP wont work over NAT
L2TP with NAT-T is supported on the server side with Windows Server 2003. On the client side, you have to have Win2K or XP SP1 with 818043 (NAT-T update) or XP SP2. (I don't know about 9x boxes since I don't deal with them any more).
My PIX was configured to allow inbound UDP 4500, and UDP 500. 1723 not needed since it is encapsulated inside of 4500 NAT-T.
I had everything configure correctly. Could connect inside the PIX but not outside. Well, the problem turned out to be a change in XP SP2 VPN. Before SP1 the default behavior was to assume both client and server were behind NAT devices. SP2 made it so that it would still work but not with the server behind a NAT device.
There is a registry key that you will need to change on the SP2 client. There used to be a MS KB article on this, but it has since disappeared from MS website.