I am using the ISA Server 2004 VPN Deployment kit chapter 12, Inbound L2TP/IPSec NAT-T connection through a back-to-back ISA Server 2004 Server DMZ.
The PPTP client works but not the L2TP client. The client is a Windows XP with SP2 on it. If I connect client directly to the Back-end ISA Server the L2TP/IPSec works.
In the Front-end ISA Server log I see the 500 and 4500 port connections the the Back-end ISA server.
In the Back-end ISA Server log I see the same 500 and 4500 Port connections to it but the is no response back from the Back-end ISA Server.
Why does the Front-end log show 500 and 1701 port connections when the L2TP/Sec client is directly attached to it when it works and not the 500 and 4500 port connections when it does not work going thru the Front-end ISA Server?
It shows 1701 because the ISA Server is the endpoint, and IPSec has decapsulated the traffic prior to ISA seeing it - it's just a matter of the RRAS / ISA / IPSec network plumbing.
Since ISA isn't the endpoint when you publish the Back End server, the front end server can't see inside the encrypted data.
Have you implemented the AssumeUDPEncapsulationContextOnSendRule regitry key on the client? I'm not sure if Tom addresses this in the guide. MS introduced a securty restriction in XP SP2 that prevents NAT-T connectiosn from being established with servers behind a NAT device. It's not a bug as this was designed into the product, but you do have to implement the reg key on you clients.