nce upon a time I wanted to be able to VPN in to my home network. I researched a bit and discovered that my Windows XP Pro computer could be set up natively as a VPN server. I followed the built-in wizard, forwarded the appropriate ports on my router, and was up and running. But, the VPN connection only worked with the PPTP protocol and I wanted to be able to use L2TP/IPSec, because of its stronger security. The XP documentation says it supports L2TP, but it’s not so easy to set up, because of lack of documentation, and lack of default support for NAT traversal (apparently, Microsoft thought that this feature was a vulnerability, because they removed it by default in SP2.)

This is my most ambitious Technical Bedtime Story yet. The solution took days of googling and experimenting to get just right, but it works (by all means, please let me know if you know of a better way to do this). Here’s are the steps; I used Windows XP Pro as the VPN server and Windows Vista as the VPN client:

Create an Incoming Connection on Windows XP Pro

1. Go to Control Panel/Network Connections
2. Click on Create a New Connection
3. Select “Set up an advanced connection”, click Next
4. Select “Accept incoming connections”, click Next, Next
5. Select “Allow virtual private connections”, click Next
6. Check the user you want to be able to connect, click Next
7. Select “Internet Protocol (TCP/IP)”, click Properties
8. Check “Allow callers to access my local area network”
9. Select “Specify TCP/IP addresses”
10. Add two addresses from your local range, click OK, Next, Finish

Use Simple Authority to create a computer certificate on Windows XP Pro

1. Download and Install Simple Authority
2. Use Simple Authority to create a Certificate Authority (CA) and then a certificate. (It should put two certificates on your Desktop, with .cer and .p12 extensions)

Import the certificate on Windows XP Pro

1. Go to Start/Run
2. Type “mmc”, click OK
3. In the window that pops up, click File/AddRemove Snap-in, click Add
4. Select “Certificates”, click Add
5. Select “Computer account”, click Next
6. Select “Local Computer”, click Finish, Close, OK
7. Expand the Certificates folder
8. Right click the Personal folder, then All Tasks/Import, click Next
9. Click Browse and find the certificate you created (pick the certificate with the .P12 extension), click Open, Next
10. Put in the password you used when you created the certificate in Simple Authority
11. Check “Mark this key as exportable”, click Next, Next, Finish
12. Navigate to Personal/Certificates
13. You should see two certificates, drag the second one to Trusted Root Certification Authorities/Certificates (if you don’t do this you will get Error 789 when you try to connect)

Import the certificate on Windows Vista

1. Copy the certificate to the Vista machine
2. Repeat the above process (the dialog boxes look slightly different, but it’s close enough to the XP method)

Create a VPN client on Windows Vista

1. Go to Control Panel/Network and Sharing Center
2. Click Set Up a Connection or Network
3. Select “Connect to a workplace”, click Next
4. Click “Use my Internet connection (VPN)”
5. Type your publicly accessible hostname or IP address (the outside address of your router) in “Internet Address”, click Next
6. Check “Don’t connect now; just set it up so I can connect later”
7. Type your username and password,
8. Check “Remember this password”, click Create
9. Click Start Menu/Connect To, find your VPN connection, right click and choose Properties
10. Click the Security tab
11. Select “Automatically use my Windows logon name and password (and domain, if any)”
12. Click the Networking tab
13. In Type of VPN select “L2TP IPSec VPN”
14. Click IPSec Settings
15. Uncheck “Verify the Name and Usage attributes of the server’s certificate”, click OK (if you don’t do this you will get Error 835 when you try and connect)
16. Select Internet Protocol Version 4 (TCP/IPv4), click Properties, Click Advanced
17. Uncheck “Use default gateway on remote network” (this will create a split tunnel), Click OK, OK

Make changes to the registry on Windows Vista

1. Open regedit on the client computer
2. Navigate to “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent” (for XP clients it’s “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec”)
3. Add a new DWORD value
4. Rename it to “AssumeUDPEncapsulationContextOnSendRule”
5. Change its value from “0″ to “2″, close regedit

Forward ports on your router

1. Configure your router to forward the following ports to your Windows XP Pro computer: UDP 500, UDP 1701, UDP 4500

This may take a bit of extra monkeying around to work complelety. For instance, I had to delete and recreate the Incoming Connection a number of times. Good luck.

Sources: IPSec NAT-T is not recommended for Windows Server 2003 computers that are behind network address translators, L2TP VPN connection between 2 XP computers, How to Import a Server Certificate for Use in Internet Information Services 5.0, L2TP/IPSEC error 789: security layer encountered a processing error during initial negotiations, How to configure an L2TP/IPsec server behind a NAT-T device in Windows Vista and in Windows Server 2008, Using a Linux L2TP/IPsec VPN server with Windows Vista

UPDATE: I have found that there is some kind of glitch in Windows XP Pro’s “Incoming Connection”. Whether I am using PPTP or L2TP as the protocol, the VPN sometimes just stops working. I try to connect and it just won’t. If I delete and then recreate the Incoming Connection in Network Connections, it starts working again. Restarting the RRAS service doesn’t do the trick, only deleting the connection does. This usually happens after the computer hibernates and then wakes up. Something doesn’t get reset properly, ARP cache maybe? If you find a solution to this problem, please let me know.

---

Possibly related posts: (automatically generated)

• Windows XP Stops Recognizing DV Camera Via IEEE 1394 (Firewire)