Skip navigation

Once upon a time I wanted to be able to VPN in to my home network. I researched a bit and discovered that my Windows XP Pro computer could be set up natively as a VPN server. I followed the built-in wizard, forwarded the appropriate ports on my router, and was up and running. But, the VPN connection only worked with the PPTP protocol and I wanted to be able to use L2TP/IPSec, because of its stronger security. The XP documentation says it supports L2TP, but it’s not so easy to set up, because of lack of documentation, and lack of default support for NAT traversal (apparently, Microsoft thought that this feature was a vulnerability, because they removed it by default in SP2.)

This is my most ambitious Technical Bedtime Story yet. The solution took days of googling and experimenting to get just right, but it works (by all means, please let me know if you know of a better way to do this). Here’s are the steps; I used Windows XP Pro as the VPN server and Windows Vista as the VPN client:

Create an Incoming Connection on Windows XP Pro

  1. Go to Control Panel/Network Connections
  2. Click on Create a New Connection
  3. Select “Set up an advanced connection”, click Next
  4. Select “Accept incoming connections”, click Next, Next
  5. Select “Allow virtual private connections”, click Next
  6. Check the user you want to be able to connect, click Next
  7. Select “Internet Protocol (TCP/IP)”, click Properties
  8. Check “Allow callers to access my local area network”
  9. Select “Specify TCP/IP addresses”
  10. Add two addresses from your local range, click OK, Next, Finish

Use Simple Authority to create a computer certificate on Windows XP Pro

  1. Download and Install Simple Authority
  2. Use Simple Authority to create a Certificate Authority (CA) and then a certificate. (It should put two certificates on your Desktop, with .cer and .p12 extensions)

Import the certificate on Windows XP Pro

  1. Go to Start/Run
  2. Type “mmc”, click OK
  3. In the window that pops up, click File/AddRemove Snap-in, click Add
  4. Select “Certificates”, click Add
  5. Select “Computer account”, click Next
  6. Select “Local Computer”, click Finish, Close, OK
  7. Expand the Certificates folder
  8. Right click the Personal folder, then All Tasks/Import, click Next
  9. Click Browse and find the certificate you created (pick the certificate with the .P12 extension), click Open, Next
  10. Put in the password you used when you created the certificate in Simple Authority
  11. Check “Mark this key as exportable”, click Next, Next, Finish
  12. Navigate to Personal/Certificates
  13. You should see two certificates, drag the second one to Trusted Root Certification Authorities/Certificates (if you don’t do this you will get Error 789 when you try to connect)

Import the certificate on Windows Vista

  1. Copy the certificate to the Vista machine
  2. Repeat the above process (the dialog boxes look slightly different, but it’s close enough to the XP method)

Create a VPN client on Windows Vista

  1. Go to Control Panel/Network and Sharing Center
  2. Click Set Up a Connection or Network
  3. Select “Connect to a workplace”, click Next
  4. Click “Use my Internet connection (VPN)”
  5. Type your publicly accessible hostname or IP address (the outside address of your router) in “Internet Address”, click Next
  6. Check “Don’t connect now; just set it up so I can connect later”
  7. Type your username and password,
  8. Check “Remember this password”, click Create
  9. Click Start Menu/Connect To, find your VPN connection, right click and choose Properties
  10. Click the Security tab
  11. Select “Automatically use my Windows logon name and password (and domain, if any)”
  12. Click the Networking tab
  13. In Type of VPN select “L2TP IPSec VPN”
  14. Click IPSec Settings
  15. Uncheck “Verify the Name and Usage attributes of the server’s certificate”, click OK (if you don’t do this you will get Error 835 when you try and connect)
  16. Select Internet Protocol Version 4 (TCP/IPv4), click Properties, Click Advanced
  17. Uncheck “Use default gateway on remote network” (this will create a split tunnel), Click OK, OK

Make changes to the registry on Windows Vista

  1. Open regedit on the client computer
  2. Navigate to “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent” (for XP clients it’s “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec”)
  3. Add a new DWORD value
  4. Rename it to “AssumeUDPEncapsulationContextOnSendRule”
  5. Change its value from “0” to “2”, close regedit

Forward ports on your router

  1. Configure your router to forward the following ports to your Windows XP Pro computer: UDP 500, UDP 1701, UDP 4500

This may take a bit of extra monkeying around to work complelety. For instance, I had to delete and recreate the Incoming Connection a number of times. Good luck.

Sources: IPSec NAT-T is not recommended for Windows Server 2003 computers that are behind network address translatorsL2TP VPN connection between 2 XP computersHow to Import a Server Certificate for Use in Internet Information Services 5.0L2TP/IPSEC error 789: security layer encountered a processing error during initial negotiationsHow to configure an L2TP/IPsec server behind a NAT-T device in Windows Vista and in Windows Server 2008Using a Linux L2TP/IPsec VPN server with Windows Vista

UPDATE: I have found that there is some kind of glitch in Windows XP Pro’s “Incoming Connection”. Whether I am using PPTP or L2TP as the protocol, the VPN sometimes just stops working. I try to connect and it just won’t. If I delete and then recreate the Incoming Connection in Network Connections, it starts working again. Restarting the RRAS service doesn’t do the trick, only deleting the connection does. This usually happens after the computer hibernates and then wakes up. Something doesn’t get reset properly, ARP cache maybe? If you find a solution to this problem, please let me know.

Advertisements

25 Comments

    • trevor
    • Posted June 16, 2009 at 6:10 AM
    • Permalink

    what ip address do you assign to your vpn clients??

    • rotwhiler
    • Posted June 17, 2009 at 8:34 AM
    • Permalink

    I just pick two ip addresses at the top of my LAN range that are not in my LAN DHCP pool in my router (like 192.168.x.200, 192.168.x.201). The default (if you skip steps 9-10 above) is for the incoming XP PC to assign from DHCP, but I find that it doesn’t release/renew properly and will eventually use up your whole pool.

    • Vahe
    • Posted December 14, 2009 at 11:30 PM
    • Permalink

    hi there, great article, thx. Not sure why but it looks like I am stuck on Error 768. I am using two XP (no vistas here), not sure if I am missing anything.

    P.S. I am using HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPsec (not PolicyAgent). Am I correc? And do I di this only on the client PC right? not the server.

    Thx,

    V~

    • colinp
    • Posted February 24, 2010 at 11:33 AM
    • Permalink

    Hi, have you solved this? I have exact same problem. I found that it is not necessary to delete Incoming Connections, it is sufficient to untick “Allow others to make private connections…”, click OK, then re-tick it.

    • rotwhiler
    • Posted February 27, 2010 at 11:23 PM
    • Permalink

    Colin, I never did solve this problem, I no longer have an XP machine. Sorry.

    • Mike
    • Posted November 5, 2010 at 7:10 AM
    • Permalink

    My god this works…Your instructions are so easy to read. Great stuff! Perhaps, the next easy to read and implement Technical Bedtime Story on how to make a million bucks. Lookin’ forward to it.

    • Ryan
    • Posted January 20, 2011 at 2:52 PM
    • Permalink

    Thank you so much! Was having an issue getting an L2TP connection setup in Windows 7. Luckily it was on the client end and your instructions helped a lot.

    Thanks,
    Ryan

    • d3v
    • Posted September 25, 2011 at 10:08 AM
    • Permalink

    Indeed a very informative article and i followed it word by word. I’m trying to connect windows 7/windows 2008 (clients) to connect to XP sp3 (VPN server) but getting Error 789. Is there’s any resolution to it that you guyz may know off? Please let me know. Thanks

    • Anonymous
    • Posted October 4, 2011 at 6:52 AM
    • Permalink

    This works, thanks you this saved a lot of time and effort

  1. Thanks its working on windows 7 to windows 7 and windows 7 to xp pro

    • Dan
    • Posted January 4, 2012 at 9:51 PM
    • Permalink

    Nice article. My XP firewall created exceptions for ports: UDP 1701, TCP 1723 and UDP 500. I only had to port forward TCP1723 in the router.

    • Anonymous
    • Posted February 8, 2012 at 1:55 AM
    • Permalink

    THANK YOU! THANK YOU! THANK YOU!

    I set up my XP box as a server and got a Win7 box logging in as a client first time with these instructions without having to mess with OpenVPN as an alternative. You are marvelous and if I ever meet you I will buy your beer all night.

    My next step is to try to get my Honeycomb tablet to use L2TP IPSec and the certificate I generated on my XP box. Please let there be a webpage as easy as this one describing that step…

    • Anonymous
    • Posted February 8, 2012 at 3:39 AM
    • Permalink

    Just to complete my comment above – I got my Honeycomb tablet talking to the XP VPN server I made in the comment above by the following:

    Get the .p12 file that you made using “Simple Authority” on your XP box to your android device ( either USB link or email to the android account and save the attachment ). I had to put the .p12 file in the top root level of my sdcard on my tablet for later steps to work.

    Next find “Credential Storage” in your Android device settings – depending on the android version and device it can be in “Security”, “Location”, “Location and Security” or the “Networks/VPN” settings. It may also be called something other than “Credential Storage” but you are looking for something along the lines of Certificates or Credentials.

    Once you have found the right setting area:
    Tick “Use Secure Credentials”
    Choose “Install from USB storage” – the android should immediately find your .p12 file ( if not, you may have to type the path to it – or check the .p12 file is definitely in the top level of your sdcard. Or the file may have to be renamed with a .pfx extension )
    The android will now ask for the password you used when you made the certificates on the XP box with “Simple Authority” – you did remember it didn’t you ?
    Finally – if you have not entered any certificates before on this device, then android will ask for a new password for certificate management – think of a password you will remember and repeat it.
    Android should then import the user and ca certificates from the .p12 file and save them on your device.

    Now when you go into the VPN setup on android you can choose to setup a VPN of type “L2TP/IPSec CRT VPN”:

    When you are entering the parameters for the VPN click on “Set user certificate” and android should open a list of certificates it has on the device and you should see the friendly name of the certificate you made/imported in the list. Click on it.

    Repeat for “Set CA certificate”

    Keep “Enable L2TP secret” unticked

    Enter a friendly name for the VPN ( in “VPN Name” ) and the IP address ( in “Set VPN Server” ).

    Choose “Save” from the android menu.

    Now you should be able to click on the VPN in the list of created VPNs and android will ask for a user name and password.
    Enter the username and password of the Windows user you are connecting as on the XP VPN server.

    It *should* then connect fine.

    • TheBigD
    • Posted April 4, 2012 at 10:11 AM
    • Permalink

    Great!, I managed to connect my android phone to a xp (embedded) machine in my home network. The not so great fact is that I seem to be able to connect even without using ip/sec. How do I make XP refuse connections that isn’t using IP/SEC?

    • Adarsh Shetty
    • Posted April 6, 2012 at 9:16 AM
    • Permalink

    As you said that there is a Glicth in Windows XP Pro’s incoming connections, use Windows XP proffesional instead as a VPN server….this will fix the issue

    • Anonymous
    • Posted August 9, 2012 at 3:28 AM
    • Permalink

    Does the same apply to setting up a LT2P/Ipsec on a windows 7 machine? Why is everyone still using windows xp?

    • Anonymous
    • Posted December 1, 2012 at 11:08 PM
    • Permalink

    How do you configure the L2TP VPN client on Mac OSX? The built-in Mac OSX client is asking for 2 things:
    – User authentication – I selected my certificate.p12, which was added to the KeyChain app
    – Machine authentication – this is the one I am not sure – what to add in here? It gave me 2 option: certificate or shared password

    • George
    • Posted January 28, 2013 at 5:15 AM
    • Permalink

    dude !

    i need you t help me out on this. I am trying to connect from an Xp Professional Client and the server is Xp Professional also. I issued the certificate imported it, created incoming connection. Then imported the certificate into Xp client also.

    When i tried to connect i get a 789 error. How can i resolve this ?

    • Ed
    • Posted October 15, 2013 at 11:09 AM
    • Permalink

    I have not been able to establish a L2TP/IPsec VPN connection using this guide. I think when people say they got it working they are really establishing a PPTP connection and not an I2TP/IPsec connection. The clue to this is that if forwarding port 1723 makes it work then you have a PPTP connection not L2TP/IPsec. It is my understanding that the only ports which need to be forwarded are related to IPsec and NAT-T. L2TP does not need to be forwarded because it is encapsulated in IPsec.

  2. It is working on Windows 8.1 to Windows 8.1 Thanks very much!

    • darin
    • Posted March 14, 2015 at 12:01 PM
    • Permalink

    Incoming connections properties

    “No hardware capable of accepting calls is installed.”

    What does this mean?

    • Anonymous
    • Posted March 25, 2015 at 9:43 AM
    • Permalink

    Is there a way to use self signed certificates instead of ca

    • mitetto
    • Posted May 31, 2015 at 6:43 AM
    • Permalink

    Yeppp, works as advertised guys. I was getting that 789 error at first but when i read again the tutorial i realized that i misplaced the certificates – “You should see two certificates, drag the second one to Trusted Root Certification Authorities/Certificates (if you don’t do this you will get Error 789 when you try to connect)”. For me it was the first certificate to drag.

    • Anonymous
    • Posted April 9, 2016 at 2:50 AM
    • Permalink

    I tried it and it didn’t work for me

    • Anonymous
    • Posted April 14, 2016 at 9:44 AM
    • Permalink

    Just followed your instructions and everything went smooth. Thank you so much, appreciate it.


4 Trackbacks/Pingbacks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: