• Illustration: VPN Scenarios
• Sidebar: Internet Key Exchange
• Illustration: IPSec Transport Mode vs. Tunnel Mode
• Sidebar: Controlling IPSec Behavior Using Group Policy
• Illustration: A Typical Microsoft-Based VPN

This is the third in a series of articles on Microsoft technologies that use public key cryptography. The first two, "Windows Public Key Infrastructure Extends Security" and "Smart Cards Provide Stronger Log-On Security," were published in the Dec. 2001 Update.

Internet Protocol Security (IPSec) allows organizations to create virtual private networks that protect private communications sent over the Internet or other insecure connections. Included in Windows 2000 and its successors, IPSec is a relatively inexpensive way to guarantee the authenticity, integrity, and confidentiality of network traffic. In conjunction with related protocols, IPSec can also enable secure access to corporate resources by remote users. However, readers should be aware of some limitations of IPSec (particularly in organizations that use private network addresses) and be alert to interoperability problems between Windows IPSec and network devices from other companies.

When Are VPNs Needed?

Virtual private networks (VPNs) enable organizations to create their own "private" networks on top of exposed networks, such as the Internet and wireless local-area networks (LANs), that are potentially open to eavesdroppers or intruders. VPNs provide some combination of the following three functions:

Authentication. VPNs enable senders to prove their identities to receivers and vice-versa, preventing attackers from stealing data by impersonating users or masquerading as legitimate servers. Senders and receivers can be computers, network devices, or human users. Authentication also enables an organization to control which users can use the VPN, and can help track down rogue users and their computers after an attack.

Integrity. VPNs can ensure that data has not been altered in any way while in transit and is not a replay of an earlier transmission.

Confidentiality. VPNs can encrypt network data such that only the legitimate recipient device can decipher it.

There are three main scenarios for which VPNs are used. (For a graphical overview, see the illustration "VPN Scenarios".)

Connecting Remote Offices via the Internet

Rather than investing in expensive dedicated WAN links (such as frame relay or T-1 links) to connect remote offices to their enterprise networks, many organizations are cutting costs by using cheaper high-bandwidth Internet access, such as digital subscriber line (DSL) connections, instead. Using an Internet-based VPN is generally less reliable than traditional WAN links, but many organizations have found it good enough to meet their needs, especially when cost is factored in.

In this "one-to-one" scenario, both the central and remote office has a VPN gateway (a router or firewall) connecting it to the Internet (if required, a single corporate gateway could also connect multiple remote offices). Each gateway must first authenticate itself to a trusted partner gateway, and then they create a secure "tunnel" between the two devices. The integrity and confidentiality of all data is ensured while in transit through the tunnel. Although most organizations readily send plain text e-mail across the Internet, few are willing to send other types of content across the Internet unprotected.

Remote-office VPNs are completely transparent to users; the gateways do all the work and users have no indication that they are using a VPN to connect with a resource in another office (such as a file server). Other than on the gateway devices, no special VPN software is required on any of the other computers or routers.

Connecting Remote Users via the Internet

Most organizations no longer use dial-in modem pools and dedicated telephone lines to provide network access to remote users. Instead, they use VPNs to enable users to connect via the Internet to a gateway at the organization. This solution has several benefits:

• Remote users can connect to local ISPs rather than making long-distance calls to a corporate modem pool, thereby saving long distance or WATS phone charges.
• Remote users with broadband Internet connections can exploit their higher bandwidth when connecting to their organization’s network.
• The organization does not need to purchase and maintain modem pools capable of serving the maximum possible number of simultaneous dial-in users. They can accommodate peak loads without encountering a hard limit on the number of simultaneous connections.

This "one-to-many" solution serves home-based and roaming users alike, and organizations that support the remote office scenario can usually use the same VPN gateway and Internet connection to serve remote users. The gateway in this scenario must authenticate each individual user, and the VPN provides data integrity and confidentiality to all traffic between remote users and the gateway. However, the VPN is not transparent to remote users in this scenario—they must manually initiate VPN connections and log on before they can access network resources. All remote users must have VPN client software on their machines.

Securing Untrusted Private Networks

A third scenario arises when an organization's own networks might be open to eavesdropping or other attacks, often the case with wireless networks and LANs in unsecured environments such as universities.

In this "many-to-many" scenario, the organization creates a VPN involving all computers that must securely communicate over the untrusted network. The VPN must authenticate all participating computers as well as provide data integrity, and frequently confidentiality, to the traffic flowing between the systems. This form of VPN is also transparent to users, although they must still perform their normal log-on (authenticating themselves with their domain) before they can connect to server resources. All participating computers require VPN software.

Some in the industry, including Microsoft, disagree with usage of the term "VPN" to address this scenario, but no other industrywide term adequately classifies it either.

IPSec Improves on Past Technologies

Microsoft included IPSec in Windows 2000 to improve on the Point-to-Point Tunneling Protocol (PPTP), a VPN technology that was first included with Windows NT 4.0. Both IPSec and PPTP components install by default on all versions of Windows 2000 and its successors, but IPSec improves upon PPTP with better security, support for untrusted private networks, better manageability, and more interoperability.

Security. Even though Windows NT 4.0 Service Pack 4 fixed most of the serious security holes that plagued earlier version of PPTP, IPSec is still much more secure. PPTP authenticates remote users to an organization's gateway, but it does not care about the identity of the user’s computer, nor does it authenticate the gateway device to the user. With PPTP, network administrators cannot control which computers users can connect from, and users have no certainty that they have connected to the real gateway, rather than to an impostor controlled by an attacker. IPSec, on the other hand, ensures that all computers on a VPN authenticate themselves to other devices with which they communicate. This requirement reduces exposure to hackers breaking in from the Internet using weak or compromised user passwords, because the hacker’s computers cannot authenticate with the organization’s IPSec network.

IPSec can also help prevent and respond to denial-of-service attacks, which usually depend on flooding a host with anonymous IP packets. A server can be configured to accept only IPSec connections, and discard regular IP packets immediately, rather than depleting limited server resources by trying to process rogue data in higher-level applications like e-mail or Web server software.

Support for the untrusted private network scenario. Using PPTP for securing untrusted private networks is impractical; PPTP is a client/server protocol that would require configuring every server as a PPTP gateway, and users would need to manually initiate a separate PPTP session with each server before communications with it was protected. Automating PPTP-based server-to-server communications would be even more cumbersome. IPSec, in contrast, easily supports this scenario because it is a peer-to-peer protocol that does not care which device initiates a secure connection, and connections occur according to policy rules and without requiring user involvement.

Manageability. PPTP client software is not centrally manageable using Active Directory (AD) Group Policy. This means it is difficult to make changes to installed PPTP clients, and impossible to vary the settings by group, organizational unit, or site membership. IPSec, on the other hand, is centrally manageable using Group Policy.

Interoperability. Although Microsoft submitted PPTP to the Internet Engineering Task Force (IETF) for standardization, the protocol never gained broad adoption beyond Microsoft and the PPTP Industry Forum (which included 3Com, Ascend, ECI Telematics, and US Robotics). IPSec, in contrast, is an IETF standard (specifically, a subset of the IPv6 standard, but compatible with today's IPv4 networks) and has broad industry support from firms like 3Com, CheckPoint, Cisco, and IBM. Although not all vendors' IPSec software and devices seamlessly interoperate, some combinations do work today and the situation is likely to improve as IPSec adoption grows.

How IPSec Works

IPSec is a networking protocol that works in conjunction with a device’s normal TCP/IP protocol. IPSec operates on the network layer and processes incoming and outgoing IP packets. Because IPSec operates at the network level, it does not care who the user is or what application is communicating over it. When activated, IPSec steps in for the standard IP protocol to process most types of unicast (i.e., point-to-point) IP traffic according to rules specified by the system administrator. The exceptions are broadcasts, multicasts, and special types of unicast traffic, such as Kerberos authentication transmissions, needed by IPSec to establish itself. If an outgoing IP packet matches either one of its rules or one of the target IPSec device’s rules, it repackages the data as an IPSec packet. Routers can forward these IPSec packets to their destinations just like other IP packets, but IPSec packets provide two extra levels of protection for the repackaged network traffic:

Authenticated Header (AH). The IPSec AH protocol provides data integrity and computer-to-computer authentication by digitally signing the header of each IP packet. (Header fields that must change at each router, such as checksum and hop count values, are not included in the signature.) The receiving computer verifies the signature of each packet using a cryptographic signing key shared with the sending computer. If any bit in the signed portion of the IP packet changes before it reaches the recipient’s computer, the receiving computer will recognize the packet as invalid and discard it. Verifying the signature also authenticates the packet; no computer other than the legitimate sender holding the proper signing key could have sent it.

Note that AH does not provide confidentiality—the data in captured AH packets is readable unless encrypted by some higher layer security protocol, such as the Secure MIME e-mail protocol.

Encapsulating Security Payload (ESP). In addition to authentication and data integrity, the IPSec ESP protocol provides confidentiality. ESP encrypts each IP packet’s payload using an encryption key (shared by the sending and receiving computer) and sends it to its destination. The receiving computer uses its copy of the key to decrypt the payload, and then hands it off to higher layers in the protocol stack for processing. Any packet that does not properly decrypt is discarded.

ESP performs all of the functions of AH, so the two are not used together. AH exists because it is much faster than ESP—it only requires creating or processing a digital signature, rather than encrypting or decrypting the entire IP payload.

The signing and encryption keys needed to perform AH and ESP cryptographic functions are securely negotiated between the sending and receiving computers using a protocol called Internet Key Exchange (IKE). During the IKE process, computers authenticate each other and determine what keys and parameters they will use to process the IPSec packets. (See the sidebar " Internet Key Exchange".)

IPSec supports several different authentication methods and communication modes, giving administrators valuable flexibility for meeting the needs of the three scenarios discussed earlier.

Three Ways to Authenticate

To securely exchange keys, IPSec computers and gateways must authenticate each other. IPSec’s IKE protocol supports three ways to perform that authentication: pre-shared secrets, Kerberos, and digital certificates.

Pre-shared secrets. The simplest IPSec authentication involves preconfigured passwords known only to those two machines (and the person who configured it). However, this method has all of the problems associated with passwords, such as vulnerability to brute-force cracking programs. (For background, see "The Problem with Passwords" on page 8 of the Dec. 2001 Update.) Additionally, because each pair of devices requires its own password, it scales very poorly. If all IPSec devices need the ability to connect to any other IPSec device, then the number of passwords grows geometrically with the number of devices that need to communicate. (N devices require N*(N-1)/2 shared passwords, so a network with 50 IPSec devices that all needed to communicate with each other would require 1225 passwords!) This limits the usefulness of this method to remote office scenarios, where the number of gateways is small.

Kerberos. A new IETF draft-level extension to the IPSec protocol allows organizations to use Kerberos to authenticate IPSec devices to each other. However, this mode is only useable in the untrusted private network scenario, since all machines must be able to reach a Kerberos key-distribution server. (In Windows 2000, AD provides this service.) Because Kerberos is not a firewall-friendly protocol and organizations keep the key-distribution server hidden behind the firewall, it is not an acceptable authentication method for the remote-office or remote-user scenarios. To date, Microsoft is the only vendor to have implemented this extension, thus it only works between Windows-based systems.

Digital certificates. A far more flexible and scalable solution is to use a public-key infrastructure (PKI) to distribute digital certificates to all IPSec-enabled devices, which use the certificates to authenticate each other using public key cryptography. Because certificate-based authentication does not require a domain controller or other broker to perform the authentication function, it works well when the devices are exposed to the Internet. This solution also has the benefit of being much more secure, since no passwords are shared among devices. As long as the PKI supports a common trusted root Certificate Authority, certificates can even be used to authenticate interorganizational IPSec VPNs. (For more information, see "Windows Public Key Infrastructure Extends Security" on page 3 of the Dec. 2001 Update.)

Two Connection Modes

IPSec can run in two connection modes:

• Transport mode, in which the communication endpoints perform the IPSec signing and encryption functions.
• Tunnel mode, in which an intermediate IPSec gateway encapsulates all IP traffic and securely passes it to a partner gateway; this partner device strips off the protection and then forwards the original traffic to its ultimate destination.

Tunnel mode is typically used in remote office scenarios, where the only time the network traffic must be protected is when it travels over a network, such as the Internet, outside an organization's control. Transport mode is always used in the untrusted private network scenario, where a threat could come from anywhere on the network. Transport mode can also be used in the remote user and remote office scenarios when combined with the Layer 2 Tunneling Protocol (covered later in this article).

(For a graphical overview of the two modes, see the illustration "IPSec Transport Mode vs. Tunnel Mode".)

Implementing IPSec on Windows

Although the internal workings of IPSec are complex, Microsoft has significantly eased the difficulty of implementing it on Windows 2000 (and its successors). The IPSec protocol drivers install automatically on all servers and workstations, and administrators can use Group Policy to centrally configure IPSec and distribute the machine certificates needed for PKI-based authentication.

(See the illustration "A Typical Microsoft-Based VPN".)

IPSec Group Policy

Windows 2000 Group Policy normally is used to centrally control the IPSec options that apply to each particular computer in a Windows 2000 AD domain; otherwise, individual local policies control IPSec on a per-machine basis. All IPSec policy is per-computer; the identity of logged-on users has no effect on IPSec settings.

For example, administrators can use Group Policy to apply IPSec protection only to traffic that needs enhanced security, reducing the load on the machine. A computer might only use ESP encryption for traffic bound for certain servers, or that matches certain protocols, such as Simple Mail Transport Protocol (SMTP) e-mail or server messenger block (SMB) file and print services.

IPSec policy also determines which of the three authentication methods (preshared key, X.509 certificate, or Kerberos) should be used, and whether a machine connects in transport or tunnel mode.

Although the details of building an IPSec policy are complex, Windows 2000 comes with several template policies that administrators can use as a starting point when designing IPSec security policies. The Windows 2000 policy editor includes wizards and selection boxes that further ease the task of building policies. (For more information, see the sidebar "Controlling IPSec Behavior Using Group Policy".)

Certificate Auto-Enrollment

To save the time and trouble of manually installing the machine certificates needed by IPSec, administrators can use Group Policy to enable any or all Windows 2000 machines to auto-enroll themselves with an issuing certificate server belonging to the organization’s PKI.

This feature is beneficial to organizations dealing with untrusted private networks that want the additional security that certificates provide over Kerberos authentication. It can also be valuable in the remote-user scenario, but requires that each remote user’s computer be attached to the LAN during initial log-on so that the Group Policy can trigger issuance of the computer certificate. If this is not possible, then the user must manually request and install a certificate using some other means—by PPTP or by floppy disks, for example.

Hardware Support

Performing ESP encryption and decryption puts a heavy load on a CPU, especially on servers or remote access gateways where a single computer must process ESP connections from many different client computers. Fortunately, Windows 2000 supports network interface cards (NICs) containing IPSec security coprocessor modules that greatly accelerate processing and unburden the computer’s CPU. These NICs are available from 3Com, Intel, and others.

Layer 2 Tunneling Protocol

Although it may seem that IPSec alone could adequately address the needs of all three VPN scenarios, some limitations in the IPSec protocol require the use of another protocol, Layer 2 Tunneling Protocol (L2TP), to perform additional user-level authentication in all remote user situations and in certain remote office situations.

L2TP and Remote Users

As noted earlier, IPSec negotiates security connections between computers but does not consider the identity of the user. This approach is not sufficient for remote user access because organizations must be able to control and track remote access activities on a user-by-user basis. Furthermore, remote users need to dynamically acquire temporary IP addresses appropriate to their organization’s internal network at the time they connect to the gateway, something IPSec does not support.

To address these limitations, Microsoft and Cisco created L2TP to work in conjunction with IPSec. In Microsoft’s implementation, an L2TP remote user first establishes an IPSec ESP transport-mode connection with the remote access gateway, which then uses L2TP to authenticate the user’s credentials and build an L2TP "tunnel" connection between the two devices. Access is granted only if the user's account permits remote access.

L2TP has a few other benefits for remote users. The L2TP tunnel, in effect, makes it appear to the remote user’s computer that it is directly connected to LAN behind the remote access gateway and the computer is assigned an IP address appropriate to the portion of the LAN that the gateway connects to. The remote computer receives all broadcast and multicast traffic, and the L2TP tunnel even passes non-IP traffic between the remote machine and the internal network. This feature especially benefits organizations that use multicast-based audio/video conferences or that use the Novell IPX protocol to connect clients to NetWare servers.

Windows 2000 L2TP also supports the use of smart card–based digital certificates as user credentials, thus making the level of authentication and nonrepudiation even stronger. (For more information, see "Smart Cards Provide Stronger Log-On Security" on page 12 of the Dec. 2001 Update.)

L2TP in Remote Offices

Although IPSec Tunnel Mode was designed to serve the remote office scenario, L2TP’s ability to pass broadcast/multicast and non-IP traffic may make it the better choice in situations where computers on a remote LAN need to use those specific types of traffic.

To use L2TP to connect offices, administrators create "user" accounts for the gateways and configure them to automatically establish L2TP connections whenever traffic is addressed to devices on the other side of the tunnel. Implementing this solution requires many complex configuration steps, but once installed and tested it works silently and is transparent to users on the remote network. (See the "Resources" section for links to details on configuring this option.)

L2TP Product Integration

All Windows 2000 and Windows XP systems come standard with the client-side facilities to create L2TP connections. All Windows 2000 server products include a service called Routing and Remote Access (RRAS) that, among other functions, allows it to be an L2TP gateway to remote users and offices. The RRAS server issues a special L2TP-specific IPSec policy that overrides any other IPSec Group Policies in effect on the server or connecting clients.

Windows 2000 Server also provides a Connection Manager Administration Kit (CMAK), which allows RRAS administrators to create a self-installing .exe file that customizes and programmatically extends the VPN client functionality. The CMAK can prepackage a phonebook with dial and VPN access points. The phonebooks can be automatically updated whenever clients connect into the corporate network.

An RRAS server can also concurrently run Microsoft’s Internet Security and Acceleration (ISA) Server application, making one machine serve both as a VPN gateway and a firewall. In fact, ISA Server has a "VPN Wizard" feature that helps administrators properly set it up as an L2TP/IPSec gateway. (See "Comet Becomes Internet Security and Acceleration Server 2000" on page 9 of the July 2000 Update.)

Limitations of IPSec

Implementing IPSec in Windows 2000 was an important step in making Microsoft systems more secure, but IPSec is not for the faint-hearted. It is very complex and has some critical limitations that might limit its usefulness to many organizations.

Complexity

The sheer plethora of IPSec’s modes, options, and algorithms is daunting, and Microsoft is careful to stress that organizations should never simply implement one of the provided IPSec Group Policy templates. Implementing IPSec requires system administrators and network architects who have a firm grasp of networking and security concepts and who are intimate with the specifics of their organization’s networks.

Furthermore, an error in IPSec deployment could be disastrous. If an administrator issued an IPSec Group Policy affecting large numbers of computers and the policy effectively blocked all further network communications, the change could not be centrally undone because the systems could no longer receive a corrected Group Policy. It would require a visit to each physical machine to undo the damage.

For this reason, it is absolutely essential that administrators carefully test IPSec changes in a lab environment before making any changes to their production systems.

Because ESP hides the contents of the IP packets, analyzing and monitoring network traffic also becomes challenging—there is no way to categorize ESP traffic by protocol, and in tunnel mode, even the IP addresses of the sender or recipients are encrypted and thus unusable for troubleshooting.

Incompatibility with NAT Devices

Probably the biggest issue holding back IPSec is that it cannot traverse devices that perform Network Address Translation (NAT). In general, NAT devices are routers or firewalls that allow organizations to use private IP addresses (networks that use the 10.x.y.z, 172.16.y.z, or 192.168.y.z reserved addressing schemes) on their internal networks, yet provide connectivity with the public Internet. Individuals and organizations use these private addresses when it is too costly to obtain public Internet addresses (especially for remote sites or small home offices) or when they are simply unavailable. Most remote sites with DSL, ISDN, or cable modem broadband Internet connections are assigned only one public IP address by their ISP, and use NAT to share that address among multiple users. Most hotel-room broadband connections also use NAT (dial-up users are not affected, because their ISP dynamically assigns them a single public IP address for the session).

NAT-IPSec incompatibility arises because NAT devices make changes to the packets that make them appear as if they were tampered with, so IPSec drops them. (See the "Resources" section for several links that explain this issue in detail.) Microsoft and others are working on a standard IPSec extension that will work around the NAT traversal issue, but Microsoft has not announced how or when it will become available. For the time being, PPTP remains the only Microsoft VPN solution capable of traversing NAT devices.

In some cases, the problem can be avoided if the NAT function can be performed before traffic enters or after it leaves an IPSec-based tunnel. For instance, a remote access server (such as Windows 2000 running RRAS) having two network adapters, one with a public Internet address and the other on the organization’s private internal network, could both perform NAT and serve as an L2TP/IPSec gateway to remote users. Address translation would occur after packets were decoded coming out of the tunnel and before packets were encoded and sent into the tunnel.

Connectivity and Interoperability

Because IPSec is an Internet standard, the industry hopes that it will eventually reach the level of compatibility and interoperability of conventional IPv4. However, the IPSec standard is relatively new and is still evolving, so universal connectivity similar to what TCP/IP provides today is still years off, and may eventually occur simultaneously with the full adoption of IPv6. (See "Microsoft Pledges Support for IPv6" on page 11 of the May 2000 Update.)

Even within Microsoft’s own operating system family, IPSec is only compatible with Windows 2000 and Windows XP systems; Windows NT 4.0 and the whole Windows 9x family are limited to using PPTP or third-party VPN solutions.

Windows IPSec does interoperate with some third-party networking products, including the following:

Cisco Internetworking Operating System (IOS). Cisco routers running IOS version 12.0.7T (or later) support both IPSec and L2TP connections with Microsoft IPSec-enabled systems. This allows a Cisco router in a remote office to establish an IPSec or L2TP/IPSec tunnel with a Windows 2000 machine, or vice versa. Routers can use Cisco’s Simple Certificate Enrollment Protocol (SCEP) to obtain certificates from Microsoft servers (with the addition of the Windows 2000 Resource Kit CEPSETUP.EXE utility to a Windows 2000 Certificate Server).

Cisco PIX Firewalls. Windows 2000 and Windows XP remote clients can use IPSec/L2TP to connect to a Cisco PIX firewall running PIX version 6.0 (or later). PIX firewalls support Microsoft’s PPTP remote access clients as well.

Check Point Firewall-1. Although market-leading firewall vendor Check Point supports IPSec as its VPN solution and has announced its intention to interoperate with Microsoft’s IPSec, its current Firewall-1 product requires the addition of special software on remote clients or remote gateways.

Resources

For a good Microsoft white paper on IPSec, see "IP Security for Windows 2000 Server" at www.microsoft.com/windows2000/techinfo/howitworks/security/ip_security.asp.

For a good article on IPSec on Windows 2000, see "IPSec and L2TP Implementation in Windows 2000" at support.microsoft.com/support/kb/articles/Q265/1/12.ASP.

For additional information concerning IPSec compatibility with NAT, see www.sans.org/infosecFAQ/encryption/NAT2.htm and www.cisco.com/warp/public/759/ipj_3-4/ipj_3-4_nat.html.

For a good overview book on IPSec, see Doraswamy’s and Harkins’ "IPSec: The New Security Standard for the Internet, Intranets, and Virtual Private Networks" (ISBN 0130118982).

For a good overview book on L2TP, see Shea’s "L2TP: Implementation and Operation" (ISBN 0201604485).

For more information on Windows VPNs, see Chapters 6 and 7 from the "Networking Deployment Guide" found in the Windows 2000 Server Resource Kit at www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/windows2000serv/reskit/deploy/netdepl/ndgch06.asp and www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/windows2000serv/reskit/deploy/netdepl/ndgch07.asp.

For prescriptive information on configuring IPSec on Windows 2000, see "Step-by-Step Guide to Internet Protocol Security (IPSec)" at www.microsoft.com/windows2000/techinfo/planning/security/ipsecsteps.asp.