• Sidebar: Vulnerabilities, Exploits, and Patches
• Illustration: Critical Patch Process

Among all the companies affected by security issues in Microsoft software, none may feel them more acutely than Microsoft itself. To protect its internal systems, Microsoft uses a combination of Systems Management Server (SMS), Windows Update, e-mail, and log-on scripts to distribute and install critical patches to clients on its internal corporate network and to ensure client PCs are running up-to-date software.

Although Microsoft’s IT groups face problems unique to software development companies, such as development labs running nonstandard client software, the processes and technologies these groups use to protect client PCs can be useful as a benchmark for other companies. Furthermore, understanding how Microsoft deploys SMS 2003 companywide could help other IT shops evaluate their own plans for the product.

How Microsoft Keeps Clients Secure

Two groups in Microsoft’s IT organization are responsible for keeping client PCs secure. The Corporate Security (CorpSecIT) group tracks and assesses the potential impact of vulnerabilities, software bugs that expose systems to attacks such as worms or viruses. The Global Client Software (GCS) group tests and distributes patches, code that fixes vulnerabilities. (For a review of patch-related terminology and concepts, see the sidebar "Vulnerabilities, Exploits, and Patches".) These groups work together to gauge user compliance with patch requests across the company and address noncompliance. These two organizations are highly placed in the company—they report directly to Microsoft’s Chief Information Officer, Rick Devenuti.

Like Microsoft's large business customers and peers, these groups face a daunting task in keeping pace with the frequent patches required for Microsoft desktop and server software products. The problem is exacerbated by both the scale and nature of Microsoft as a business: it employs more than 55,000 people in about 400 sites around the world. These users network more than 300,000 computers—many of which are not joined to one of the company’s official corporate domains and thus are difficult to manage.

The difficulties faced by the groups that manage patching internally give Microsoft development teams a first-hand look at this problem in the real world, which serves two main purposes. First, it helps Microsoft identify product requirements and features, such as patch management features in SMS. Second, it helps product groups recognize and take steps to minimize the pain their products inflict on IT groups. For example, based both on customer feedback and direct feedback from Microsoft’s IT organizations, the company has moved from a weekly patch distribution schedule to a monthly one.

(For an overview of the process by which Microsoft patches PCs, see the illustration "Critical Patch Process".)

Tracking and Assessing Threats

The CorpSecIT group learns of new vulnerabilities the same way as other IT security organizations: it monitors multiple sources, such as antivirus vendor sites, the CERT Coordination Center, and Microsoft’s own security-related sites, for security alerts, vulnerabilities, and the availability of patches. For threats not caused by flaws in Microsoft products (certain e-mail–borne viruses, for example), the group will assess and implement corrective measures (for example, notifying corporate clients to update antivirus signature files). For a vulnerability in a Microsoft product, the group assesses the potential impact of an exploit that targets the vulnerability, and then assigns the threat a level of critical, moderate, or low.

Critical. If exploited, critical vulnerabilities could result in Microsoft’s corporate security being significantly compromised. Such exploits involve one of what Microsoft calls the 3 E’s:

• Escalation of privilege—for example, code that can assign itself administrative privileges on a machine
• Expansion of scope—viruses that can infect other computers, for instance
• Exposure of confidential data, such as Windows source code or Microsoft financial results.

The Blaster worm exemplifies a critical threat—the worm could pass itself from one host to another, thus triggering concern about expansion.

Moderate. Potential exploits limited to individual machines or data on those machines, or that could target less widely used applications, are generally deemed moderate threats. Although such threats could result in degradation or denial of service, they are unlikely to be self-propagating or threatening to corporate security. A recent vulnerability, MS03-038, which could allow a hacker to run code on a site visitor’s machine if that machine ran a specific Microsoft Access component was deemed a moderate threat by CorpSecIT.

Low. These threats present little or no systematic risk to either corporate security or individual machine performance or data. Generally, vulnerabilities assigned to this level are simply bugs in a product.

Gauging Response

Microsoft’s internal response to vulnerabilities is determined by assigned threat level. Only for critical threats does the company require immediate user attention. In such cases, it aggressively advertises and distributes patches and sets enforcement deadlines for their installation. Prior to the deadline, corporate users are given the opportunity to patch voluntarily. Once a deadline has passed, the group takes action to force patch installation on noncomplying clients.

The enforcement deadline for critical patches is a judgment call made by CorpSecIT that takes into account the severity of a potential exploit and the likelihood that such an exploit will appear publicly. In most cases, the deadline is set at two weeks, although this deadline is abbreviated for more severe risks. The group also has procedures in place to take immediate action for severe exploits that pose an immediate threat.

For example, CorpSecIT gave corporate clients eight days to install the patch for the Blaster worm voluntarily and began forcibly installing it after that period; it also established a goal of reaching 99% voluntary compliance by the end of the eight-day period.

The patching of vulnerabilities deemed moderate or low-level threats is typically left up to the individual user. Although patches for such vulnerabilities may be advertised, the company does not force users to apply these patches on a set schedule. In most cases, moderate and low-level threats are addressed with the availability of service packs (SPs), cumulative roll-ups of patches released periodically by Microsoft product teams for both internal and external use.

Testing and Distributing Patches, Managing Compliance

When the CorpSecIT group has determined that a specific vulnerability represents a critical threat and has developed enforcement guidelines for the corresponding patch, it notifies the GCS group, which does a fast (ordinarily no more than several hours for critical threats) validation of the associated patch. This test pass is performed by members of the GCS team and small groups of volunteers around the company; the idea is to involve a wide enough spectrum of users to give some assurance that there won’t be installation or stability problems when the patch is rolled out to the broader user community.

GCS uses multiple mechanisms to distribute patches and enforce compliance; it groups these users into two general categories. All PCs that are joined to a domain managed by Microsoft’s IT organization are called managed clients: these clients are required to run SMS and are affected by corporatewide group policies. PCs that do not belong to a Microsoft IT-managed domain (for example, product development test teams sometimes manage their own domains) are called unmanaged clients—GCS cannot guarantee that unmanaged clients are running the SMS or Automatic Update clients.

Managed clients. Following patch testing, GCS distributes patches to managed clients using SMS. For most critical threats, CGS allows about 72 hours to test and distribute critical patches to users. GCS has begun to take advantage of patch management features in SMS 2003 to take inventory of patches on client machines, and to report and force patch installation on noncomplying clients. When an advertised enforcement date has passed, SMS silently installs the critical patch on noncomplying clients.

Microsoft also configures (via Group Policy) managed clients to poll Windows Update for updates every four hours. Updates are downloaded when available but are not automatically installed: it is up to the user to install a patch or update downloaded from the site.

Unmanaged clients. To make unmanaged clients aware of critical vulnerabilities, GCS sends corporatewide e-mail that includes information about the vulnerability, an internal link to the corresponding patch, and compliance deadlines. These notices are broadcast to all users (managed and unmanaged) of Microsoft’s corporate network, thereby providing a redundant reminder to managed clients.

Several times per day, the CorpSecIT group scans Microsoft’s corporate network for unpatched clients using a suite of in-house tools that mimic exploits of known vulnerabilities and allow CorpSecIT to probe unmanaged clients for compliance. When a critical patch enforcement deadline has passed, CorpSecIT sends a final warning in e-mail to noncomplying unmanaged clients. These clients are probed again in 15 minutes, and if they are found to still be vulnerable, they are forcibly removed from the network (for example, by shutting down the network port through which they are connected).

Microsoft continues to monitor managed and unmanaged client compliance to all critical patch updates between product SP releases. Once an SP has been released, the CorpSecIT and GCS groups advertise SP availability and track and enforce upgrade compliance employing a process similar to that used for critical patches. (However, SP update compliance deadlines are longer than those of critical patches—typically 21 days.)

Special Case: Imminent Threats

In certain extreme cases, a critical vulnerability may be escalated to the status of "imminent threat." An imminent threat could represent a critical vulnerability that an existing worm or virus targets, for example.

When an imminent threat has been identified, Microsoft dispenses with advertising deadlines and giving clients a period of time to voluntarily patch their systems. Managed clients that are connected to the network will be upgraded immediately and automatically via SMS. Unmanaged clients are sent e-mail notifying them that they have 15 minutes to upgrade, after which they are forcibly disconnected from the network.

For users that are not connected, log-on scripts will inspect their machine on their next domain log-on attempt and force an upgrade immediately if the machine has not already been patched. Additionally, Microsoft has begun to implement a feature in Windows 2003 called the RAS Quarantine Service (RQS), which performs similar checks on remote clients (such as those tunneling into the corporate network from home) before those clients touch Microsoft’s corporate network.

Next Steps

As Microsoft’s CorpSecIT and GCS groups further refine the processes used to keep corporate network clients patched, these teams will focus on the following:

Separating managed and unmanaged clients. Microsoft will use the IP security protocol (IPSec) to logically separate the company’s corporate network into a secure group of managed clients and a separate group of unmanaged machines. This separation will prevent unmanaged clients from making inbound network connections to managed clients. Although unmanaged clients may be able to access some noncritical intranet sites, they will be prevented from accessing systems containing key business data (Microsoft’s SAP system, for instance) or product development source code.

Using SMS 2003 exclusively. Today, Microsoft uses both Windows Update and SMS to advertise the existence of vulnerabilities and distribute the corresponding patches to managed clients. Windows Update can help infrequently connected remote clients stay updated and provides a measure of redundancy for clients that are connected to the corporate network. However, running both mechanisms is also a source of confusion and annoyance for many users, and a source of network inefficiency, as the same patches are distributed to users by SMS and Windows Update. With the rollout of SMS 2003, Microsoft will phase out the use of Windows Update for managed clients, although it will continue to use e-mail to reach unmanaged clients and as a redundant notification method for managed clients. With the improved software distribution capabilities, enforcement management features, and better client configuration checking in SMS 2003 coupled with the improved perimeter security afforded by RQS, Microsoft feels it is sufficiently covered without continuing to use Windows Update for managed client patching.

Reducing time to compliance. As the writers of viruses and worms hone their craft, the time between discovery of a vulnerability and code that targets it steadily decreases. To combat this ever-decreasing window, Microsoft’s IT groups continue to push for speedier patch testing and distribution, and faster and more complete voluntary compliance among corporate network users. Whereas the group was able to push network users to 99% compliance in eight days for the Blaster worm, it hopes to achieve 99% compliance in five days for comparable future threats. Additionally, the company intends to reduce the amount of time for patch testing and distribution from 72 hours today to between 24 and 48 hours by the end of 2004.

Resources

Microsoft provides a central resource for Security information, including links to recent security bulletins and patches at www.microsoft.com/security/.

General recommendations on Microsoft products, tools, and procedures for patch management are at www.microsoft.com/technet/security/topics/patch/secpatch.

Case studies of Microsoft’s internal IT methods and practices can be found at www.microsoft.com/technet/itsolutions/MSIT/Security/mssecbp.asp.