By Michael Cherry [bio]
Posted: Feb. 10, 2003

• Sidebar: How Slammer Works
• Sidebar: Microsoft Baseline Security Analyzer: Indeterminate Results

The following is the full text of an article published by Directions on Microsoft, an independent research firm focused exclusively on Microsoft strategy & technology. Each month we make one or more key articles available to non-subscribers.

Just as 2001's Code Red worm forced the Windows product group to seriously reexamine its approach to security issues, the Jan. 25 Slammer worm has served as a wake-up call for Microsoft's SQL Server product team. Slammer points out the difficulty in driving changes to development practices across a company as diverse as Microsoft, and leaves customers with the concern that each product group might have its own crisis before customers see the benefits of Microsoft’s security focus.

A Framework for Measuring Progress

The timing of the Slammer attack was particularly unfortunate for Microsoft, occurring days after Chairman and Chief Software Architect Bill Gates sent an e-mail to customers entitled "Security in a Connected World," and Mike Nash, corporate vice president of the Security Business Unit, gave a speech entitled "Microsoft Security Customer Road Map 2003" as part of Microsoft’s Silicon Valley Speaker’s Series. These events marked the one-year anniversary of Trustworthy Computing, a companywide initiative to improve the security, privacy, reliability, and business integrity of Microsoft and its products.

To help product teams address security, Craig Mundie, Microsoft's chief technology officer for advanced strategy and policy, has articulated a framework with the acronym SD3+C: secure by design, default, and deployment, plus communications. Mundie and security strategist Scott Charney evangelized SD3+C to Microsoft’s product groups and customers as part of the Trustworthy Computing initiative, and the Windows product group used the SD3+C framework to review its practices and products—evidence of which can be seen in the improvements to Windows Update, the Software Update Service, and Windows Server 2003 (the next version of Windows due for release).

Although the SQL Server team could have also used the SD3+C framework, the Slammer worm shows they had not made similar progress. While SD3+C provides framework for managing security programs, it can also be used as a framework for understanding what went wrong, and in the case of Slammer it shows a clear failure in three of the four areas. The framework also provides clues as to where the company as a whole has more work left to do.

Secure by Design

The goal of "secure by design" is to reduce the number of security bugs or vulnerabilities that are present in new software. To accomplish this goal, security must be a factor in all phases of product design, from creating the specification, through writing the code, and testing the product. A secure-by-design philosophy manifests itself as security training, code reviews and walkthroughs, threat analysis, and friendly hacking of both currently shipping and future products.

What Went Wrong?

Microsoft product groups, including Windows (both client and server versions), tools (Visual Studio), and enterprise servers (such as SQL and Exchange) have halted work on the next versions of their products to conduct training and code reviews and to update their specifications and plans for security by design. Publicly, Microsoft has measured the impact of its secure-by-design efforts by presenting metrics such as the number of personnel trained, the number of weeks the review took, and the dollar cost to Microsoft of the training, review, and delay to product schedules and delivery dates. (For more information on these metrics, see the link to Gates’s e-mail "Security in a Connected World" in the Resources section below.)

The problem is that customers do not know how to take advantage of the product group’s secure-by-design effort. Little is known about any vulnerabilities that the reviews uncovered and whether recent service packs, such as SQL Server 2003 Service Pack 3 (SP3) contain patches for these vulnerabilities. In addition, Microsoft is not discussing any architectural changes it will be making as a result of these reviews. For example, the review of SQL Server uncovered the fact that it is hard to patch SQL Server, but Microsoft is not forthcoming about any plans or schedule to improve this—will customers have to purchase the next version to obtain a patchable product?

What Remains to Be Done?

Customers could make informed decisions on products and service packs if they knew the number of security vulnerabilities the reviews uncovered and which existing product service pack(s) contain the patches. Microsoft does not have to provide information that a hacker could exploit, but in the absence of basic information, customers cannot gauge the importance of rolling out a given service pack and may delay its deployment. Such a delay could leave vulnerabilities open to exploitation in the future (perpetuating the problem of the Code Red and Slammer worms, which exploited known bugs for which patches were available, but that many organizations had not deployed).

Secure by Default

The goal of "secure by default" is to create default product installations and configurations that are more resistant to attack, by reducing the attack surface (the number of points a hacker can attempt to exploit). To accomplish this goal, software must be installed in its most secure configuration and must stay that way until the customer takes informed steps to loosen it.

What Went Wrong?

Traditionally, Microsoft’s approach was to make all product features easily discoverable and functional the first time the product was used.

The Slammer worm exploits a known buffer overflow in the SQL Resolution Service, a service that is only needed when there are multiple instances of SQL Server running on the same machine—not a common default. (For technical details on the Slammer worm, see the sidebar "How Slammer Works".)

A secure-by-default approach would have not activated this service until a second instance of SQL Server was installed. This would have dramatically reduced the number of systems that the worm could enter and use to propagate itself.

The SQL product group should have been able to learn from their peers in Windows, who have already begun to adopt the secure–by-default framework for Windows Server 2003. The Windows product group shipped Windows XP with a new Universal Plug and Play (UPnP) service that was turned on, even though at the time no hardware was available to take advantage of the UPnP service. Shortly after the release of Windows XP, a buffer overflow in the service created concern about the security of deploying Windows XP in its default configuration. Having learned a valuable lesson, the Windows team has now turned off many services by default in Windows Server 2003.

What Remains to Be Done?

Microsoft has acknowledged that the SQL Resolution Service could have been disabled until there were multiple instances of SQL Server, and that this approach will be used in the future.

In the meantime, Microsoft could make it easier for SQL Server administrators to disable the SQL Resolution service and any other features and services that are enabled but not frequently used. It could provide a SQL Server "lockdown" wizard to help administrators identify which of SQL Server's services and features they need on a particular computer and turn off those not used. Such a tool would be similar to the Internet Information Service (IIS) Lockdown Wizard released by the Windows product group.

Secure by Deployment

The goal of "secure by deployment" is to ensure that deployed software remains free from known vulnerabilities or security weaknesses. To accomplish this goal, Microsoft must respond effectively to new threats or newly discovered vulnerabilities—for example, by issuing appropriate and easy-to-deploy patches in a timely fashion.

What Went Wrong?

Slammer really exploited weaknesses in SQL Server’s security by deployment—SQL Server is hard to patch, and Microsoft provided few tools to help administrators find servers that needed patching.

SQL Server is hard to patch. Microsoft had released a bulletin and patch six months ago that addressed the vulnerability that the Slammer worm exploited, but as the impact of the worm illustrates, few organizations, including Microsoft (which suffered internal outages because of the worm), had managed to apply the patch to all of their computers running SQL Server 2000 and the Microsoft SQL Server Desktop Engine (MSDE), a redistributable version of SQL Server that ships with many Microsoft and third-party products, including versions of Office.

SQL Server is one of the most difficult Microsoft products to patch. For example, applying one security patch required the user to download the patch, decompress the files into a directory, and then complete several steps in the proper order:

• Apply Service Pack 2 (SP2)
• Manually back up 27 old files
• Manually copy new versions from the directory where they were decompressed to the correct SQL Server directory
• Run two SQL queries and one additional executable file.

Correctly applying SQL Server patches is often time consuming, and an administrator will sometimes encounter undocumented problems, such as instructions to replace a file that does not exist on his server, or errors, such as when SQL script code included in a patch fails to execute completely. Furthermore, there is no easy way to automate the distribution of such complex patches; patches often require a system reboot (especially problematic for production systems); and only a fraction of patches ship with the ability to roll back in the event of installation failure or other unanticipated problems. Together, these factors make many administrators reluctant to apply patches to critical production SQL Server systems.

Microsoft might justify such convoluted patching steps if SQL Server ran only on servers for which administrators are responsible. But thanks to MSDE, which ships in Office and Visual Studio, relatively unsophisticated end-users have instances of SQL Server running on their desktops, making it extremely unlikely that all instances will be patched correctly.

MSDE uses the Windows Installer for installation and patching, and although this can simplify the initial installation of MSDE, it does nothing to simplify the application of patches or service packs. To apply an MSDE service pack, the user must determine which of 16 different Windows Installer (MSI) files was responsible for the installation(s) of MSDE on their desktop. (Prior to MSDE Service Pack 3, which was released only after the Slammer worm wreaked havoc, determining the MSI version required the user to check the Registry to see which SQLRunxx.MSI provided the installation.)

No tools to track servers and patches. Organizations cannot easily track the number of SQL Server installations because it is used for many different purposes: it can be a database engine on its own; a prerequisite for other products, such as Application Center; or, in the case of MSDE, a desktop service installed by other applications. (Instances of MSDE on user desktops are the most likely reason that Microsoft itself was hit by the bug.)

Prior to the Slammer worm, there were no security or patching-related tools, other than the Microsoft Baseline Security Analyzer (MBSA), to help administrators identify SQL Servers running in their organization and determine what patches had been applied to those servers. MBSA helps customers identify common security misconfigurations in Windows NT 4.0, 2000, XP, IIS 4.0 and 5.0, SQL Server 7.0 and 2000, Internet Explorer (IE) 5.01 and later, and Office 2000 and 2002. MBSA will also scan for missing security updates for Windows NT 4.0, 2000, XP, IIS 4.0 and 5.0, SQL Server 7.0 and 2000, IE 5.01 and later, Exchange 5.5 and 2000, and Windows Media Player 6.4 and later. While MSBA was available, its results are often suspect or inconsistent, and because the free tool was developed by a third-party (Shavlik Technology, which sells a more robust version), Microsoft’s commitment to the tool is unclear.

What Has Microsoft Done?

Microsoft has begun to address the complexity of SQL Server patches. In the weeks following the initial Slammer outbreak, Microsoft provided a "SQL Server Critical Update Kit" with the following tools:

SQL Critical Update. This update includes the tools to scan computers for instances of SQL Server 2000 and MSDE 2000 that are vulnerable to the Slammer worm and updates the affected files (the SQL Critical Update includes two SQL Server tools, SQL Scan and SQL Check).

SQL Scan. This tool (Sqlscan.exe) scans an individual computer, a Windows domain, or a range of IP addresses for instances of SQL Server 2000 and MSDE 2000, and identifies instances that might be vulnerable to the Slammer worm.

SQL Check. This tool scans the computer on which it is running for instances of SQL Server 2000 and MSDE 2000 that are vulnerable to the Slammer worm. SQL Check also identifies vulnerable SQL Server 2000 clusters but does not disable them. SQL Check runs on Windows 98, Me, NT 4.0, 2000, and XP. On computers running NT 4.0 or greater, it stops and disables the SQL Server and SQL Agent services. On computers running Windows 9x it identifies vulnerable instances but does not stop or disable any services.

What Remains to Be Done?

The SQL Server Critical Update Kit begins to address patch complexity and the lack of tools, but Microsoft will have to develop these rudimentary efforts into more sophisticated and wider-ranging tools. For example, the update kit appears to make it easier to install the Slammer vulnerability patch, but other patches and service packs remain very difficult to install. In addition, the existing tools have difficulty determining the exact status of a server and the state of patches. For example, administrators have two different ways to determine the state of service packs on SQL Server—they can run MSBA on the server or they can create a query that calls the version-stored procedure (@@version). On some servers, where the administrator has installed SP2, MBSA will report that the latest service pack applied is SP1. Running the version-stored procedure as a query on the same MSDE instance indicates that SP2 is installed. Such results leave administrators doubting the reliability of MBSA.

Communications

Security improvements, patches, and information are useless if customers don’t know about them. Communication requires getting accurate bulletins quickly to customers when vulnerabilities are discovered, helping customers understand which bulletins apply to the systems they have deployed, and providing updated information and best practices that evolve in response to threats and changes in technology.

What Went Wrong?

Microsoft’s communications about Slammer failed twice, first when the initial patch was release, and then when the Slammer outbreak began.

The initial failure. The initial security bulletin about the SQL Resolution Service vulnerability did not adequately identify which systems were affected (for example, there was no list of products that might have installed MSDE) or provide information about turning off the communication ports used by the SQL Resolution Service until the patch could be applied.

The second failure. When Microsoft acknowledged the Slammer worm’s outbreak, its initial communications were defensive, stressing that Microsoft had already released a patch and implying that the fault was with customers failing to apply the patch, rather than prescriptive (telling customers to turn off the affected ports). Instead, it pointed customers to their virus software vendor for additional information.

What Remains to Be Done?

During Feb. 2002 Microsoft updated information about the Slammer worm and, as already noted, provided improved patches and rudimentary tools.

Microsoft will need to improve its bulletins to ensure that information is timely, that it communicates the importance of applying the patch, and that it gets to the correct audience. In particular Microsoft needs to be clear about product dependencies. For example, many enterprise servers rely on other services, such as IIS or SQL Server, and relationships between products must be clear so that administrators apply all necessary patches to all affected products.

The Top Security Challenge: Consistency

Since acknowledging the need for Trustworthy Computing and beginning down the long road to delivering on the initiative, Microsoft has made progress. The company is showing that it understands the basic issues underlying Trustworthy Computing, and it has put the SD3+C framework in place to focus its security efforts. This is best illustrated by the progress shown by the Windows product group; since the crisis around the Code Red worm in 2001, the Windows product group began to completely reassess the security of its products, using the SD3+C framework.

The SQL Server product team has now had a similar crisis with the Slammer worm. And that illustrates the biggest security challenge Microsoft faces: consistency across all the product groups. Because of the autonomy granted to product groups, customers are not seeing consistent companywide progress toward SD3+C. Microsoft must prove that each product team does not need its own crisis that negatively impacts customers before it demonstrates progress in security for its products.

Security-related issues requiring consistency include standardizing on a patching technology, a single Microsoft Update site, life-cycle tracking of security-patched files, complete bulletin information, and service packs without new features.

Standardized patching technology. Product groups use multiple patching methods, including executables that are self-extracting and self-installing, executables that are self-extracting but require manual installation, and the Windows Installer patch (.MSP) files. Microsoft says a committee has been formed to examine this problem.

Microsoft Update. Administrators and users must go multiple places to find patches, including the separate Windows and Office Update sites and individual download sites for each server. It seems reasonable that a single Microsoft Update site could offer all critical downloads.

Life-cycle tracking of files. Microsoft does not appear to be tracking updated files through multiple patches. For example, when a file requires a security patch, the developer of the security patch provides information about the patched file to the MBSA so that the MBSA can detect the security patch when it scans the file. But if a non-security patch is made to the same file (such as a fix for a critical non-security bug), then it appears that no update is made to the MBSA. As a result, the MBSA reports that it has detected file versions which are greater than it expected. An administrator reading such a report does not know how to interpret the results. Does it mean that the system is secure, but a non-security patch has been applied? Or does it mean that someone has placed a Trojan horse or virus in one of the files and did not know to disguise the change from MBSA?

(For an example of such a report, see the illustration "Microsoft Baseline Security Analyzer: Indeterminate Results".)

Complete security bulletins. Microsoft needs to ensure that security bulletins, regardless of the product group, contain the same set of essential items. For example, not all bulletins have a full date and revision history: some just have a "last reviewed date" or "originally posted." All bulletins need a revision history with an explanation of what changed between releases. This is especially important in the case of a Slammer-like incident in which Microsoft is frequently updating previously released bulletins. Bulletins should also clearly indicate when a patch is contained in a later released rollup or patches.

Feature-free service packs. The debate continues on whether a service pack should contain both fixes and new features. Microsoft says that most customers want service packs with features and do not want to wait for the next release if a compelling new feature is available. But other customers are reluctant to install a service pack, even if they need the security and other bug fixes, until they have had a chance to evaluate their need for and the stability of the new features. Given its security problems and customers’ reluctance to install needed patches and service packs, Microsoft should consider erring on the side of being consistent, by keeping service packs for bug fixes and creating feature packs to ship new features.

What Should Customers Do?

Customers will always have to be vigilant, but until Microsoft has achieved a companywide, consistent approach to vulnerability checking and patching, vigilance will be critical. Customers might find that they can use the SD3+C framework within their organization to guide their security efforts. For example, customers can use secure-by-design concepts to look at in-house applications for any vulnerabilities; secure-by-default practices to examine installed software and ensure that only essential products, features, and services are used within their organization; and secure-by-deployment procedures to ensure they know which patches they will deploy and what it will take to deploy them.

Finally, customers should consider whether they have other products like MSDE that might have been deployed without the knowledge of information technology personnel. For example, many organizations, including Microsoft, had MSDE installations that were unknown to their information technology team and that were not monitored and updated in a timely manner. Such products create a large attack surface for the next worm.

Resources

For more information on Trustworthy Computing, including the role of the customer in ensuring security, see the Sept. 2002 Research Report "Trustworthy Computing: Making Software More Secure."

Bill Gates's e-mail "Security in a Connected World" is at www.microsoft.com/mscorp/execmail/.

The latest information from Microsoft on SQL Slammer is at www.microsoft.com/security/slammer.asp.

The complete list of Microsoft products that ship with MSDE is at www.microsoft.com/technet/treeview/?url=/technet/security/MSDEapps.asp.

The original Security Bulletin about the vulnerability that the SQL Slammer worm exploits is at www.microsoft.com/technet/security/bulletin/MS02-039.asp.

Service Pack 3 for Microsoft SQL Server 2000 and MSDE 2000 is at www.microsoft.com/sql/downloads/2000/sp3.asp.

A detailed technical analysis of the Slammer worm has been produced by the Cooperative Association for Internet Data Analysis and others, and can be found at www.caida.org/analysis/security/sapphire.