A shift from printed documentation to electronic documentation of Microsoft software would seem to promise greater depth and currency, but the transition has been an uneven one. Some product groups continue to offer thorough, high-quality documentation, but others have shipped major products without fully documenting them, to the dismay of customers and partners. Customers find it difficult to evaluate and use products with poor documentation, while partners are less able to deploy, integrate, or customize it, although partners may also find new opportunities in consulting services that compensate for missing guidance from Microsoft.

The Role of Documentation

Documentation is used by many audiences to understand the various aspects of a product, including the general design and major features; deployment, installation, administration, and upgrade techniques; dependencies and conflicts with other software; and programmatic interfaces and built-in scripting languages.

End customers need documentation to determine what a product does and what business advantages they might gain from employing it, as well as how to use and administer the software after they have licensed and deployed it.

Microsoft integration partners need to understand how to customize or configure the software to meet specific requirements. Third-party developers want to use Microsoft's APIs, development tools, and software development kits (SDKs) to write complementary software that extends or enhances Microsoft's products. They also want to understand protocols used to pass information among programs or devices in order to build products that interoperate with Microsoft's software.

Documentation Deficiencies

Microsoft is not always meeting these requirements. Some major applications have shipped with inadequate, incomplete, or no documentation of important features or underlying technologies.

When BizTalk Server 2004 shipped in the spring of 2005, for example, one of its highly touted features, Human Workflow Services (HWS), was largely undocumented. HWS permits common client applications, such as Outlook or Word, to participate in business processes orchestrated by BizTalk or track the state and progress of those processes, but Microsoft documented how this could be done only with the relatively unused InfoPath client.

Similarly, Live Communications Server 2005, Microsoft's platform for business-oriented instant messaging, shipped with numerous undocumented features and administrative interfaces.

More recently, SharePoint Server 2007 shipped in the fall of 2006, but its documentation remains incomplete as of May 2007. According to the documentation that accompanies the SharePoint Server 2007 SDK, "This SDK release contains most of the highest-priority documentation for developing solutions on Office SharePoint Server 2007. Some of the SDK topics are more or less complete than others—the intent is to provide as much information as is currently available."

Some important features that were part of Microsoft's marketing for SharePoint Server 2007 were undocumented or poorly documented, leaving reviewers scrambling to find and work with the feature. For example, one highly touted feature, the Site Manager utility for managing large collections of SharePoint sites, was not mentioned in the product's technical documentation at release time (the feature apparently was renamed).

The unhelpful messages in the event logs in Microsoft products also continue to be a sore point with many administrators. Clicking on an error often produces a cryptic message with a Web link on it. Clicking on that link produces a help page that seldom sheds additional light on the problem. In many cases, it simply advises the customer to contact the vendor, a nontrivial task for many users; sometimes the vendor is Microsoft itself.

Changing Processes and Technology

Behind the documentation difficulties are some significant changes in Microsoft processes and technology.

In the company's first two decades, when all software shipped on disks and all documentation was on paper that accompanied those disks, the User Education specialists who wrote the documentation were part of the product teams. These specialists functioned as peers to developers and program managers who were responsible for technical development and a product's shipping schedule. They created documentation by reading product specifications, using the product, and reviewing documentation drafts with program managers. Problems with documentation were logged in Microsoft's bug-tracking databases—an error in documentation could hold up shipment as easily as a technical bug could. (They are still logged, but have less impact than in the past when determining whether a product is ready to ship.) In fact, the lead time required for printed books often meant that documentation was completed before the actual product.

Product shipping schedules have changed as well. A series of alphas, betas, technical previews, and release candidates—with key features sometimes being pulled at the final release candidate stage—has led some teams to decide that documentation need not be a priority until weeks before a product's general availability to customers. Given that shipping products on time is more critical in performance reviews than documentation, some product teams may assign a low priority to documentation in order to focus programming resources on getting code shipped.

Finally, the company has turned to new technologies to reduce the documentation burden. Web-based help (sometimes accessed from inside the program interface itself, such as the help menu inside common Office programs) permits Microsoft to deliver extensive help documentation on demand, tweaking and adapting help documentation in response to program changes (such as service packs) and new information (such as that recorded in the company's online Knowledge Base). However, while this data can be useful, search results sometimes return large amounts of irrelevant material.

The Web has also vastly expanded the resources available to developers. In addition to the Microsoft Developer Network (MSDN) and TechNet, which once shipped CDs on a regular basis to developers but are now available worldwide over the Internet, Microsoft sponsors special developer Web sites, such as Channel 9 (www.channel9.net) and CodePlex (www.codeplex.com).

SharePoint's documentation, although it remains incomplete, is updated in an ongoing schedule and includes a growing library of complementary articles and videos that implement what the product team calls a "See It, Code It, Read It, and Explore It" approach to developer instruction, which was not possible in the age of printed manuals.

Also, Microsoft source code licensing has been liberalized, enabling outside developers to discover the actual workings of poorly documented APIs, and leading to a proliferation of working source code that shows how APIs work. However, this source code is so far only widely available for peripheral products; code for important commercial products is often restricted to large customers, key partners, or academic use.

Finally, Microsoft product teams, and even individual program managers and developers contribute to blogs, giving the public unprecedented access to product team insiders.

Information Everywhere, but...

Unfortunately, these approaches do not add up to a consistent documentation effort. While some product teams (the SQL Server team is often cited as an example) consistently provide high-quality and timely documentation, others do not.

The company often points to blogs, wikis, and other online resources as an example of how it is helping customers obtain up-to-the-minute information. But these unconventional sources lack authority or disclaim it entirely: a note on a blog maintained by one Microsoft developer asserts "All posting and code samples are provided 'AS IS' with no warranties, and confers no right." In effect, customers are pushed to blogs and wikis and are then advised not to trust that information.

Furthermore, with documentation scattered across the Web, customers and partners have no single authoritative source, with a clear hierarchy or navigational structure, for all the known information about a given product. Specific fixes from the company's Knowledge Base are linked only sporadically to documentation, and the microsoft.com search engine often fails to uncover known bulletins. The likelihood that a customer will solve some problems depends on her skill with a search engine and the goodwill of non-Microsoft personnel who have either solved the problem on their own or who can point to Microsoft resources that don't show up in common searches.

Some of the documentation on MSDN is produced by partners or contractors who have indirect access to product teams and who many not have access to source code.

Future Dangers

Ultimately, poor documentation causes problems that may be invisible but real nevertheless. It reduces the speed with which customers can deploy products, the ability of partners who can integrate and extend them, and the likelihood that customers will use advanced features or upgrade to products whose features and requirements are poorly documented.

Unlike some of its competitors, which have large consulting forces that guide customers, Microsoft relies on partners, most of whom rely on its public documentation. In some cases, a partner may have access to an engineering team (if, for example, it has been selected to assist in a Technology Adoption Program, in which a small group of customers work with new products in the alpha stage of development), but such knowledge only serves some partners—who may guard it jealously, since it gives them a competitive advantage. Partners can use the general lack of documentation to their advantage, as well. Expertise in some Microsoft products may be more valuable because it is not widely available from other sources.

Blogs, though useful, may have the perverse effect of getting many key personnel involved in ad hoc documentation or customer assistance, actually increasing Microsoft's investment in documentation while doing little to improve its completeness or reliability.

One useful starting point would be to add stricter documentation requirements to Microsoft's Common Engineering Criteria (CEC), which provide "a common set of engineering requirements across the company's infrastructure server software products, in the effort to help customers improve reliability, security, and manageability while reducing total cost of ownership." While the CEC would seem to be an obvious place to set companywide documentation standards, the criteria today address only narrow documentation needs, such as a requirement that "nonreversible" changes, such as installations that modify existing data, should be documented, and that server documentation should be organized similarly for both online and printed material.

Meeting the CEC is not a mandatory requirement—a CEC report on Microsoft servers (itself not updated for more than six months, as of May 2007) identifies only 10 (of 23) Microsoft server products that have met any CEC requirements. Nevertheless, making adequate product documentation a CEC requirement would be a first step toward getting product teams across the company to consider it a feature of well-engineered software.

The company may be getting better in this regard. The requirement that online and printed documentation follow similar formats was instituted for Microsoft's 2007 product line, and the company says all nine products in that product line are meeting the new documentation requirements.

Resources

The Common Engineering Criteria can be found at www.microsoft.com/windowsserversystem/cer.

Expanded Microsoft source-code licensing is described in "Shared Source Licensing Simplified" on page 32 of the Dec. 2005 Update.