|
|
|
||||
| Shipping Software: The End Game Revisited | ||||
|
||||
|
By Greg DeMichillie [bio]
The following is the full text of an article published by Directions on Microsoft, an independent research firm focused exclusively on Microsoft strategy & technology. More samples of our content, as well as a list of upcoming articles and reports are also available. Microsoft has been slowly moving to a new development process that will affect how partners and customers evaluate and test its software. The new process replaces a series of relatively infrequent alpha and beta releases with more frequent Customer Technology Previews (CTPs). The new process should help Microsoft gain more feedback earlier in the development cycle, but it won't necessarily help the company ship its products on time or with fewer bugs. CTPs Began in Developer Tools As part of the development process for Visual Studio 2005, Microsoft made available to developers a series of prebeta builds that it called CTPs. CTPs gave customers access to the product at an earlier stage of development than was available with any previous version of Visual Studio. The Beta Feedback Problem CTP releases help Microsoft solve a nagging problem of software development: gathering feedback on the product while there is still time to do something about it. Under the traditional model of software development practiced by Microsoft and most large software developers, builds of a product are not made broadly available until the "beta" stage, when all of the features have been designed and implemented and the product has reached a reasonably high level of stability. (For a breakdown of the traditional stages of development, see "Traditional Phases of Software Development".) But by the time a product has reached a beta release, it is very difficult for a development team to make significant changes to a product's design without significantly delaying the release. Former Microsoft manager Jim McCarthy describes the role of a beta in his book Dynamics of Software Development, which is based on McCarthy's own experiences in managing a large Microsoft development project: "There is a nearly universal misconception that Beta testing is for soliciting input on the design and implementation of the features in the product. Nothing could be further from the truth.... Although opinions from Beta testers are always interesting, unless the feedback is overwhelmingly calamitous, no genuine changes except those that fix configuration problems are warranted." CTPs for Earlier Feedback Although software developers sometimes make prebeta builds available to highly trusted partners and customers, they have been reluctant to make them broadly available, usually out of fear that the incomplete and buggy nature of prebeta builds would scare customers away. With Visual Studio 2005, Microsoft took the risk that the product's core audience of software developers would understand the nature of prebeta software and see the release for what it was—an early build designed to give Microsoft the opportunity to gather feedback while it still had time to act upon it. To further help customers provide feedback, Microsoft provided a publicly available database that any customer could use to record bugs in the product, identify features that needed to be reworked, suggest new features, and vote on the priority of bugs. This database was monitored by Microsoft's development teams and gave them unprecedented insight into the concerns of their developer customers. Practice Building and Shipping In addition to providing a vehicle for customer feedback, the CTP releases of Visual Studio 2005 gave Microsoft practice in building and shipping a product. Most of Microsoft's important products are large-scale software projects involving hundreds, if not thousands of developers, and thousands of modules of code produced by multiple development teams. In many cases, the seemingly simple task of building the project (creating a runnable copy of the software) can itself be a major undertaking. The correct version of each of thousands of source code files must be collected on one machine and then compiled (translated from text into binary code) in the correct sequence and assembled into the many executables, DLLs, and .NET assemblies that make up the final program. But building the product is an unglamorous, and often thankless, task. Unless a team is diligent about making sure the product builds correctly on a regular (hopefully daily) basis, and makes preventing and fixing build errors a top priority for its developers, product builds can fall behind, making it difficult to get the product into customers' hands. McCarthy describes it this way: "A frequent, public build exposes the real (vs. the imagined) dependency tree. The product just plain won't build if everything isn't properly lined up.... And build problems force the team to confront issues they'd prefer to ignore." A public commitment to producing CTPs on a regular basis can therefore act as a "forcing function" for a team and cause it to fix build problems that it might otherwise ignore in the name of expediency. A team that is able to regularly produce CTPs is a team that has solved the "build problem." Risks of CTPs Although the use of CTPs has so far been beneficial, Microsoft and its customers face some risks, particularly as the company employs CTPs in later parts of the development cycle. Can CTPs Replace Betas? After the Visual Studio team's success, the CTP model has been adopted by other teams within Microsoft—first for products focused on developers, such as SQL Server 2005, the Windows Presentation Foundation (WPF), and Windows Communication Foundation (WCF), and later on for Windows itself. But the biggest change in Microsoft's use of CTPs came in early 2006, when it announced it would not be releasing further betas of Windows Vista and would instead rely on a series of CTPs. Although more frequent releases could help Microsoft achieve the broad hardware and software compatibility testing required by an OS, they also could make it harder for Microsoft to meet its overall quality and schedule goals. Each CTP release requires some level of testing to make sure the build meets a set of basic quality requirements. Even highly organized and well-functioning teams are extremely unlikely to produce daily builds that can simply be handed over to customers—a testing and stabilization period must take place. During these stabilization periods, developers are allowed to fix only the most serious bugs because any change to the source code, even one made to fix a bug, runs the risk of introducing new bugs or reintroducing old bugs. McCarthy describes this process as "Don't Shake the Jell-O": "Shipping a product is like watching a large-sized serving of quivering Jell-O. Gradually, the Jell-O slows its vibrations. But then you fix a bunch of bugs, and it starts quivering frenetically again." With Visual Studio 2005, Microsoft used CTPs only in the early phase of the development process and made it clear to customers that each CTP received less testing and stabilization than it would have received with the previous development process—in essence, trading depth of testing of each release for increased frequency of releases. Since the purpose of the early CTPs was to gather feedback, the tradeoff was worthwhile and helped Microsoft's efforts to ship the right product. But the beta process has a different goal—to stabilize and ship the product as it is currently defined and to make sure the product works correctly on the wide range of installed hardware and software. Microsoft claims that it has reworked its internal processes, particularly in the Windows team, to allow it to produce CTP releases more frequently without compromising the overall product quality or causing it to further delay its already-late schedule. But it has provided no details on that process and is in uncharted territory: no product as large as Windows has completely abandoned the traditional beta process and Windows has the additional burden of maintaining backward compatibility with a huge array of third-party hardware and software. In addition, Microsoft's Trustworthy Computing initiative has forced product teams, particularly Windows, to add security-focused activities and deliverables to their life cycle, which further complicates scheduling. (For more details, see the sidebar "Security Development Lifecycle".) Complex for Customers In addition, the increasingly widespread use of CTPs within Microsoft has led to a new problem for customers and partners: managing "CTP madness." Prior to its use of CTPs, Microsoft would produce a relatively small number of beta releases and would work to ensure that relevant betas worked correctly together. With the CTP program increasing the frequency of releases, there are now more prereleases in circulation at any given time and Microsoft no longer tries to make every new CTP release compatible with other previously released CTPs. Microsoft has tried to help customers by producing a Web site called "CTP Madness" that helps developers identify the CTPs installed on their systems and provides a list of other CTPs that are compatible. (For an illustration of this Web site, see "Managing CTP Madness".) Partners and customers should investigate compatibility information via the CTP Madness site before installing and evaluating any CTP and should invest in a desktop virtualization product, either Microsoft Virtual PC or VMware Workstation, to allow them to safely run multiple CTPs on a single machine without risking that they will interfere with one another. Resources For a more detailed description of the traditional process, see "Shipping Software: The End Game" on page 21 of the Oct. 2004 Update. The CTP Madness Web site is at channel9.msdn.com/ctpmadness. Jim McCarthy's Dynamics of Software Development captures much of the collected wisdom used by many Microsoft teams; it is published by Microsoft Press under ISBN 1-55615-823-8 and is available from Amazon.com at www.amazon.com/gp/product/1556158238. The Trustworthy Computing Security Development Lifecycle is described at msdn.microsoft.com/security/default.aspx?pull=/library/en-us/dnsecure/html/sdl.asp. Microsoft Press will release a Security Development Lifecycle book in May 2006 describing the process in more detail. Preliminary information about the book is at www.microsoft.com/MSPress/books/8753.asp.
|
||||
