Service Quality Monitoring (SQM), an optional monitoring feature introduced in Office 2003, is influencing design of the next version of the Office suite. SQM transmits anonymous performance data, command usage counts, and other statistics from customers to the Office product team for analysis. SQM has helped Microsoft fine-tune Office 12's new user interface and identify hard-to-reproduce bugs, and could one day provide useful data for corporate help desks and ISVs. However, SQM data aren't yet available for widespread use outside Microsoft, and Microsoft itself will have to interpret SQM results with care.

Live Application Statistics

Formally known as the Customer Experience Improvement Program, SQM saves data to the user's PC at regular intervals and transmits them over the Internet when PC usage is low, typically daily. Statistics collected cover the following:

• Errors, including crashes and error dialog boxes shown to the user
• User interface events, such as menu pull-downs and use of keyboard shortcuts
• Hardware characteristics, such as processor speed and main memory size
• Performance metrics, including command response times and memory usage.

SQM has a number of characteristics to limit its intrusion on user privacy. Users must opt in, typically in a dialog box that appears when they first use the suite on a particular computer, although they can later activate the feature by changing privacy settings. Organizations can disable SQM altogether by customizing Office installation and Group Policy. If enabled, SQM does not transmit any user-identifying data; rather, SQM data are identified by a hash code generated for the specific computer when SQM is activated. Moreover, SQM records only generic performance and usage statistics for the application, not the content of documents or messages. For example, SQM data might report that a user printed 10 documents but would contain nothing about the contents of the documents.

SQM is separate from the Watson crash dump service (also known as Microsoft Error Reporting or Windows Error Reporting). When an application crashes, Watson transmits to Microsoft a dump of the application's memory; this dump might include document sections or other sensitive data. SQM, in contrast, collects data continuously, but collects only statistics that are not traceable to a user.

Impact on Office and Beyond

SQM is currently active on about 400,000 computers running Office 2003, and the Office product team is already using these data to design the next version. For example, the team placed commands in the new Office 12 "ribbon" based in part on usage statistics from SQM. SQM also helped avoid some gaffes: for example, the team had considered making the Paste command less prominent in the Office 12 ribbon because it is often activated by keystrokes, but backed off when they found that even heavy keyboard users frequently activate Paste from menus.

SQM is being used outside the Office product team. For example, Windows Messenger and the Windows Media Player use it, and SQM on Office has been employed by Business Productivity Advisers (BPAs) in the field to evaluate Office usage at customer sites.

Microsoft is considering offering a SQM data capture and analysis utility to large organizations, analogous to the Corporate Error Reporting utility that it already offers for Watson dumps. This would enable organizations to control transmission of SQM data by their users (rather than simply turning off SQM at installation), track application usage, and identify common errors. ISVs might also benefit from adding SQM to applications, as some do with Watson. However, Microsoft has no immediate plans to enable this, in part because SQM data collection requires much more extensive modification of applications than Watson does.

Design by Numbers?

Microsoft has benefited from SQM, but the mechanism is no panacea. Like any statistics, SQM data are subject to bias: users who turn SQM on might differ systematically from users in general. The Office team tries to identify potential biases by comparing SQM data from the field with data collected from groups of users carefully chosen to provide a cross-section of the population. The design team also relies on data gathered in separate laboratory and on-site usability studies.

Nevertheless, selectively gathered statistics can be misused to support weak design decisions with what look like "hard numbers". In the end, SQM can only be one data source for design, and no data source can substitute for sound design judgment.

An explanation of the Customer Experience Improvement Program (the public name for SQM) appears at www.microsoft.com/products/ceip.

Windows Error Reporting (Watson) was outlined in "Windows Error Reporting Tracks Down Bugs" on page 3 of the July 2003 Update.

The Office 12 user interface was previewed in "Office 12: New User Interface, Content Management " on page 23 of the Oct. 2005 Update.

An Office user interface design blog kept by Microsoft engineer Jensen Harris is at blogs.msdn.com/jensenh.