• Sidebar: The Advantages of the Local Web Storage System

Microsoft’s Exchange data storage strategy is up in the air, following the decision to stop development of the Local Web Storage System (LWSS) and related development tools. The LWSS, originally slated to ship in Office XP, was a client-side message database for Outlook that would have dramatically improved both offline and online functionality and performance. The LWSS team has been combined with other development groups in Microsoft in a push to develop the next version of the SQL Server storage engine, code-named Yukon, and to make that engine the core storage component for a future version of Exchange. Although Yukon promises a quantum leap in database architecture for Microsoft, it is a long way from realization. In the meantime, Microsoft’s competitors have a golden opportunity to win ground against Microsoft in the groupware arena and even its friends—developers focusing on Exchange-based communications and collaboration applications—have been left in limbo while Microsoft solidifies its strategy and plans.

Canceling the Local Store for Yukon

Both the LWSS and Office Designer (a new development tool targeted at giving power users an easy way to create Web applications that accessed the LWSS) were announced at the MEC 2000 Exchange conference in Dallas in Oct. 2000. Aimed at making Outlook and Exchange more competitive with Lotus’ Notes and Domino, the LWSS would greatly improve both the online and offline capabilities of Outlook. Office Designer would allow relatively easy creation of browser-based applications that exploit Exchange’s collaboration features, e.g., discussions, surveys, workflow, and that would still work when users are offline. Microsoft had released a beta version of the LWSS, and both users and developers were extremely enthusiastic about the potential of the technology.

Two months later, Microsoft announced a sudden change in plans. LWSS and Office Designer would be dropped in Office XP (and, by extension, in Outlook, which is part of Office). The reason given was extensive bugs found during the beta program. However, Microsoft’s long-range database strategy may have been just as responsible: in an internal memo to the entire company later in December (see www.zdnet.com/zdnn/stories/news/0,4586,2666543-1,00.html), Microsoft CEO Steve Ballmer disclosed that Microsoft would be refocusing the LWSS team on a future release of SQL Server code-named Yukon. He went on to state that Yukon "will be key to our next-generation storage, database, file system, e-mail, and user interface work" and asked that other product groups reorganize their product schedules around Yukon, which is not scheduled to ship for at least two years.

Microsoft has since announced that Yukon will replace the Extensible Storage Engine (ESE) as the basis for future "Web Store" object databases in Exchange and SharePoint Portal Server. (For links to more information on the Web Store, see Resources.) At press time, Microsoft has not divulged further detail on the reasons for the move and how it will affect their messaging and collaboration product strategy. Nevertheless, it has major implications for Exchange users, developers, application service providers (ASPs), and competitors.

The Need for Local Storage

The most immediate impact of the decision could be on Microsoft's Outlook mail client. Since its inception, Outlook has always had a mechanism that allowed users to work offline (disconnected from their Exchange server) using a client-side message database. They could bidirectionally synchronize all types of Exchange data, including calendars, tasks, contacts, and public folders, with their Exchange server. Applications on the client could read from and write to Outlook’s "offline store," giving Outlook/Exchange capabilities similar to that of its main rival, Lotus’ Notes/Domino. However, Outlook’s offline mechanism suffered from many important drawbacks that, at least in theory, the LWSS would have remedied, including:

MAPI dependency. Outlook uses the Messaging API (MAPI) over Microsoft’s remote procedure calls (RPCs) to communicate with the Exchange server. This is a problem for two reasons:

• RPCs are not an appropriate mechanism for communicating over the Internet. They typically require the added expense and complication of virtual private networks (VPNs) to tunnel through firewalls.
• Exchange 2000 can scale to handle higher volumes of e-mail by using front-end processors that are separate from back-end message storage. However, this architecture does not support MAPI-based clients like Outlook.
• Furthermore, local client-side applications can currently access Outlook’s offline store only through MAPI; local browser-based applications cannot access data in Outlook’s offline store using Web protocols. These applications cannot function without a live connection to the Exchange server, making them unsuitable for road warriors.

Poor mode transition. Outlook cannot transition between online and offline modes without the user having to shut down and restart Outlook in the other mode. Outlook will lock up if it loses its connection to the server when online, making undocking a laptop or taking an Exchange server offline a poor experience for users.

No use of the offline store when online. Outlook’s local offline store is not used when the user is working online, meaning that it provides no performance advantage by caching messages. Even if a message already resides in an offline store, Outlook insists on downloading it from the Exchange server each time it is opened, adding to network traffic and slowing response time, especially when low bandwidth or high traffic slows the network link between Outlook and the Exchange server.

For more details on how the LWSS would have solved these problems, see the sidebar "The Advantages of the Local Web Storage System".

A Cloudier Future for Outlook

Abandonment of the LWSS and plans to later release Office Designer without the LWSS interfaces raises many questions about Microsoft’s future plans for Outlook.

• Will Microsoft continue to support only MAPI-based Outlook offline replication when the next version of Exchange, code-named Mercury, ships?
• Will Mercury give ASPs and other large organizations hosting Exchange a better way to provide offline messaging and collaboration services to their customers?
• Without the LWSS, is upgrading to Office XP nearly as compelling?

Microsoft has not released any information to date that answers the first two questions, and many analysts and IT managers feel that many current Office customers will sit tight with Office 97 or Office 2000.

Why Wait for Yukon?

A client-side collaboration database technology closely related to that used on the Exchange server has several benefits for both Microsoft and its customers:

• It allows Microsoft to concentrate its resources on building a single homogeneous database technology, rather than on different database engines for the server and client.
• Symmetry between the client and server database architectures makes it much easier to build more powerful and flexible replication mechanisms.
• It allows development of client-side collaborative applications using the same APIs and development tools used to develop equivalent server-based applications.

Although Microsoft could have kept a team working on the current ESE-based Web Store and LWSS in parallel with development of their Yukon-based successors, a combination of limited development resources and insufficient return on a technology likely to be retired made the decision clear: Microsoft had to risk some short-term customer disappointment in favor of longer-term benefits.

Microsoft’s Yukon strategy will allow it to focus its database, file system, and replication development teams around a single core technology, rather than diffusing its efforts across the NTFS, ESE, Exchange (Web Store), and SQL Server technologies.

Moving to a Unified Store

Although Microsoft has not yet released details, it clearly sees Yukon as its universal data storage technology for .NET and future versions of most of its .NET Server applications. Future Web Services, such as Microsoft’s recently announced Hailstorm project, will rely heavily on back-end databases that can store both structured and unstructured data.

As the database market has evolved into a three-horse race between IBM, Oracle, and Microsoft, all three have been finding new ways to leverage the strengths of their database products to the point where the line between database and file system has become blurred. In particular, Oracle’s Internet Filing System (IFS) uses the Web Distributed Authoring and Versioning (WebDAV) protocol to provide many of the same capabilities of Exchange’s current Web Store, but is built on a much more robust database engine than the ESE.

The current Web Store has characteristics of a database, a Web server, a file server, and an application server, and can easily store hierarchical and unstructured data. Like Oracle’s IFS, Yukon will morph from SQL Server’s current relational model to a new hybrid architecture that can also support the Web Store’s hierarchical model.

Microsoft has been talking about creating a relational file system for at least five years and it demonstrated a "SQL Internet File System" at the July 2000 Professional Developers Conference. It seems likely that Yukon will be a big step in that direction, which will undoubtedly involve close ties to the NT File System (NTFS)—to the extent that the two may become inextricably linked.

ActiveX Data Objects (ADO) is a data-access technology that will evolve into ADO.NET under the .NET Framework. Microsoft has already stated that ADO.NET, using new managed providers to replace OLE-DB, will expand in versatility and scope beyond the current ADO. ADO.NET will become the standard way that applications access data of any kind, including files, messages, relational tables, and objects. Most or all of this data could reside within Yukon.

Although Microsoft added Extensible Markup Language (XML) support to SQL Server 2000, adding native WebDAV support to Yukon will require even more extensive use of XML data types.

Benefits for Exchange

Exchange 2000’s storage model supports a maximum of 20 databases. According to an Exchange program manager, this limit is a function of ESE’s architecture and Windows’ 32-bit address space. This becomes a problem for ASPs and large organizations that want to isolate organizations or groups of users into separate databases on a single back-end Exchange server. For the maximum number of Exchange databases to grow, Microsoft indicated it would need to develop a 64-bit version of ESE.

By moving Exchange onto Yukon, Exchange is freed from the constraints of ESE, and should be capable of scaling to support millions of users. Furthermore, SQL Server is much more flexible than ESE in the ways it can be partitioned, distributed, replicated, clustered, backed up, and restored.

Today, Exchange 2000 systems can make only limited use of Windows 2000 Datacenter Server’s capabilities because the Web Store does not utilize Address Windowing Extensions (AWE) to move beyond the 3GB memory space imposed by 32-bit Windows 2000. Because Yukon should continue SQL Server’s support for AWE and will also ship in a 64-bit version, ASPs and large organizations should at last be able to build highly consolidated, manageable, flexible, and reliable messaging and collaboration application servers rivaling those of Microsoft’s Unix competitors.

Lastly, with the absorption of the LWSS team into the Yukon team, client-server data replication should become a major focus. Client-side versions of Yukon (analogous to today’s SQL-Server-based Microsoft Data Engine and SQL Server CE) could provide users with all of the offline and caching benefits that would have been offered by the LWSS, plus offer access to collaborative applications across a range of different devices, such as cell phones and wireless PDAs, not just on Outlook running on a PC.

Broader Benefits

As databases scale ever larger and the applications built on top of them require continuous availability, Microsoft plans to add 64-bit support, improved clustering, and single-image management to Yukon. Microsoft claims Yukon will "continue software scale innovation by introducing the second generation of scale-out partitioning, shared-nothing clustering technology. Yukon will provide customers with tremendous additional gains in scalability while pushing the state of the art for reliability and manageability." (Microsoft defines "shared-nothing" clustering as "each server maintains its separate processors, memory, storage, record locks, and transactions. Each of the servers in the cluster communicates and coordinates with the others across a network or using a high-speed, low-latency interconnect technology, such as Compaq ServerNet II and Giganet cLAN.") It was extremely unlikely that the ESE would have ever seen this level of attention to scalability and availability.

The new unified storage-access programming and security models provided by the ADO.NET and Yukon should greatly simplify development, benefiting Microsoft, ISVs, and corporate application developers alike.

A Golden Opportunity for IBM and Oracle

Microsoft had just begun a full-court press last fall to encourage developers to use Exchange 2000 as a development platform (see " Web Store Overhauls Exchange Development Platform" in the May 2001 Update). The cancellation of LWSS and the move to Yukon raises doubts that the Exchange platform and programming model is stable enough to risk major investments in developing commercial or corporate applications based on it. Concerned managers and developers are wondering the following:

• Will Exchange’s storage model be essentially frozen until Kodiak (the code name for the Yukon-based Exchange release following Mercury) ships?
• Will collaborative applications hosted on Exchange 2000’s or Mercury’s Web Store still run on Yukon, or will the programming interfaces be radically different?
• Is the Yukon schedule even feasible? Could it push out the Kodiak schedule even further? Will Kodiak also be tied to a post-Whistler release of Windows (code-named Blackcomb)?
• Will ESE/JET be eliminated completely? What will happen to other technologies that use it, such as Access, Active Directory, or DHCP?

The Meta Group has been especially critical of Microsoft’s collaboration strategy, calling it "muddled."

"Without a clear direction from Microsoft, we recommend users be cautious in developing on top of the Web Storage System," says Matt Cain, an analyst with Meta Group. "The problem is as you write to APIs within Exchange, they call the underlying data store. If they separate the data store, do you still get the same performance?"

If developers properly use the ADO, CDO, and WebDAV interfaces to the Exchange 2000 (or SharePoint Portal Server) Web Store, these applications should, in principle, be relatively easy to migrate to Kodiak and Yukon, and Microsoft has already indicated that they will support migration of Web Store data to Yukon. However, Microsoft has released so little detail that many developers are now thinking twice before committing resources to development on the Web Store platform.

Because Yukon is still two to three years off, this represents a golden opportunity for both IBM’s Lotus division and Oracle to attack Microsoft, and in fact they have begun to do so. Ed Brill of Lotus has published strident attacks on Microsoft’s Exchange 2000 product and future strategy, and although Microsoft has rebutted many of Lotus’s claims, it may be enough to create doubt among Microsoft’s customers. Lotus went so far as to hire an outside agency to compare the current releases of Notes/Domino and Exchange 2000 as a development platform for hosting collaborative applications, and Lotus came out the winner. (See the Lotus link in Resources to see the results of this study.)

Both Oracle and IBM are also offering knowledge management products that compete against SharePoint Portal Server, and the confusion surrounding Microsoft’s plans may allow its competitors’ efforts to gain traction.

Microsoft has indicated that it will release more detail on Yukon and Kodiak in the May timeframe, so look for a future Update article that clarifies many of the issues raised in this story. Much rides on how convincing a case Microsoft can make that the short-term drawbacks are worth the longer-term gain.

Resources

For more information on the Exchange 2000 and SharePoint Portal Server Web Store, see "What is the Web Storage System?" on page 6 of the Dec. 2000 Update.

For more information on Exchange 2000 and Web Store as a development platform, see "Web Store Overhauls Exchange Development Platform" in the May 2001 Update.

For more information on Lotus’s competitive position against Exchange 2000, see www.lotus.com/home.nsf/welcome/domino and click on "Compare."

For Microsoft’s rebuttal to Lotus’s attack, see www.microsoft.com/Exchange/productinfo/LotusResponse.htm.

For more information on Oracle’s Internet File System (IFS), see otn.oracle.com/products/ifs.