• Illustration: Direct Indexing vs. Federated Search

A forthcoming update to Microsoft's enterprise search servers, which enable business users to search a wide variety of resources on internal and public networks, could make it significantly easier for users to find data locked up in corporate applications, such as business intelligence and enterprise resource planning (ERP) systems. Microsoft is also lowering prices dramatically—even offering a free version called Search Server Express—in an effort to counter Google's Search Appliance. The improvements could spark demand for the product and provide partners with opportunities to help customers customize the server, particularly its user interface and connectivity to secured resources.

Enterprise Search as Product Category

Enterprise search tools help organizations (including small and midsize businesses—not just enterprises) find information in a wide variety of data sources, including file systems, e-mail systems, and corporate business applications. Although these tools often employ a user interface similar to Internet search engines, such as Google or Microsoft's Live Search, enterprise search tools must understand many more types of data (e.g., information in ERP systems) and must provide access control to ensure that searchers do not see information they are not authorized to view. Enterprise search tools also rank search results differently and generally give customers programmable control over the relevance rankings of results.

Microsoft has long offered search capabilities in many of its desktop and server products, but SharePoint Portal Server was the company's first true enterprise search product, able to crawl a wide variety of networked data sources and make them searchable through a single user interface. With SharePoint Server 2007, the company updated the product's search capabilities and added a dedicated product edition for organizations interested only in enterprise search: SharePoint Server 2007 for Search. This edition did not require Client Access Licenses (CALs), which in most cases made it less expensive than the full version of SharePoint Server.

Microsoft's next generation of enterprise search server products, which will be available in the first half of 2008, show a significant evolution in the company's approach. While the company is still including and promoting enterprise search in the full version of SharePoint Server, it is repositioning the search-specific version of the product, as follows:

• Although still based on SharePoint technology, the next version of SharePoint Server for Search will be rebranded as Search Server 2008.
• A free edition, Search Server 2008 Express, will offer the same features as Search Server 2008 but will be installable on only one server (limiting its scalability)
• All artificial document-indexing limits will be removed from all versions of the product (SharePoint for Search 2007 Standard Edition has a 500,000-document limit)

Search Server 2008 could put pressure on Google's search appliances, whose prices are based on the number of documents they can index: for example, the cheapest version of the Google Mini costs US$1,995 and can index 50,000 documents, while the Google Search Appliance starts at US$30,000 for a 500,000-document limit. Although Google reportedly has only about 10,000 customers for its search appliances, some of these customers have used the product in major installations—for example, Honeywell serves more than 500,000 employees and visitors to its public Web site using Google appliances. In principle, these installations give Google an entry to sell other enterprise applications, such as Google Apps for Your Domain, which could pose a threat to Microsoft's lucrative Office franchise and other products.

Search Server will also compete with a free version of IBM's OmniFind enterprise search application, which IBM released with Yahoo in Dec. 2006.

Search Server 2008 still lacks capabilities of the most sophisticated enterprise search (or information management) applications from companies like Autonomy, Endeca, and FAST, but given Microsoft's new focus on search as a product category, the company could eventually move up the search stack to challenge these products.

New Features

The most significant addition to Search Server 2008 is federated search, which will make it easier to conduct searches of applications and Web sites that have their own search engines. Other improvements include streamlined setup and direct connectivity to more applications.

Federated Search

Until now, SharePoint Server could run search queries only against its own index. In other words, before a user could search a particular resource, a SharePoint indexing server had to crawl that resource for data and then organize that data in its own local index.

While this offered administrators a high degree of control over search results, some sources—particularly corporate applications, such as ERP and customer relationship management applications—were not natively supported and required Microsoft, ISVs, or customers to write specialized connectors so that SharePoint could index their data. In lieu of these specialized connectors, customers could deploy and configure a SharePoint 2007 feature called the Business Data Catalog, but it was fairly complicated (and poorly documented at the launch of SharePoint 2007) and only available in the most expensive Enterprise Edition of SharePoint Server—not in any edition of SharePoint Server for Search. In addition, for security reasons, some data repositories are purposely closed to indexing crawlers.

Search Server 2008 adds support for federated searches: instead of having to crawl every data source and store results in its own index, Search Server can simply query the search engines already in place in many corporate applications and Web sites. (For a diagram illustrating the difference between direct indexing and federated search, see "Direct Indexing vs. Federated Search".)

To connect Search Server to corporate applications, Microsoft, ISVs, or customers will still have to write code to allow the queried application to return a response in an XML format that Search Server understands, but this should be significantly easier than building specialized indexing connectors. As of Nov. 2007, 15 companies, including Business Objects (which is being acquired by SAP), Cognos (being acquired by IBM), EMC (for its Documentum document management application), Endeca, FAST, SAS, and Symantec, have promised to create federated search connectors for their products, and more are expected.

Search Server 2008 also supports federated search against other SharePoint or Search Server indexes, saving capacity by eliminating the need for duplicate entries in each index.

Federated search has some disadvantages compared with indexing content directly. For example, administrators cannot tailor results for relevance, determine how frequently source data is crawled, or specify which metadata to crawl; all of these functions are determined by the federated site. This could lead to less accurate results from federated sources.

Because users conducting federated searches must sometimes be authenticated to view results from indexes outside their own domain, Search Server 2008's federated search capacity allows querying users to submit some types of authentication tokens (such as Kerberos) out of the box. However, other types of authentication (such as cookie-based authentication for Web sites) will require custom code.

Other Improvements

Search Server 2008 also delivers the following improvements:

Simplified setup and administration. Search Server 2008 will boast a streamlined setup and installation process. For example, a new Search Server Preparation Tool will automatically download and install prerequisites (the .NET Framework 3.0 and SP1 for Windows Server 2003) if necessary and will activate the Internet Information Services (IIS) Application Server role on the server. Additionally, the administration dashboard has been consolidated into a single screen, and help for particular tasks (e.g., defining which resources to crawl) is more readily available. Even with these improvements, however, Search Server 2008 may still require significant work and knowledge to set up properly. For example, building a useful results interface (particularly for public-facing resources such as Web sites), connecting Search Server to resources that require user authentication, and designing advanced installations (e.g., that use SQL Server for greater indexing capacity) are all opportunities for partners and integrators.

New indexing connectors. While federated search is the main highlight of Search Server 2008, the company has also added new connectors that let the engine index files stored in Documentum and FileNet enterprise content management systems.

Availability and Pricing Considerations

A release candidate of Search Server 2008 Express Edition is available for download as of Nov. 2007. Final release of both the Express and full versions are expected in the first half of 2008, after the release of the first service pack for Office 2007.

Functionally, there is no difference between the Standard and Express Editions of Search Server, except that the Express Edition can be run only on a single application server, which limits its capacity. (Microsoft has not yet issued any recommendations or guidelines suggesting when a multiserver installation might be required.)

As with SharePoint Server 2007, the indexing and query engine of Search Server 2008 can run on a separate computer from the Web front-end servers that provide the user interface. In addition, while Search Server Express is limited to a single application server, customers may have multiple Search Servers configured in the Web front-end role connected to that installation of Search Server Express.

Although Search Server Express will be available free of charge, it requires Windows Server 2003 SP1, and Microsoft recommends running it on a dedicated machine.

In addition, the basic installation option for Search Server Express will automatically install the free SQL Express database, which has a data limit of 4GB, which would impose a practical limit of approximately 400,000 indexable documents. For greater indexing capacity, customers will have to use the advanced installation option for Search Server Express, which requires SQL Server 2000 SP3a or later, or SQL 2005 SP1 or later, running on dedicated hardware; a software license for SQL 2005 is priced around US$3,899 per processor or US$739 for a server license with five CALs and US$149 for additional CALs. (Exact prices depend on volume license agreements and other factors.)

Microsoft has not revealed pricing for the full version of Search Server 2008. However, the Standard Edition of SharePoint 2007 for Search starts at US$8,189.

Microsoft has said it will update existing SharePoint Server 2007 installations with a free feature update in the first half of 2008 to deliver improvements such as support for OpenSearch and new indexing connectors; customers will not need Software Assurance on SharePoint 2007 to receive this update.

Microsoft's enterprise search home page is www.microsoft.com/enterprisesearch/default.aspx.

Download the CTP of Search Server 2008 Express at www.microsoft.com/enterprisesearch/serverproducts/searchserverexpress/download.aspx.

Technical information about Search Server 2008 Express is available from www.microsoft.com/enterprisesearch/serverproducts/searchserverexpress/techres.aspx.

An overview of federated search is available at msdn2.microsoft.com/en-us/library/bb931080.aspx.

Architectural guidance for building federated search connectors is at msdn2.microsoft.com/en-us/library/bb931083.aspx.

OpenSearch information is available at www.opensearch.org.