• Illustration: Human Interactive Proof

As increasing volumes of unsolicited commercial e-mail, or "spam," continue to raise costs for Hotmail and MSN Internet Access and dampen consumer excitement about the Internet and computing, Microsoft will work with AOL and Yahoo to define acceptable terms for commercial e-mail, share information and technology, and take legal action against spammers. The initiative will be overseen by a new team at Microsoft which will also concentrate on developing new antispam technology for Hotmail, Outlook, and Exchange.

2.4 Billion Spams per Day Intercepted

According to market researcher Message Labs, up to 50% of all e-mail sent today is spam. The main reason for spam's prevalence is simple economics: for less than US$100, an advertiser can send messages to millions of customers, meaning that even minuscule response rates can generate positive revenues.

Spam impacts businesses and the consumer market. Corporate IT departments and ISPs make major efforts to block spam from reaching their users and may have to invest in bandwidth and e-mail server capacity that would otherwise be unnecessary. Wading through piles of spam could also cause consumers to sour on the Internet experience, reduce the amount of time they spend with their PC, and make them less likely to upgrade their hardware and software.

The problem is particularly acute for Microsoft: spam accounts for more than 80% of all messages sent to Hotmail and MSN Internet Access users, and Microsoft intercepts about 2.4 billion spams intended for these users every day. Microsoft claims it is a favorite target of spammers because of Hotmail's 130 million users, which makes it easier for spammers to reach real users by guessing many addresses (a process known as a "dictionary attack"). This forces Microsoft to maintain extra bandwidth and e-mail servers and staff to combat spam, and threatens to drive users away from these services.

Partnership with AOL, Yahoo

In Apr. 2003, Microsoft announced that it would work with AOL and Yahoo to help fight spam. Although a detailed plan of action is still being determined, the companies will collaborate in the following ways:

Create a standard definition of spam. E-mail provides an inexpensive way for companies to communicate with customers, but the line between legitimate commercial messages and spam is not always clear. For instance, many online services and e-commerce sites (including some Microsoft sites) allow users to sign up, or "opt-in," to receive e-mail updates or announcements of special offers, whereas other sites automatically sign users up for such e-mails unless they specifically request otherwise ("opt-out").

To combat spam without harming a cost-effective method of customer communication, the three companies will work on defining clear standards for legitimate commercial e-mail (including mandatory opt-in, rather than opt-out) and will create or join industry self-policing bodies. Eventually, the three companies could decide to deliver e-mails only from organizations that have been approved by these bodies.

Share information. The three companies have agreed to share information about possible spammers and spam attacks. For example, if one company is intercepting a high volume of suspicious e-mails from a particular IP address range, it would share this information with the others so they could take action, such as blocking messages from these addresses.

The companies will also publish white papers and lists of best practices that will help other ISPs and consumers cut down on spam.

Make it hard to sign up for multiple free accounts. Because of spam, ISPs today often place a limit on the number of e-mail messages that can be sent from a single address at a time. To get around this problem, spammers have created programs that automatically sign up for hundreds of free Web-based accounts at Hotmail and similar sites.

The three companies have agreed to develop and share technology that would make it harder for spammers to do this. For instance, in Dec. 2002, Hotmail introduced technology called Human Interaction Proof: when signing up for a new e-mail account, users must enter a text string displayed as blurry letters in a .bmp file. Automated programs cannot read the letters and therefore are not able to complete the task. (See the illustration "Human Interactive Proof".)

Stop spoofing. Although the technical details are sketchy, the three companies want to make it harder for spammers to send e-mail messages with disguised or "spoofed" information in the header. Spoofing allows spammers to impersonate legitimate users or send e-mail that appears to be from a fictitious domain, and makes it difficult for ISPs to block messages from known spammers and for authorities to enforce antispam regulations.

Spoofing is possible because the Simple Mail Transport Protocol (SMTP) used by most e-mail clients to send messages does not natively support authentication, and few ISPs or e-mail servers take advantage of an SMTP extension that enables a client to negotiate a secure transaction with a server. One possibility is that the companies will begin to take advantage of this extension and require users to get an unforgeable certificate from a trusted third party before allowing them to send e-mails through their services.

Use the legal system. The three companies will lobby for federal and state laws that prevent spammers from using misleading subject headers or falsifying their e-mail and IP addresses and will continue to pursue litigation, such as the "John Doe" suit that Microsoft recently filed against dictionary-attack spammers in California.

New Group Formed

The cooperative initiative with AOL and Yahoo is being overseen by a new Anti-Spam Technology and Strategy Group led by General Manager Ryan Hamlin. The group includes about 25 members, including some developers who have worked on antispam technology for Microsoft Research, and reports to Vice President Blake Irving, who oversees Hotmail and other MSN services. It will also work closely with developers on the Outlook and Exchange teams, as well as Microsoft's legal department.

In addition to overseeing the initiative with Yahoo and AOL, the group is in charge of improving the antispam capabilities of Microsoft products and services, both through in-house development and licensing third-party technologies:

Machine-learning filters. Today, most spam filters rely on specific rules, rejecting e-mail messages based on their supposed IP address or content in the header or body. Spammers can get around such rules quickly: for example, when many rules-based filters began blocking messages with obvious come-ons (such as "Make Money Fast") in the subject line, spammers began sending messages with subjects that appeared to be legitimate personal or commercial messages (such as "Your order has been delayed").

Instead of relying on simple rules-based filters, Microsoft is concentrating on "machine-learning" filtering. Users will be asked to tag spam by taking a certain action, such as placing it in a special folder. The filtering program will then look at characteristics of these messages and determine the probability of incoming messages being spam; as spammers try new tactics, the filter will automatically adapt based on users' actions. Currently, Microsoft is collecting spam data from 100,000 Hotmail users and using this data to maintain a machine-learning filter for the service; similar technology could find its way into future versions of Outlook and Exchange.

Image beacon prevention. Some spammers use "image beacons" in their spam messages to determine whether an e-mail address is valid. When a user opens the message, it activates a unique link to a graphics file (the "beacon") on a Web server owned by the spammer. To block this type of spam attack, Hotmail recently added a client-side filter that prevents users from viewing graphics in e-mails unless the message comes from an address in that user's address book.

Open relay blocking. Many spammers use open relay or open proxy servers—e-mail servers through which messages may be routed without having originated in the server's domain—to disguise their IP addresses. Hotmail keeps a list of suspected servers, as well as IP addresses of known spammers or ISPs that cater to spammers, and blocks messages from them. Although Exchange 2003 has the capacity to use third-party blacklists (such as that maintained by SPEWS.org), which list known open relays, Hotmail does not intend to use these blacklists.

Detecting dictionary attacks. Microsoft uses third-party filtering company Brightmail to filter spam as it comes into Hotmail and MSN servers. Although Brightmail uses rules-based filtering, which Microsoft admits is limited, it also maintains a "honeypot" of e-mail addresses that are never publicly revealed. Any time an e-mail message reaches one of these accounts, Brightmail knows that the spammer has guessed the e-mail address via a brute-force dictionary attack. This is useful for detecting spam and also in litigation, as it prevents spammers from claiming that they only sent messages to customers who requested them.

Resources

For more on the antispam technology in Exchange 2003, see "Exchange 2003 Boosts Performance, Eases Administration" on page 3 of the Apr. 2003 Update.

For background on the suits filed against dictionary-attack spammers, see "Antitrust, Antispam, and Other Legal News" on page 30 of the May 2003 Update.