Sponsored search: A brief history

Web search is a fundamental technology for navigating the Internet. It has become an essential part of the online experience — 98.8% of Internet users utilize search (iProspect, 2004) — yet, it is provided free to consumers. Sponsored search, the delivery of relevance-targeted text advertisements as part of the search experience, makes this possible. In the week ending August 21,2005, search engines displayed 13 billion sponsored results (Nielsen/ NetRatings, 2005). Sponsored search has evolved to satisfy users' need for relevant search results and advertisers' desire for qualified traffic to their websites, and it is now considered to be among the most effective marketing vehicles available. Advertiser-provided content: a set of advertiser hyperlinks annotated with keyword tags, titles and descriptions Advertiser-provided bids that value traffic on specified concepts or keywords Review process combining manual and automated methods to ensure that advertiser content is in fact relevant to the target keyword Matching of advertiser content to user queries as they are received by a search engine Display of advertiser content in some rank order in some placement alongside other algorithmic (that is, non-sponsored) search engine content Processes that gather data, meter clicks and charge advertisers based on consumer clicks on their displayed content In 1998, GoTo, later Overture Services, was the first to combine these elements. Yahoo! acquired Overture in 2003, re-branding it to Yahoo! Search Marketing in 2005. BeFirst — now MIVA — followed with a similar product in 1999. Google adopted the model and modified it to incorporate click feedback in 2002. In 2005, Ask Jeeves adopted it, and MSN Search extended it to support behavioral targeting. Sponsored search engines display results not only on their own sites, but also in space rented on other sites. For example, when a user searches at MSN Search, the search query is passed to Yahoo!'s sponsored search engine, which returns results to the MSN server, which in turn renders the page that the end user sees. Similarly, Google rents space on AOL. In the early days of Web search, sites used annotation via 〈META〉 tags that contained keywords supposedly describing the site's content. Some sites used these tags to outrank other sites that may have been more popular or relevant. Even worse, some sites — in particular, pornographic ones — manipulated 〈META〉 tags and appeared in the results for essentially irrelevant queries. To address this problem, GoTo focused on roughly the top 1,000 queries, employing editors to create handcrafted result pages with images, product and service offers and informational links. When GoTo introduced sponsored search, each listing was associated with a keyword. The keyword set was open ended, and advertisers could request to add new ones. In other words, GoTo's editors ensured the relevance of the advertiser content to the keyword. Given the rapid growth of the sponsored search industry, this process is now partially automated. The majority of Yahoo!'s sponsored search listings are initially evaluated by software. In the beginning, Internet advertising mimicked offline advertising. Viewers of print ads are more likely to remember ones that are more graphical and less text-oriented (Du Plessis, 2005, p. 128). HotWired introduced graphical banner advertisements to the Web in 1994. Search engines relied on banner advertising before the adoption of sponsored search, so they faced a dilemma — keep users on the site as long as possible to view more banners or send the users promptly to the sites appearing in the search results. Paid search reconciled this dilemma by tying the search engine's revenue to the act of transferring the user to an advertiser's site. In 1996, the search engine Open Text briefly offered preferred listings, in which sites would pay to be inserted into the search result set for particular keywords. From the start, advertisers were concerned with how to measure effectiveness, and publishers were concerned with how to set prices. Some publishers charged a fixed fee to appear on a site, while others such as Netscape and Infoseek in 2005 used the cost per mille (CPM) — the cost to display an advertisement a thousand times –just as for offline advertising. A very basic question was whether the banner advertisements were actually causing users to visit the site. Some advertisers were practicing direct response marketing, which seeks a measurable and attributable response. The easiest response to measure was the click on a hyperlink. In 1996 Yahoo! agreed to charge Procter Hoffman, 2000). Both conversion rate and CTR are conditional probabilities: of an action given a click and of a click given an impression. In principle, with enough data, one can directly estimate these probabilities. A sponsored search campaign will usually encompass a huge number of unique queries, and for most of those queries, there will be insufficient data. Somehow, data must be combined across contexts — for example, by grouping queries together or comparing the current campaign to others. Yet, additional variables such as the time of day or day of week can be introduced that predict variations in conversion rate and CTR. This variation is one of the reasons that sponsored search bids fluctuate throughout the day. In 1997 FlyCast and Narrowline introduced marketplaces that enabled advertisers to manage campaigns across multiple publisher sites, each of which placed a bid for advertisements. In 2003, Yahoo! was the first major sponsored search provider to offer a tool that enabled campaign management across multiple online advertising types, including banners, sponsored search and email campaigns. Both Yahoo! and Google now run continuous auctions, in which a competitor willing to pay more can displace an advertiser at any time. Yahoo! sorts search results first by the preciseness of the match, then by the CPC bid selected by the advertiser. Direct Hit ranked search results using CTR. When Google relaunched AdWords in 2002, they modified Overture's ranking rules to use CTR*CPC instead of just CPC. Google generalized the CTR prediction, incorporating an analysis of the listing text (Google, 2005). Similarly, Yahoo! calculates a Click Index score by comparing a listing's observed CTR to a standard. This score is displayed in the advertiser's interface, but not used when ranking search results. When a listing's score is very low, it is flagged for possible removal. Yahoo! continues to transform the original sponsored search engine. Increasingly, there is academic interest in sponsored search. Weber and Zheng (2004) proposed ranking search results by combining merchant ratings and bids. Varian (1999) presented an argument for sometimes placing a more obscure interpretation of a query above a more likely interpretation based on the average user's search cost in the economic sense. Yahoo! helped organize a workshop at the 2005 E-Commerce Conference (http://research.yahoo.com/work-shops/ssa2005/sched.html) to bring together the fields of user experience, economics and information retrieval. Sponsored search continues to improve advertisers' effectiveness in targeting their campaigns and to provide highly relevant results for searches with commercial intent.