Moving Up the Information Food Chain: Deploying Softbots on the World Wide Web

I view the World Wide Web as an information food chain (figure 1). The maze of pages and hyperlinks that comprise the Web are at the very bottom of the chain. The WebCrawlers and Alta Vistas of the world are information herbivores; they graze on Web pages and regurgitate them as searchable indices. Today, most Web users feed near the bottom of the information food chain, but the time is ripe to move up. Since 1991, we have been building information carnivores, which intelligently hunt and feast on herbivores in Unix (Etzioni, Lesh, Segal 1993), on the Internet (Etzioni Weld 1994), and on the Web (Doorenbos, Etzioni, Weld 1996; Selberg Etzioni 1995; Shakes, Langheinrich, Etzioni 1996). Motivation Todays Web is populated by a panoply of primitive but popular information services. Consider, for example, an information cow such as Alta Vista. Alta Vista requires massive memory resources (to store an index of the Web) and tremendous network bandwidth (to create and continually ...