The Search Continues

Its next to impossible to follow the latest developments in search tools. But, fortunately, there are only three types you need to keep track of—subject directories, true search engines, and meta-search engines—and each have their own particular advantages.

But let's begin with a little history. Back in 1995, when the Web first became popular, there were few search tools from which to choose. The two most-visited tools, Yahoo! (www.yahoo.com) and the now-defunct Magellan, were subject directories. Employees of these two enterprising companies scoured the Web, classifying its sites into broad categories and creating subject lists that could be easily browsed. A bit later, search boxes were added to those tools, enabling visitors to find what they were looking for by simply typing in a keyword—but those early searches were limited to the contents of the individual subject directory. Because of the search boxes, Internet users began to refer to these subject directories as "search engines"—although technically, they were truly searchable subject directories.

Around that time, companies such as WebCrawler and Excite began to create automatic indexing programs (called spiders or bots) that searched the Web, gathering various types of information—title tags, frequently used words, or full text. The collected information was then stored on the company's server in a large database of indexed words. Search-engine users simply typed in keywords and received a list of results or "hits." Then by simply clicking an item, users were linked to the corresponding Web page. This was, and still is, how true search engines, like Google (currently the most popular search tool of them all), work.

It's no longer easy to distinguish a subject directory from a search engine. Some large subject directories now redirect their users to true search engines, when there's nothing relevant in their own directories. And some true search engines now include an "expert-created" list of links—similar to those found in subject directories—to help users who can't find what they're searching for by using solely keywords.

Meta-search engines are now another common type of search tool. These tools search far and wide—examining the contents of both subject directories and other search engines. Some meta-search engines let you select which search engines you want them to browse. For example, Vivisimo (vivisimo.com), one of the most useful meta-search engines, has an "advanced search" option that lets users specify both the Web search engines (AltaVista, MSN, Looksmart, and Lycos) and the news search engines (CNN, New York Times, BBC) they want to explore.

Large, general subject directories, such as Yahoo!, let you browse through progressively smaller series of subject headings and control the search process. This is especially helpful when you want to get a feel for the available materials on a particular topic and aren't sure of the keywords associated with it. Some users also find that smaller topic-specific subject directories that have been created by experts—for example, PECentral (www.pecentral.org), which provides information for health and physical educators—are great places to begin their search.

True search engines, like Google (www.google.com) or AlltheWeb (www.alltheweb.com), let you search a database of terms taken from millions of pages on the Internet. Although the estimates vary, it is generally accepted that any single search engine cannot provide more than 20 or 30 percent of the total available information on the Internet.

Meta-search engines are a good place to start when you want to get an overview of what's available in both subject directories and search engines. The number of search results can often be overwhelming, but you can fine-tune your search by using the sites' "advanced search" options.

But there is a significant portion of the Web that most search engines ignore—the "Invisible Web," which consists of Web sites that have their own search tools and won't allow typical search engines' spiders into their servers. True search engines often have trouble indexing the content of dynamically created Web pages—the Invisible Web pages that are created in response to a particular query. Since dynamic pages don't exist before you type in a search request, many search engines skip right over them.

But don't despair. There are now subject directories, such as InvisibleWeb.com (www.invisibleweb.com) and SportingNews.com (www.sportingnews.com), that collect the Invisible Web's search tools and browse its databases. Search engine software and interfaces are also becoming more sophisticated and will soon be able to provide "smarter" information based on a user's search patterns and previous selections. And researchers are now designing tools that will present search results in a user-friendly graphic form, complete with related topics and subtopics and graphic organizers, or "mind-maps." One of these next-generation search tools is called Kartoo (www.kartoo.com). Kartoo creates a visual representation of the search results that resembles a flowchart, with lines connecting the related topics; you can also easily refine or expand the search. Other new technologies are now in the works that will enable you to search the surface Web and the Invisible Web simultaneously, providing users with the best results possible. And according to the Wall Street Journal, someday you'll be able to phone in a search request and have the results sent to your computer.