Chat Room: The Invisible Web

Lots of useful, high-quality information exists on the Internet for you and your students—but popular search engines, like Google and Yahoo!, won't help you find it. The information I'm talking about is part of the Hidden or Invisible Web, and you need to use special search tools to locate it.

Typical search tools won't find "dynamic" pages—pages created, for example, by a newspaper or magazine's Web site in response to a particular query. For instance, when I visited CNN.com and typed in the term "Supreme Court," a dynamic or custom page was created by the software in response to my inquiry, listing relevant articles with useful links. Because dynamic pages like this one literally do not exist before you type in your search request—only the pieces such as the text of the article and the ads do—most search engines skip over them. Most search tools use what's called a spider program—software that cruises the Internet, examining the contents of the pages it finds and adding them to the search tool's list of sites. But it usually doesn't find dynamic pages.

To learn more about the Hidden Web, I recently talked to Marcia A. Mardis, a librarian who directs the Michigan Teacher Network of the Center to Support Technology in Education (www.merit.edu/cste). She has written several articles on the topic, most notably "Accessing the Deep Web" (see mtn.merit.edu/pdf/w01mardis.pdf). Mardis says that the Hidden Web isn't a secret: entire books and Web sites are devoted to teaching online users how to find it. So why don't more educators and students take advantage of this valuable resource? It's a matter of time and convenience, she explains. Google's interface is simple, easy, and fast, so most users don't bother looking any further. The trick is to convince educators that the Hidden Web is an essential resource for research and information needs.

Well-known subscription databases, such as Galenet and EBSCOhost, should be the first online destination for students searching for periodical articles published within the last 20 years. But there are also free Hidden Web periodical databases that are helpful. One of the best is FindArticles (www.findarticles.com). When I looked for information on the changing roles of Afghan women since the fall of the Taliban, for example, I found several excellent articles. FindArticles accesses articles from magazines like The Humanist and UN Chronicle, which don't charge online visitors. Although FindArticles doesn't offer articles from Time or Newsweek, it's a terrific database for schools with small resource budgets. Mardis also recommends a similar tool, MagPortal (www.magportal.com).

Educators who want to teach their students about the buried treasures of the Web should prepare a list—or a page on their library's Web site—of the most effective Hidden Web tools. Many of these resources focus on a single topic, such as FindLaw (www.findlaw.com), a popular Web site that offers information on federal and state laws, bankruptcy, lawsuits, and traffic tickets. Teachers should also know about the Gateway to Educational Materials (GEM) site (www.geminfo.org), which provides a meticulously cataloged collection of educational materials culled from federal, state, and university Web sites. GEM, a project sponsored by the U.S. Department of Education and operated by Syracuse University, lets educators search for lesson plans and other useful curriculum materials by grade level and topic. One of the best articles on the Hidden Web is "Invisible Web: What It Is, Why It Exists, How to Find It, and Its Inherent Ambiguity" (www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html), which includes an excellent list of resources for educators.

Mardis looks forward to the December debut of the National Science Digital Library (www.nsdl.org), which she calls "the ultimate Hidden Web tool." The new site will give educators access to thousands of documents in the sciences and mathematics. For more information, see www.ericit.org/digests/EDO-IR-2001-10.shtml.

Finding quality online information takes effort. We need to let our students know there are some terrific resources waiting to be found. The Hidden Web offers a great collection of materials, including government documents, research reports, and periodical articles. As great as Google is, it can't turn up everything.