THE SMART WEB PRIMER, PART 1: Lost (& Found) in Cyberspace; How to Make Search Engines Work For You

By Walter Minkel -- School Library Journal, 3/1/1997

Scudder Illustration My library is open Sunday afternoons so I'm used to all the "It's due tomorrow!" homework questions that the day brings us. On one such day, at 15 minutes to five, a high school student ran up to the Information Desk and told me, "I have to do a presentation on lupus, the disease, tomorrow. Do you have anything?"

Despite, or perhaps because of, the large amount of time I've spent using web search engines, my first choice when faced with a reference question is to use our collection. Only after I've exhausted that avenue do I brace myself for a spin on the Web.

So that's what I did. Using the online catalog, we found a book, but after the student looked through it rapidly, she said, "But there aren't any pictures. You know, of the -- what are they -- the skin eruptions. I need them for a visual aid."

General medical books were no help. Then, at eight minutes to five, I hada flash. "Let me check the Web," I said.

Normally, I wouldn't dare tackle a question that required finding very specific information or pictures on the World Wide Web in only eight minutes. Doing what I call a "cold search" -- jumping into one of the many search engines armed with only a search term or two -- can be time-consuming; success is far from guaranteed. In this case, however, using an engine wasn't necessary.

We should never forget that the search engines and directories are businesses. Our library system has assembled a directory of links on all kinds of subjects, so I quickly clicked my way to a site I knew, the MedWeb site and found, in its enormous directory of diseases, treatments, and syndromes, the lupus link.

The MedWeb directory's compilers had done the searching for me. I clicked through a group of text-only professional articles. At one minute to five, I found a "photo" link in one of the articles, clicked on it, and was treated (hmm . . .) to a picture of an unfortunate lupus sufferer's arm, complete with great pink eruptions. I showed it to the student.

"Oh, gross!" she said. "That'll be perfect!" I printed it on our color printer. She thanked me and was gone at 5:02.

Making Sense of Search Tools

What made the difference in this case was my library's pathfinder, a tool that allowed me to bypass search engines. The Web was never intended to be like a library. Search engines and directories -- AltaVista, HotBot, Yahoo, and the rest -- are businesses, not carefully designed public service tools. Did you ever wonder why five search engines -- Yahoo, Magellan, Lycos, Infoseek, and Excite -- are always the default choices when youclick on the "Net Search" button in Netscape Navigator? It's because each of those companies paid Netscape a cool $5 million to be there. Being Netscape's default engines brings them a huge number of searches, and a lot of extra advertising revenue.

Tools with Good
Track Records

Here are Walter's
favorite web
search tools.

Familiarity with search engines and other web resource finders is difficult to come by and difficult to keep, because the engines, like everything else on the Web, change all the time. Here is an introduction to current tools.

Subject Directories. The most accessible tool for the web noviceis the subject directory, like Yahoo and Magellan. These directories aren't really search engines proper, although they're often lumped together with them. They have search engines as well, but they generally don't search well for sites outside the directory's database. On the plus side, directories use human beings to sort sites. The downside is thatthis process takes time, so directory databases tend to include fewer sites and be the least up-to-date.

The best-known subject directory is Yahoo and its juvenile-oriented counterpart, Yahooligans.Magellan is also a directory, one that rates sites on afour-star system.

If you're doing a general search, say, for information on the Civil War, use a directory first. Why? Because people sort the sites, the selected sites are more likely to zero in on your topic. However, if you're willing to do more of the work yourself, a basic engine (see next paragraph) will give you a larger universe of sites to view. Also, directories often misssubtopics: you may find many pages on medieval castles, but nothing at all on medieval castles in Portugal.

Basic Engines. Then there are the basic search engines, like AltaVista, and like Lycos. If you're doing a specific search, for instance, Medusa rather than Greek mythology, use a basic engine. Enter your search terms as specificallyas possible and spin the Wheel of Fortune; nobody knows what will come up. Because the engines use "spiders" or "robots" -- bits of software that read and file the title and text of millions of pages of HTML -- they tend to have very large collections of current sites.

Putting the Web
to the Test

Walter puts search
engines to the test.

But choose your search terms carefully. I once had two boys who were looking for a large picture of the Egyptian god Anubis for a project. But because every middle school class in town was studying ancient Egypt, we had no circulating titles left on Egypt or its gods. The only pictures of the god we could find in reference books were about an inch high. "Let's try the Web," I said. In Lycos, I typed in "Anubis."

Bad move. You would be amazed at how many proper names or subject headingsare now the names of high-tech businesses or commercial products. Did you know (I hadn't) that there's a piece of security software named Anubis? The first 10 hits I received were all pages for this software. I tried again, eliminating these pages by typing in "Anubis Egypt god," which finally led us to a useful hit. When you use a basic search engine,include as many terms as possible to narrow the focus. Always put the most specific term first; some engines weight the search in favor of the first term.

Metaengines. Metaengines submit your search terms to several, or all, major search engines simultaneously. Results from metaengines are sometimes spotty, sometimes excellent. MetaCrawler, begun at the University of Washington but now a commercial engine, is the best of the metaengines in my opinion; it searches AltaVista, Excite, Infoseek, Lycos, WebCrawler, and Yahoo. A reader of the original print version of this article recommended Inference Find, and I like it as well. It's fast and returns hits that are worth considering. It only returns site titles, however, not descriptions.

Yahooligans is a good place to point students who are searching the Web on their own. Finding the Good Stuff

Before you begin your search, remember that very few Web sites will have information useful to a middle or high school student. Most sites are either businesses selling things, frothy chat, or university pages for a specific institution.

None of these will provide answers to typical homework questions (and in my opinion most young people are notoriously bad searchers, lacking the ability to sort the good stuff out of the dross). When working with young people, it's good to use tools that have already sorted out useless stuff in favor of pages that are truly useful and interesting to kids. Yahooligans does not have a large database, and like its parent, some links lead to dead ends. On the whole, though, it's a good place to point students who are searching the Web on their own.

There are several good non-commercial homework-oriented directories out there. Probably the best known are the privately created B.J. Pinchbeck's Homework Helper, which must be scrolled through to be believed, and Kids Web, produced at Syracuse University.

I'm unaware of any search engines like AltaVista or HotBot that have been or are being created by librarians. But there are many subject directories coming out of libraries. The Internet Public Library is probably the best known, and it contains several excellent features like POTUS (Presidents of the United States).

Our reference staff and some extremely Web-literate clerks and pages have put together what I think is a very good reference directory. (If you visit, note that we include both a direct link to each site plus the full URL. Including the URLs allows us to quickly print a directory page, ready for the customer to use outside the library.) Also check out the humongous project we've been working on this past summer, the Homework Center.

8 Ways to
Outsmart
an Engine

Here are a few tips
I've found useful when
using Web search tools
to do reference
work with students.

Plenty of other libraries have put up great reference directories. Creating and maintaining a page like ours takes a lot of time, but we use it many times daily. If you're in a smaller library, there may be a larger system nearby with a directory you can bookmark; the librarians who labor over these will be flattered.

Don't Blink!

This article is already obsolete. Change is the normal state of things in the web world, and brings all of us who spend a lot of time staring at a screen both hope and anxiety. I expect that search engines will become increasingly sensitive to our needs and more accessible. But in exchange, we may see fees, more "in-your-face" ads, and a smaller number of sites from which to choose.

The drive for better organization and cataloging of the Web is bound to increase as more "non-techies" get lost and confused among all the froth and advertising. There are customizable search engines in the works that will remember the kinds of searches we do and the addresses of the sites we access, and save that information in a database to help us when we perform our next search -- for a fee.

When will we see these developments? Some are happening now; industry pundits predict the rest will appear by 1998. In the meantime, don't cling too tightly to a favorite search engine, search strategy, or web resource, and don't teach web searching to students as if there's just one way to do it. That method may be moved, gone, changed, or sold tomorrow.

For Further Reading

Pfaffenberger, Brian. Web Search Strategies. New York: Henry Holt, MIS: Press, 1996.
Tweney, Dylan. "Searching Is My Business: A Gumshoe's Guide to theWeb," PC World (December 1996): 182196.
Westera, Gillian. "Robot-Driven Search Engine Evaluation Over-view." October 1996.

Related pieces: