The Trouble with The Gold Standard: School Libraries & Research

Strongly held beliefs die hard, even when they are unfounded or lack evidence. Some still believe that fish is brain food and rumors of Elvis sightings continue to fly through cyberspace. Other times, unfounded beliefs are denials of an uncomfortable or “inconvenient truth.” For example, the denial of global warming was a dangerous, unfounded belief that persisted for years despite the presence of empirical evidence to the contrary.

For most of recorded history, however, superstition and tradition have been the accepted ways of knowing: they’ve addressed burning questions about life, death, and everything in between. Although Isaac Newton was a genius, he was convinced that he could transform metal into gold through alchemy. And as late as the 18th century, bleeding with leeches was a commonly accepted medical practice. In 1799, a physician using that technique miscalculated the number of quarts of blood in his patient’s body and the sick person died. The patient’s name? George Washington.

Nowadays, our unfounded beliefs frequently take the form of anecdotal evidence, which evolves from the human urge to convert experience into story. Anecdotal evidence often drives our decisions about what works. In the workplace, it’s known as professional opinion. Anecdotal evidence isn’t reliable because it’s based on an individual case, which may, in fact, be the only case that exists. This kind of evidence is not generalizable: we can’t, with validity, apply the anecdote to other cases, and certainly not to all or even most cases.

Although science has provided empirical evidence for centuries, it’s only as recently as the 1940s that a scientific method called randomized controlled trials (RCT) emerged as a technique to test the efficacy of drugs and medical procedures. In RCT, participants in a study are chosen randomly and randomly placed in either a group that receives an intervention or a group that doesn’t.

There’s a mathematical logic to random sampling that says if you flip a coin enough times and have enough people in your sample, the groups will be “statistically equivalent.” The only difference between the experimental and control groups, for the purposes of the experiment, is the intervention being tested. Since RCT is unique in that it can claim causality, it’s considered the “gold standard” of research because it yields the most reliable evidence.

Why isn’t gold-standard research used to establish the effectiveness of all instructional interventions, including the school library’s influence on student achievement? That’s what the U.S. Department of Education (DOE) would like to know.

You won’t find school library impact studies and other recent research on school libraries and student achievement in Effective Instruction: A Handbook of Evidence-Based Strategies, a new book that includes educational research that tests interventions (EDIE, 2006). Nor are school library studies among the 14 gold-standard studies published in the DOE’s What Works Clearinghouse (www.whatworks.ed.gov).

Research in school librarianship is considered educational research, and it’s vulnerable to the same criticism that has plagued this research for decades. According to the DOE’s 2003 guide, Identifying and Implementing Educational Practices Supported by Rigorous Evidence, “The field of K–12 education contains a vast array of educational interventions… that claim to be able to improve educational outcomes and, in many cases, to be supported by evidence. This evidence often consists of poorly-designed and/or advocacy-driven studies.”

The issue isn’t whether the gold standard generates the most reliable evidence. It’s whether it is the only way to determine what works. The claims of studies that don’t use RCT—including comparison studies such as the school library impact studies of Keith Curry Lance and others—are criticized because they’re not seen as controlling for variables other than how the school library may affect student achievement (even though these studies use statistical controls).

Does this mean the medical model is the best way to test instructional interventions in education? From a practical standpoint, randomized controlled trials involve large samples (a minimum of 300 students or 50 to 60 schools), because random sampling and statistical analysis require large samples to ensure that the effect is not occurring by chance.

Additionally, large samples presuppose healthy funding. RCT requires that half the participants in a study are randomly placed in a control group. That raises the problem of conducting the research outside of a school environment, since students aren’t always placed randomly in classes or aren’t placed randomly enough to ensure that the classes are statistically equal.

Even if you could transport hundreds of schoolchildren to a laboratory-like environment, the new setting would lack the social and cultural aspects (familiar classmates, a teacher with whom the students have rapport) of the original environment in which the intervention actually takes place. Unlike a scientific laboratory, a school is an organic, dynamic, complex institution with hundreds of variables intervening to complicate the “experiment.” And some of those variables—such as scheduling, diverse administrative and teaching styles, access to resources, and school climate—simply defy control.

Can it be that educational research is more complex, and less understood, than medical research?

Researchers choose their methodologies based on the kinds of questions they are investigating. There’s little disagreement that the quality of educational research needs to get better. According to the prevailing view among policymakers, educational progress won’t be made until we have research in hand to guide our decisions.

But what does getting better mean? Education is generally accepted as a social science; it’s a field that includes philosophy, psychology, and sociology. These disciplines ask qualitative questions that seek to understand the how and why of a phenomenon. For example, a historian may ask, “What really happened in the past?” That question, in turn, sparks an investigation that relies on primary documents and the content analysis of those sources. An historian, for obvious reasons, can’t conduct a survey to find out what people were really thinking during the Crimean War.

Likewise an educational researcher asks, “How do people learn?” as a prerequisite to determining what works. The appropriate methodology for such investigations is qualitative, and data is collected through observation, journaling, or other means. The Italian physician and educator Maria Montessori developed her model of schooling through observation and documentation.

The powerful theory of learning called constructivism originated with the work of Jean Piaget, the Swiss psychologist who observed and chronicled the development of his young daughters. Constructivism was behind many of the 20th century’s most prominent educational theories—from John Dewey’s child-centered, hands-on approach to learning to library researcher Carol Kuhlthau’s theory of “zones of intervention,” which articulates how information seekers move from uncertainty to understanding.

The generalizability of this research is in its cumulative effect: the resulting body of knowledge is a virtual gold mine that’s still yielding effective educational practices. Constructivist learning theory has led us to learner-centered, authentic teaching. It has helped us discover how to assess and remediate student learning and critically evaluate our teaching methods. It has presented evidence-based practice as a cornerstone of best practices and action research as a viable tool for improving our practices. We have already struck gold and should treasure that knowledge. No matter how rigorous gold-standard research is, when applied to educational research, it’s not the silver bullet that’s going to lead us to the “right” answer.