Why fuzzy definitions are a problem in the social sciences

U.S. millennials are rejecting suburbia and moving back to the city. That was a prevailing idea in 2019, when I started as the social sciences reporter at Science News. But when I began digging into a possible story on the phenomenon, I encountered an incoherent mess. Some research showed that suburbs were growing, others that suburbs were shrinking and yet others showed growth in both suburbs and cities.

Unable to make sense of that maze of findings, I shelved the story idea. Then, several months later, I stumbled across a Harvard University white paper explaining that disagreement in the field stems from competing definitions of what distinguishes a city from a suburb. Some researchers define the suburbs as areas falling outside census-designated cities. Others look only for markers of suburbanism, such as a wealth of single-family houses and car-based commutes, the researchers wrote.

I have encountered this type of fuzziness around definitions of all sorts of terms and concepts in the years I’ve covered the social sciences. Sometimes researchers simply assume that their definition of a key concept is the definition. Or they nod briefly at other definitions, and then go forth with whichever one they choose, without much explanation why. Other times, researchers in one subfield choose one definition, and researchers in another subfield choose a different one — each without ever knowing of the other’s existence. It’s enough to drive any reporter to tear their hair out.

“If you look … you will find this morass of definitions and measurements” in the social sciences, says quantitative psychologist Jessica Flake of McGill University in Montreal. My experience was a common one, she assured me.

Definitional morasses exist in other scientific fields too. Biologists frequently disagree about how best to define the word “species” (SN: 11/1/17). Virologists squabble over what counts as “alive” when it comes to viruses (SN: 11/1/21). And not all astronomers are happy with the decision to define the word “planet” in a way that left Pluto out in the cold as a mere dwarf planet (SN: 8/24/21).

But the social sciences have some special challenges, Flake says. The field is a youngster compared with a discipline like astronomy, so has had less time to sort out its definitions. And social science concepts are often inherently subjective. Describing abstract ideas like motivation or feelings can be squishier than describing, say, a meteorite.

It’s tempting to assume, as I did until I began researching this column, that a single, imperfect definition for individual concepts is preferable to this definitional cacophony. And some researchers encourage this approach. “While no suburban definition will be perfect, standardization would increase understanding of how suburban studies relate to each other,” the Harvard researchers wrote in that suburbia paper.   

But a recent study taking aim at how we define the middle class showed me how alternative definitions can lead to a shift in perspective.

While most researchers use income as a proxy for class, these researchers used people’s buying patterns. That revealed that a fraction of people who appear middle class by income struggle to pay for basic necessities, such as housing, child care and groceries, the team reported in July in Social Indicators Research.  That is, they live as if they are working class.

What’s more, that vulnerable group skews Black and Hispanic, a disparity that arises, in part, because these families of color often lack the generational wealth of white families, says Melissa Haller, a geographer at Binghamton University in New York. So when calamity strikes, families without that financial cushion can struggle to recover. Yet a government or nonprofit organization looking to direct aid toward the neediest families, and relying solely on income-based metrics, would overlook this vulnerable group. 

“Depending on what definition you start with, you will see different facts,” says Anna Alexandrova, a philosopher of science at the University of Cambridge. A standardized definition of middle class, for example, could obscure some of those key facts.

In the social sciences, what’s needed instead of conceptual unity, Alexandrova says, is conceptual clarity. 

Though social scientists disagree about how to go about solving this problem of clarity, Flake says that failure to tackle the issue jeopardizes the field as much as other crises rocking the discipline (SN: 8/27/18). That’s because how a topic is defined determines the scales, surveys and other instruments used to study that concept. And that in turn shapes how researchers crunch numbers and arrive at conclusions.  

Defining one’s key terms and then selecting the right tool is somewhat straightforward when relying on large, external datasets. For instance, instead of using national income databases, as is common in the study of the middle class, Haller and her team turned to the federal government’s Consumer Expenditure Surveys to understand people’s daily and emergency purchases.

But often social scientists, particularly psychologists, develop their own scales and surveys to quantify subjective concepts, such as self-esteem, mood or well-being. Definitions of those terms — and the instruments used to study them — can take on a life of their own, Flake says.

She and her team recently showed how this process plays out in the May-June American Psychologist. They combed through the 100 original studies and 100 replications included in a massive reproducibility project in psychology. The researchers zoomed in on 97 multi-item scales — measuring concepts such as gratitude, motivation and self-esteem — used in the original studies, and found that 54 of those scales had no citations to show where the scales originated. That suggests that the original authors defined their idea, and the tool used to measure that idea, on the fly, Flake says. Research teams then attempted to replicate 29 of those studies without digging into the scales’ sources, calling into question the meaning of their results.

For Flake, the way to achieve conceptual clarity is straightforward, if unlikely. Researchers must hit the brakes on generating new ideas, or replicating old ideas, and instead interrogate the morass of old ones.

She points to one promising, if labor-intensive, effort: the Psychological Science Accelerator, a collaboration of over 1,300 researchers in 84 countries. The project aims to identify big ideas in psychology, such as face perception and gender prejudice, and accumulate all the instruments and resulting data used to make sense of those ideas in order to discard, refine or combine existing definitions and tools.

“Instead of running replications, why don’t we use [this] massive team of researchers who represent a lot of perspectives around the world and review concepts first,” Flake says. “We need to stop replicating garbage.”

I couldn’t agree more.