Calling Bullshit: The Art of Skepticism in a Data-Driven World, by Carl Bergstrom and Jevin West, delivers a thoughtful introduction to critical thinking for the serious reader. Its provocative title is clearly intended to attract readership. After all, how many people would pick up a book simply named for the subtitle, The Art of Skepticism in a Data-Driven World. Despite the clickbait title, the book delivers a novel approach that goes well beyond looking at bullshit to broader critical thinking. It will benefit anyone assessing the news, a friend’s claim, or a scientific analysis.
The authors are both professors at the University of Washington and the volume has its roots in their course INFO 270 / BIOL 270 – Calling Bullshit: Data Reasoning in a Digital World. Like their course, this is not a fluff volume with superficial content, but a thorough consideration of the topic in an accessible form.
The book begins with a description of the different types of bullshit, a concept as old as Plato, who complained about philosophers who cared little of the truth, only about winning arguments. It then describes why bullshit is such a problem. Drawing on the writings of Alberto Brandolini, Uriel Fanelli, and Jonathan Swift, the authors note:
- bullshit takes less work to create than clean up
- takes less intelligence to create than to clean up and
- spreads faster than efforts to clean it up
The authors divide the topic into two phases: 1) spotting bullshit, and 2) calling bullshit. The bulk of the book is devoted to spotting bullshit, with the final chapter addressing how to call bullshit.
Readers might avoid thinking critically in our data-driven world because they lack confidence in their math, statistics, or computer science skills. The authors reassure readers that such knowledge may not be required. To do this, they divide the critical analysis process into three parts: 1) the inputs to the analysis, 2) the analysis (considered as a black box, i.e., statistical procedures or data science algorithms), and 3) the results of the analysis.
While an analysis may be technical and beyond the average reader’s understanding, it is often not essential to a critical evaluation. By looking at the inputs to an analysis and the results, a reasonably informed person can effectively judge the validity of a claim. As the authors observe “If the data that go into the analysis are flawed, the specific technical details of the analysis don’t matter.”
The authors illustrate their approach by examining of a 2016 research study that claimed to be able to identify criminals with 90% accuracy from facial images. By looking at the biased source inputs and outputs that confuse facial expressions with facial features, the authors discredit the claim without evaluating the internal analysis of the data.
The book goes into great depth for each topic in its chapters on “Causality,” “Numbers and Nonsense,” “Selection Bias,” “Data Visualization,” and “Calling Bullshit on Big Data.” There is simply too much good content to describe here, but some examples from “Numbers and Nonsense” give an idea of their approach.
“Numbers and Nonsense” was one of my favorite chapters. It contains pithy insights on the nature of numbers. For example, “Data can help us understand the world based upon hard evidence, but hard numbers are a lot softer than one might think.”
They note the difference between numbers like counts and numbers which are estimates. Estimates are broken down into direct and indirect assessments, noting that “There are many ways for error to creep into facts and figures that seem entirely straightforward.” This is clearly and entertainingly described with a whiskey distillation example.
A good deal of attention is paid to Pernicious Percentages. The examples are highly relevant to discussions of topics like COVID-19 and election results, where data is often described using techniques to favor the writer’s perspective. The authors note that
Listing a raw total like this can make a small quantity, relatively speaking, appear to be large. We put this number into context by expressing it as a percentage. …. But percentages can also obscure relevant comparisons in a number of ways. For starters, percentages can make large values look small.
They illustrate this point with a decision on drinking a 99.9% caffeine-free coffee before bed. This sounds like a good choice, until it is revealed that strong coffee is also 99.9% caffeine-free.
The chapter ends with an amusing example of mathiness where formulas and expressions might look like math, “while they disregard the logical coherence and formal rigor of actual mathematics.” One example of this is the Trust Equation
The general concept of the equation might be correct, i.e., trust is positively associated with credibility, reliability, and authenticity; and trust is negatively associated with the perception of self-interest. However, this is not a mathematical equation!
For example, you could still have large trust if reliability is high, but you have no credibility or authenticity. Also, if there is no self-interest, trust would be infinite. And then there is the issue of units. The units on each side of a math equation must be equal, but this is not possible with an equation that lacks any units.
When wrapping up the spotting bullshit topics, the authors ask the question “If bullshit is everywhere, how can we avoid being taken in?” The “Spotting Bullshit” chapter summarizes six simple, but powerful guidelines to help you avoid being “taken in.”
The book’s final chapter, “Calling Bullshit,” or refuting claims, is not merely about reporting skepticism, but declaring it publicly. Once identified, calling bullshit is a delicate and tricky proposition, as it can be inflammatory and taken personally. The authors suggest that you keep the discussion simple, take it offline, find common ground, don’t overemphasize the myth, and fill in knowledge gaps with alternate explanations. First and foremost, you need to be correct. But that is not enough, you also need to be charitable, clear, and pertinent, as well as admit fault where appropriate.
In summary, Calling Bullshit: The Art of Skepticism in a Data-Driven World, will significantly improve any reader’s critical thinking. The idea that bullshit can often be detected by only looking at inputs and outputs empowers the general reader by identifying the necessary skills to evaluate arguments and claims without a deep understanding of statistics or data science. This is both novel and powerful.
Resources
Calling Bullshit: The Art of Skepticism in a Data-Driven World. The printed book.
Calling Bullshit. The web site includes videos, tools, and case studies.
INFO 270 / BIOL 270 – Calling Bullshit: Data Reasoning in a Digital World. The class syllabus.
Calling Bullshit Class Lectures. Fifty-six lectures from the class are available on YouTube.
Carl Bergstrom. Bergstrom is a Professor of Biology at the University of Washington.
Jevin West. West is an Associate Professor in the Information School at the University of Washington.