Calling Bullshit: The Art of Skepticism in a Data-Driven World
The title is catchy, isn’t it? Dive deeper and you shall see that the authors are university professors and the book is based on their popular course at the University of Washington. On their website they stated the aim is to teach you how to think critically about the data and models that constitute evidence in the social and natural sciences. I use a lot of information from this book to teach students how to read a paper. The followings are bits and pieces that I find useful.
How would the book benefit me?
Simple, can you tell when a clinical trial reported in the New England Journal or JAMA is trustworthy, and when it is just a veiled press release for some big pharma company?
The example:
While short of statistical significance (p = 0.13), our results underscore the clinically important effect size (relative odds of survival at five years = 1.3) of our targeted oncotherapy and challenge the current therapeutic paradigm.
What does it mean for a result to be clinically important if it is not statistically significant? Is five-year survival a relevant measure for this particular cancer, or are most patients deceased within three years? Why should we imagine that any of this “challenges the current therapeutic paradigm”?
That was insightful as we go through our literature we see this all the times.
The bullshit problem is wide-spread and exacerbated by the internet.
The rise of the Internet changed what kinds of information get produced, how information is shared, and the ways in which we find the information that we want.
Prior to the Internet, newspapers and magazines made money by selling subscriptions. Subscribing to a periodical, you embarked on a long-term relationship. You cared about the quality of information a source provided, its accuracy, and its relevance to your daily life. To attract subscribers and keep them, publishers provided novel and well-vetted information
Your click generates advertising revenue for the site’s owner. The Internet site is not necessarily designed to perpetuate a long-term relationship; it is designed to make you click, now. Quality of information and accuracy are no longer as important as sparkle. A link needs to catch your eye and pull you in.
If I come across a political leaflet or poster on the street, I am immediately skeptical. If my dear uncle forwards me a story on Facebook that he “heard from a friend of a friend,” my guard drops. Disinformation flows through a network of trusted contacts instead of being injected from outside into a skeptical society
Simple questions to ask oneself when reading literature.
Are the data unbiased, reasonable, and relevant to the problem at hand?
Do the results pass basic plausibility checks?
Do they support whatever conclusions are drawn?
How it is said matters.
We might express a correlation with a plain factual statement in the indicative mood: “If she is a Canadian, she is more likely to be bilingual.” We express causation using a counterfactual statement in the subjunctive mood: “If she were a Canadian, she would be more likely to be bilingual.” The former statement simply suggests an association. The latter statement suggests that being Canadian causes bilinguality.
Suppose that on January 1, the sales tax increases from 4 percent to 6 percent of the purchase price. This is an increase of 2 percentage points: 6% − 4% = 2%. But it is also an increase of 50 percent: The 6 cents that I now pay on the dollar is 50 percent more than the 4 cents I paid previously. So the same change can be expressed in very different ways that give substantially different impressions. If I want to make the tax increase sound small, I can say that there has been only a 2 percentage point increase. If I want to make it sound large, I can say that taxes have gone up 50 percent. Whether accomplished by accident or intent, we need to be wary of this distinction.
When a measure becomes a target, it ceases to be a good measure. Impact factor doesn’t mean as much as it used to as it’s been distorted.
The use of citation metrics to measure journal quality has led editors to game the system. Some pressure authors to include citations to papers in the same journal. Some publish an excess of articles in January, when they have the most time to be cited during the year. Others publish annual summary papers that cite many of the articles published within a year; yet others shift their focus to disciplines or types of articles that tend to attract more citations. All of these perverse behaviors undermine the mission of the journals and the effectiveness of citation measures as indicators of quality.
This is exactly what has happened in Endodontics.
A healthy dose of visual skepticism
The point of data graphics is often to provide a quick and intuitive glimpse into a complex data set. Always look at the axes when you see a data graphic that includes them.
A bar graph emphasizes magnitudes, whereas a line graph emphasizes the changes. As a result, a bar graph should always have a baseline at zero, whereas a line graph is better cropped tightly to best illustrate changing values
Ask yourself whether a graph has been designed to tell a story that accurately reflects the underlying data, or whether it has been designed to tell a story more closely aligned with what the designer would like you to believe.
And the holy grail…the great and powerful p-value
Loosely speaking, a p-value tells us how likely it is that the pattern we’ve seen could have arisen by chance alone. If that is highly unlikely, we say that the result is statistically significant.
By looking at enough different hypotheses, I’ll always find some sig results even if there's no real pattern or colloquially, a statistical fishing expedition.
Rather than representing a definitive fact about nature, each experiment or collection of observations merely represents an argument in favor of some particular hypothesis. We judge the truth of hypotheses by weighing the evidence offered from multiple papers, each of which comes at the issue from a different perspective.
A single study doesn’t tell you much about what the world is like. It has little value at all unless you know the rest of the literature and have a sense about how to integrate these findings with previous ones.
The first thing to recognize is that any scientific paper can be wrong. That’s the nature of science; nothing is above questioning. No matter where a paper is published, no matter who wrote it, no matter how well supported its arguments may be, any paper can be wrong. Every hypothesis, every set of data, every claim, and every conclusion is subject to reexamination in light of future evidence
How to navigate online info?
Corroborate and triangulate-check different sources, Journalists are trained to ask the following simple questions about any piece of information they encounter: Who is telling me this? How does he or she know it? What is this person trying to sell me
Pay attention to where information comes from
Dig back to the origin of the story
Use reverse image lookup
A few frames from a video, and the search engine tells you where on the Web that picture or video can be found. This is one of the more underutilized tools on the Web for fact-checking. If you are suspicious of a Twitter or Facebook account, check to see if the profile photo comes from a stock photo website.
Be aware of deepfakes and other synthetic media
Use professional fact checking sites e.g. snopes.com , factcheck.org
Consider a website’s track record. How do you know if a website is reliable? Try to find out if the site has been known to create and push fake news sources. Wikipedia often provides an overview of media outlets; this can be a good place to start. No one gets the facts right all the time, so see if the website issues corrections. Is the site reflective about the challenges it faces in getting at the truth
Be aware of the illusory truth effect. The more often you see something, the more likely you will be to believe it. Think more and share less. If you are not certain that it is true, sharing it contributes to the spread of misinformation and could do harm.
Reduce your information intake. Take a break This will enhance your ability to process information with skepticism when you are online.
If you don’t like to read I encourage you to watch. The authors’ entire lecture series can be freely viewed on https://www.callingbullshit.org/videos.html