Correlation, Causation & BAD POLICY

You probably have heard the phrase “correlation vs. causation” many times.

Bottom line: Correlation DOES NOT equal causation.

While this phrase gets thrown around quite a bit, there are distinct technical definitions for both of these terms that aren’t always comprehended. While this bottom line is ubiquitously touted by media personalities and political leaders to rebut criticisms, it is frequently misunderstood by the individuals that use it as a shield.

What you don't know WILL hurt you...

Some of the most highly trained professionals lack basic statistical literacy.

We’d like to believe that most physicians are statistically literate. Sadly, there is significant evidence to the contrary. Here are just a few examples, which represent a drop in the proverbial bucket. 

A 2006 BMJ study found that only 1 out of 21 obstetricians are able to correctly calculate the probability of an unborn child with Down’s Syndrome conditional on a positive test.¹ 

A 2012 study published in the Annals of Internal Medicine found that 70% of U.S. primary care physicians erroneously recommend treatments based on screening tests.² 

A 2013 study published in The Journal of Graduate Medical Education found that only 12% of obstetrics-gynecology residents are able to correctly answer two elementary statistics questions³:

  • What is the meaning/significance of the P-Value?
  • How do you assess the positive predictive value of a mammography screening?

There are many more examples of physicians failing to demonstrate a basic understanding of fundamental statistical axioms and concepts. Unfortunately, there is no reason to believe that other educated professionals, on average, fare any better than physicians with respect to statistical literacy; the opposite is likely true. 

 

Figure 1: National Science Foundation Survey (2018)

Unfortunately, the public is not statistically literate. To illustrate this, here are some statistics from the National Science Foundation’s 2018 Science & Engineering Indicators study.† 

According to the organization’s surveys, in 2016, only 51% of Americans correctly understood the elements and methodology of a scientific experiment. 23% understood how to read a scientific study. And, most relevant to this post, only 64% understood basic principles of probability. 

For this third data point, the bar was low. To be classified as “understand probability correctly,” a respondent only had to answer these two questions correctly:

Situation: “A doctor tells a couple that their genetic
makeup means that they’ve got one in four chances of having a child with an inherited illness.

  • Does this mean that if their first child has
    the illness, the next three will not have the illness? (No – see Fig. 3)
  • Does this mean that each of the couple’s
    children will have the same risk of suffering from the illness? (Yes)”

So, this certainly does not mean that 64% of the American public has the statistical knowledge equivalent to the content of a standard Intro to Stat undergraduate course.

Widespread statistical illiteracy makes the confusion of correlation/causation highly probable (pun intended). This confusion has profound implications for public policy. 

Before I dive into HOW and WHY the confusion between correlation and causation leads to bad policy, I need to review those concepts. It’s not obvious why this confusion is harmful without a basic technical understanding. 

 

Figure 2: Definition of the P-Value
Figure 3: Two events A and B are statistically independent if and only if their joint probability can be factored into their marginal probabilities, which means that P(A ∩ B) = P(A)P(B). If two events A and B are statistically independent, then the conditional probability equals the marginal probability: P(A|B) = P(A) and P(B|A) = P(B).
Figure 4: Understanding scientific inquiry, by respondent (2018) - NSF Graph

CORRELATION

Correlation is a concept from statistics that describes an association between one variable and another variable, or between one variable and multiple variables. Simply put, correlation refers to a mutual relationship or connection between two or more things.

Correlation can be negative. For example, academic performance and school absences are negatively correlated. Higher numbers of missed school days are associated with lower test scores; perfect attendance records are associated with higher test scores.

Correlation can also be positive. For example, cigarette smoke and lung cancer are positively correlated. Higher smoking rates are associated with higher lung cancer rates and lower smoking rates are associated with lower lung cancer rates.

In statistics, correlation generally refers to linear relationships, which are measured by the correlation coefficient. While it’s not necessary to understand how this figure is calculated, it is very useful for providing on intuition of what correlation is. This correlation coefficient ranges from -1 to 1 where -1 represents a perfect negative linear relationship, 1 represents a perfect positive linear relationship and 0 represents the absence of any linear relationship. Correlation can also refer to non-linear or curvilinear relationships, but these are outside the scope of this post.

Figure 5: Correlation is commonly illustrated with a scatterplot
Figure 6: Formula for Correlation Coefficient (this is symbolized by the letter "r" or the Greek letter Rho)
Figure 7: Correlation Coefficient examples

CAUSATION

Causation refers to how one event, person, state or process contributes to the production of another event, process or state, either fully or partially. Put simply, causation is the act of directly causing something. 

Unlike correlation, causation isn’t a mathematical measure that can be directly derived into a number with data. However, through econometric analysis and experimentation, we can make accurate determinations of whether one variable x has a direct effect on another variable y – in other words – whether x causes a change in y.

Spurious Correlation & Dubious Inference

Figure 8: Maine Divorce Rate & Margarine Consumption Comparison

Divorce rates have been rising in the U.S. for the past several decades. Yet, Maine’s divorce rate has dropped.

Could this be due to the deleterious effects of margarine on the institution of marriage? I doubt it.

If someone looked at the Fig. 8 graph and assumed that margarine consumption increased divorce rates, they would be confusing correlation with causation.

In economics, the statistical association in the graph above is referred to as spurious correlation, which is illustrated and defined in Fig. 9. Divorce rates, X, and margarine, Y, are both associated with some endogenous variable or set of variables Z, that causes them to move together. These two variables have no logical connection. In the emerging age of analytics and big data, it is very easy to generate spurious correlation from gargantuan datasets. 

Figure 9: Definition of Spurious Correlation

Public Consequences of Statistical Illiteracy

The correlation/causation distinction is a critical concept when it comes to developing public policy. We love to make decisions that are “evidence-based” (or at least we love to claim that our decisions are evidence-based). Unfortunately, people can make decisions based on actual evidence while still misunderstanding the distinction between correlation and causation. As we saw is the case with physicians, valid statistics can be referenced when making recommendations with invalid interpretations of those statistics. 

Policies are designed to solve problems. For example, regulations that place restrictions on greenhouse gas emissions, such as CO2 and methane, are designed to combat the problems of climate change and environmental destruction. The collection of legislation that was passed following the Civil Rights Movement sought to begin solving the problems of overt racial discrimination and other more subtle consequences of racial injustice. Effective policies target the root cause or main causes of a problem. Ineffective, and often counterproductive, policies target the symptoms of a problem or the perceived symptoms of a problem, i.e. policies formulated by individuals who fail to recognize the difference between correlation and causation.

Suppose legislators believe that a variable, Y, is a problem and they believe a variable, X, is the cause due to the strong correlation between the two variables. If a policy is targeted at variable X, which is correlated with a variable Y, and the nature of the correlation is spurious, the policy will not actually solve the problem when implemented. The reason for this is that there is some other variable (or collection of variables), Z, that is causing the problem Y. X does not cause Y. Rather Y moves with X merely because they’re both correlated with Z.

For example, criminologists have found a strong correlation between the level of crime (# of homicides, for example) in a city and the number of police officers in a city. It is incorrect to conclude that larger police forces cause greater levels of crime. What is the Z variable in this case, i.e. what factor is positively correlated with both a city’s crime level and the size of a city’s police force? That would be the population of the city. Larger cities have more people. So, even if two cities have identical crime rates (measured by homicides per 100,000 residents, for example), the larger city will have a greater absolute level of crime. Larger populations require more police officers. It would be a mistake to fire police officers due to a mere positive correlation between total crime incidents and total number of police officers.

The use of the adjective "evidence-based" is ubiquitous, but largely cosmetic in nature.
When the market fails due to externalities that distort supply/demand equilibria, sound government policy can optimize outcomes. Solutions based on simplified understandings of causation are bound to fail.
Social gatherings, X, are correlated with drunk driving deaths, Y. Most would agree that banning social gatherings would be a poor way to prevent drunk driving deaths because the real cause of drunk driving deaths is alcohol consumption, Z, with which both X and Y happen to be correlated.

The Seduction of Narrative and Univariate Reductionism

Humans are drawn to narratives. We orient ourselves to the world through stories. In our spare time, we read novels and watch TV shows to lose ourselves in an entertaining narrative. Stories have good guys and bad guys. Plots with less clearly defined hero/villain roles are harder to follow. Most stories have a protagonist and an antagonist. That’s the way most of us like it. 

Politicians, the media, and most everyone else paint the world in black and white terms. Trump says that multi-national trade deals are the main cause of the decline in the manufacturing sector. In today’s political climate, many feel that racism is the sole cause of disparities in wealth between different racial groups. Both of these lines of thinking follow a clear-cut, digestible narrative. They illustrate the single predictor methodology shown in the first model of Fig. 10 and 11.

Is there truth to both of these narratives? Definitely. Do these narratives omit other important considerations? Certainly. 

These stories are essentially simple linear regressions, as seen in Fig. 10 & 11: they explain a particular social outcome SOLELY in terms of a single variable (or two or three). Any competent social scientist will tell you that the epistemological utility of a univariate or bivariate analysis is essentially zero. Another term for a simple linear regression is a NAIVE regression, which seems appropriate. It is naive to describe the world in this way. Whenever someone references a disparity, such as the gender pay gap, as conclusive proof of wrongdoing, they are confusing correlation and causation. 

Sadly, this is how most of us view the world: a montage of black and white narratives. Every raw correlation is a univariate analysis where one variable, X, is said to cause an outcome, Y. Policy formulated from this naive reasoning is ill-advised.

Figure 10: Linear regression model examples
Figure 11: MLR vs. SLR illustration
As Andrew Yang highlighted in his Presidential campaign, Trump's narrative does not address automation and technological change.
Who doesn't love a good story with a compelling plot and clearly defined characters?

Case Study: The Gender-Equality Paradox in STEM

The tweet shown in Fig. 12 is a classic example of bivariate reductionism. The Senator undoubtedly fails to understand the difference between correlation and causation. The narrative states that variances in pay between men and women are caused by gender and race (Sanders also fails to state that Asian men earn more than white men). While econometrically dubious, his message resonates with the public. 

A 2018 study published in the Journal of Psychological Science uncovered a surprising paradox: the more gender-equal (think Norway, Netherlands, etc.) a country is, the lower the percentage of women that are employed in STEM fields in the country.‡ Countries with low levels of gender equality (Turkey, Algeria, etc.) have much higher rates of female employment in STEM, which stands for science, engineering, technology and mathematics. See Fig. 13 for a visual. 

Intelligence and cognitive capability had nothing to do with this difference:

“…girls performed similarly to or better than boys in science in two of every three countries, and in nearly all countries, more girls appeared capable of college-level STEM study than had enrolled.”

Yet, the differential wasn’t driven by discrimination:

Paradoxically, the sex differences in the magnitude of relative academic strengths and pursuit of STEM degrees rose with increases in national gender equality. The gap between boys’ science achievement and girls’ reading achievement relative to their mean academic performance was near universal. These sex differences in academic strengths and attitudes toward science correlated with the STEM graduation gap.”

The authors concluded that a lack of opportunity in less gender equal countries motivates women to pursue careers based on the financial prospects of the profession. It is more likely for a woman to achieve financial security in Algeria, for example, with an engineering degree than an English degree. STEM degrees tend to lead to higher paying jobs than degrees that lead to jobs in other fields dominated by women, such as nursing and teaching.

Gender equal countries, on the other hand, provide more opportunities for women. Thus, when given the freedom to choose their careers and life paths, women tend to make different choices than men due to a wide variety of factors. 

Figure 12: Sen. Bernie Sanders' Tweet
Figure 13: Graph from Study
Misogyny is not the only culprit...

Does sexism play a role in this disparity? NO DOUBT. However, as this study has shown, univariate (or bivariate) reductionism is a naive mistake. Policies that seek to increase the percentage of females in STEM by targeting sexism alone will not be successful; the most gender equal countries in the world have the lowest rates of women in STEM. Our addiction to narrative, coupled with our statistical ignorance, hobbles our ability to solve problems with well-crafted policies. 

Sources Referenced:

  1. Bramwell R, West H, Salmon P. Health professionals’ and service users’ interpretation of screening test results: experimental study. BMJ. 2006;333:284–286.
  2. Wegwarth O, Schwartz LM, Woloshin S, Gaissmaier W, Gigerenzer G. Do physicians understand cancer screening statistics: a national survey of primary care physicians in the US. Ann Intern Med. 2012(156):340–349.
  3. Wegwarth O. Statistical illiteracy in residents: what they do not learn today will hurt their patients tomorrow. J Grad Med Educ. 2013;5(2):340-341. doi:10.4300/JGME-D-13-00084.1

† National Science Board. (2018). “Public Knowledge about S&T.” National Science Foundation. https://www.nsf.gov/statistics/2018/nsb20181/report/sections/science-and-technology-public-attitudes-and-understanding/public-knowledge-about-s-t

‡ Stoet G, Geary DC. The Gender-Equality Paradox in Science, Technology, Engineering, and Mathematics Education [published correction appears in Psychol Sci. 2020 Jan;31(1):110-111]. Psychol Sci. 2018;29(4):581-593. doi:10.1177/0956797617741719

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.