Most specifically, there are a variety of different approaches that people use to address this causality issue. I kind of arrayed them here on a chart because there's a real tradeoff here, I think, between difficulty and effectiveness. You know, the things where you really think, yeah, we absolutely are convinced that this is what's driving performance, tend to be quite hard to do, There are a lot of easier things that give us more comfort but can't be definitive. And so the simplest thing, if we're using some sort of regression framework, is this idea of measuring and controlling the omitted variables. If we believe that there's some other factor that's driving a predictor that's a potential limited variable. If we can include that in our aggressions and control for it, or if we can make sure that the people, the high and the low performers, have same levels of that omitted variable, then we can say, no, that's not what's going on. And so that's first strategy. Kind of a variance of that is trying to hold the person constant, and so sometimes looking at changes and performance of the same individual before and after some kind of intervention, training, or something like that, can be a way to avoid a lot of these omitted variables. It's not perfect, because as we saw in the example of people improving in training because its people who are experiencing a dip who go in, there are also some things even the same person that change at the same time as our intervention. But this is kind of the first thing that we do. The reason why this doesn't always work is that there a lot of things we can't measure with the precision that we would like to. So a lot of the early work on this was done looking at the question of the value of education. How much money do we get for each extra year of education? People are always worried about, well, is that driven by the effects of what we're learning, or is that driven by smarter people getting more education? Obviously, as a PhD, I'm deeply wedded to that second explanation. The question was can we control for ability? Obviously you can throw in things like GPA and so on, but we always know GPA, grades, are not the same thing as being smart. And so a big challenge was always how do we really feel confident that we have measured ability? Because there's always going to be a lot of things that you haven't measured, and that's going to compromise this approach. The second thing you can do that's on the easier side is what's evidence that you can use to rule out alternatives? If you think there's another reason why you might be seeing the same pattern, what could you do to address this? I talked a little earlier about some work that I did where I was trying to understand whether promotion or hiring was more likely to lead to high performance. One of the challenges with that work was I was using performance evaluations. And so I'd find that people who are promoted perform more highly. Well, could that just be that their evaluation's more biased? The managers know them better, they like them better, they give them better evaluations. It's hard to rule that out. One thing that I could look at is at least say, well, sometimes these evaluations are more objective and sometimes they're more subjective. So they're going to be more objective when you're thinking about evaluations that are tied to results, did they achieve results, and less objective when you're thinking about competence. They're going to be more objective when you're looking at a job like sales where there's a hard number at the end of the year, less objective when you're looking at something like advisory business. And so what I did was, I at least compared what the effects looked like when they're more objective versus less objective. The effects were stronger on measures that were more objective, which gave me some comfort that this wasn't a biased story. So always think about, okay, what other things do those alternative explanations suggest and can I find evidence for them in the data, is another way to at least get some comfort around causality. Ultimately, to really feel confident here, what you need is something that looks more like an experiment, okay? In experiment, what we do is we randomly assign people to a treatment condition, and then we have controls and we compare the difference between them. One way to do this is look for places where the experiment has already been done with these. So people describe these as natural experiments. What are conditions under which people have been pretty much randomly assigned to the treatment or at least their assignment to getting treatment has nothing to do with performance. A famous example of this in a different setting, again this question people are interested, how much does education affect earnings? Well, it turned out during the Vietnam War in the U.S. there was conscription. And there was a draft. And how likely you were to be drafted depended on a lottery number that you got. One way to get out of military service was through going to university. And so depending on the draft number people got, they were more versus less likely to actually go to university. And so people are just being randomly assigned to higher versus lower probability of going to university. And this was used to try and estimate, okay, if they're being randomly assigned this, then we have more confidence that we can get a sense of what the effect of going to a university on pay is because we know there's some component of this that has nothing to do with their underlying ability, ability to learn, or anything like that. So looking for those kind of trials, those natural experiments where we know something's been randomly assigned, is a great way to be more confident about causality. The big challenge here, obviously, is you need to be lucky. Not everything is randomly assigned. Chances are your training was not randomly assigned. And if it isn't, you can't use this kind of approach. In that case, what you can do is conduct an experiment. If this is something that matters enough to you, then persuade people that you want to randomly assign training, and see what the impact is. And that way, you really can have a decent control in treatment group, and you can even make sure that those groups are really balanced in terms of some of the other characteristics of individuals that you think might affect the outcome. This is the gold standard. It's what we do in medical trials. If we really want to know the answer, it's the right thing to do. Obviously the challenge here is it's expensive, right? You need to persuade people to let you do the experiment. It's going to take you time to set up the experiment. It's going to take time to run. In a way, the often archival data it's all already there, okay? And so that's the tradeoff here. But always something that you kind of want to think about as you look at these studies is, why am I seeing the results that I'm seeing? And if this really matters, what can I do to be absolutely sure that my predicted variable is driving this performance in the way that I think it is? How much am I willing to invest in finding that out?
