In quantum mechanics there is something called the Heisenberg Uncertainty Principle: we can never know both the exact location and momentum of a particle – at small enough scales the more precise we are with one, the less precise we can be with the other.  I just realized that when it comes to drawing policy implications, development economics might have an uncertainty principle of its own.
 
Esther Duflo spoke at TED this week, talking about the need for evidence and experimentation in development.  She’s on one side of an ongoing debate about the value of randomized control trials in development economics.  “Randomistas” as Angus Deaton calls them, are suspicious of evidence of causal inference that does not result from randomization, because only through randomization can we be sure that treatment and control groups have the same characteristics.  It’s only if the two groups are the same in every way but the treatment that we can be positive of a policy’s effect on outcomes (what economists call internal validity).

This is of course impossible at the macro level.  We obviously can’t randomly assign policies to countries and even if we could there aren’t enough of them to be sure of our conclusions.  Instead at the macro level we use creative econometrics to do our best to isolate the relationship between policies and outcomes.  And even then we run into issues of reverse causality.  This gives macro level studies an issue with internal validity.  It’s messy econometrics.

Opponents of randomized experiments argue that their results can’t be generalized to a broader context (i.e. they aren’t externally valid).  They may be useful for academic research, but they are of limited policy value.  Messy econometrics, on the other hand, incorporates many places across many time periods, which makes a stronger case for their results being broadly applicable.

Do you see where I’m going with this?  Could it be that development economics has to live with a minimum level of uncertainty with its experimental outcomes, trading off internal and external validity?

Dani Rodrik’s “We Shall Experiment, But Shall We Learn?” elaborates (and inspired this post).  I highly recommend it, he even uses one of our favorite examples, bed nets for malaria, to explain the merits of various experimental approaches.  The real aha moment hit me on page 4, talking about Cohen and Dupas' random control trial:

“But do the results extend to other settings in Africa as well? One can certainly make the case that it does, but the arguments one would need to deploy are perforce informal ones and they are convincing to varying degrees.  In fact such arguments are not too different in kind from those that researchers may offer in defense of a set of instrumental variables employed in a conventional econometrics study with weaker internal validity.” 

I knew my physics degree would come in handy some day.