ESSA

ESSA

Surely a well-researched, promising solution by economists would work in the real world, wouldn’t it?


Ze Xin Yuan

By

March 20th, 2020


In his first article, Ze Xin Yuan highlights the discrepancy between economic research in academia and its application in the real world.


Experiments by economists are no longer an academic curiosum. Insights from experiments have been endorsed by policymakers over the years to support their policies as scientific evidence makes their policies more convincing to the public.

However, that does not mean a policy that is backed up by experimental evidence will always work. For example, the program ‘Parent Academy’ by economist John List (teaching parents how to improve their child’s cognitive skills and executive functions skills such as self-control) failed to scale up in the U.K. when carried out across London. According to himself: ‘It failed miserably because no parents actually signed up’.[1]This is not an uncommon phenomenon, findings from Tennessee STAR suggested reducing class size will lead to a better academic outcome.[2] However, when implementing it to scale, people have found the results to be way poorer compared to the initial findings.[3]

So what is the problem? Were the initial researches flawed and inaccurate? It turns out the problem was that people have a lack of understanding of how to scale results. In other words, we lack knowledge of the science of using science.[4]

Statistical inference and scalability[5]

The first issue is that competition between independent research teams will cause researchers to commit greater inferential errors when interpreting data.[6] In other words, researchers commit more false positives (a result that indicates that a given condition is present when it is not) when competition intensifies.

Publication bias exacerbates this problem.[7] Publication bias is a bias which encourages publishers to cherry pick findings that show significant results.

The way academic journals work also promotes findings of false positives.[8] The reasoning behind it is that only significant academic findings are published by journals. The results are drawn from a distribution of results for the given subject according to their significance. However, if one draws results from the high end of the distribution (ranking them by significance), one is taking the findings that are more likely to be statistically unlikely despite there being no errors in interpreting the results by the researchers themselves.[8]

Representativeness of the population: the attributes and behaviour of participants and scalability[9]

Unrepresentativeness of population is an issue for scaling. If initial findings are based on a population that is unrepresentative of the real world, applying them to scale (to the real world) might produce different results.

This type of bias can occur when researchers strategically choose a population that is likely to give them a significant result.

It can also occur because participants who care more about the research are more attracted to the experiment and are likely to participate. For example, in an educational experiment, people who care more about education are more likely to attend. Because of the fact that these participants care more than ordinary people, a larger effect is likely to be found.

The limited budget of researchers also plays a role in creating the problem. One of the goals of the researcher is to get more participates with limited money. However, because the people who care more about education require less compensation for their time (since they value the program more and are compensated by the program), they are more attracted to the program, hence a larger effect is more likely to be found.

Unnaturally high levels of compliance are also a source of unrepresentativeness.[10] Participants in experiments are more likely to comply with instructions compared to the general public. This may be caused by the fact that experiments attract compliant participants because of its selection procedures, researchers favouring compliant participants, or the environment of the research encourages higher levels of compliance.[10]

Representativeness of the situation: the attributes and behaviour of administrators and scalability[10]

In experiments, the administrators are compliant of the directions given. Since they have incentives to maximise the effects of the project. However, when the findings are administered in the real world, the person carrying out the program does not have such incentives. The original team may not care either since their discovery is complete.

So, what to do about it?[11]

More replications of existing experiments should be encouraged. Unless policy makers are 95% certain the result is true, which requires 3 or 4 ‘well powered independent replications of original findings’, the finding should not be implemented into policies.[12] This in turn requires adjustments of incentives in the research sector, including linking wages and funds of researchers not only to original findings, but also to provide adequate funds to replicate existing experiments since they are equally important to the world. When this incentive is put in place, it will also eliminate some of the biases researchers face since they now have an incentive to keep their research replicable.

Most importantly, people should draw their attention to exploring implementation science, the key to make polices and research more valuable to our whole society.


[1] M. Hickey (Producer). (2020). Policymaking Is Not a Science (Yet) Retrieved from https://freakonomics.com/podcast/scalability/

[2] Mosteller, F. (1995). The Tennessee Study of Class Size in the Early School Grades. The Future of Children, 5(2), 113-127. doi:10.2307/1602360

[3] Hippel, P. von and Wagner, C. (2018), ‘Does a Successful Randomized Experiment Lead to Successful Policy? Project Challenge and What Happened in Tennessee After Project STAR

[4] Al-Ubaydli, O., List, J. A., & Suskind, D. (2019). The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments. National Bureau of Economic Research Working Paper Series, No. 25848. doi:10.3386/w25848

[5] Al-Ubaydli, O., List, J. A., & Suskind, D. L. (2017). What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results. American Economic Review, 107(5), 282-286. doi:10.1257/aer.p20171115

[6] Maniadis, Z., Tufano, F., & List, J. A. (2014). One Swallow Doesn’t Make a Summer: New Evidence on Anchoring Effects. American Economic Review, 104(1), 277-290. doi:10.1257/aer.104.1.277

[7] Young, N., Ioannidis, J., & Al-Ubaydli, O. (2008). Why Current Publication Practices May Distort Science. PLoS medicine, 5, e201. doi:10.1371/journal.pmed.0050201

[8] Al-Ubaydli, O., List, J. A., & Suskind, D. L. (2017). What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results. American Economic Review, 107(5), 282-286. doi:10.1257/aer.p20171115

[9] Al-Ubaydli, O., List, J. A., & Suskind, D. (2019). The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments. National Bureau of Economic Research Working Paper Series, No. 25848. doi:10.3386/w25848

[10] Al-Ubaydli, O., List, J. A., & Suskind, D. L. (2017). What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results. American Economic Review, 107(5), 282-286.

[11] Al-Ubaydli, O., List, J. A., & Suskind, D. (2019). The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments. National Bureau of Economic Research Working Paper Series, No. 25848. doi:10.3386/w25848

[12] M. Hickey (Producer). (2020). Policymaking Is Not a Science (Yet) Retrieved from https://freakonomics.com/podcast/scalability/

Image: 16 MacNelly – Dist. by King Features, Gary Brookins and Susie MacNelly

The views expressed within this article are those of the author and do not represent the views of the ESSA Committee or the Society's sponsors. Use of any content from this article should clearly attribute the work to the author and not to ESSA or its sponsors.

Founding sponsors

 

 

Partner

Platinum sponsors

Gold sponsors

 

 

 

 

Silver sponsors

 

 

 

 


Affiliates