Why are natural experiments important?

David Card, Joshua Angrist and Guido Imbens won Nobel economics today.

The work of David Card, Joshua Angrist and Guido Imbens, who were awarded the Nobel Prize in Economics on Monday, is based on "natural experiments", an innovative method of empirical research developed in the 1990s.

Natural experiments are real-life situations that economists study and analyse to determine cause-and-effect relationships.

In some ways they are similar to clinical trials, in which researchers evaluate the effectiveness of new drugs by separating test and control groups at random.

"We are replicating something that could be done in a laboratory," says Julien Pinter, a researcher at the University of Minho in Portugal and an economist at BSI Economics.

But doing something in the controlled conditions of a laboratory and doing it out in the world are two very different things.

Natural experiments differ from therapeutic trials in that -- unlike scientists in the lab -- economists do not control the parameters of the experiment.

The scope of these studies is vast: in the cases of the Nobel winners, they covered education, the labour market and immigration.

- Challenging preconceptions -

For example, Canadian David Card and his American colleague, the late Alan Krueger, who died in 2019, studied the relationship between the minimum wage and employment in the early 1990s.

They compared the labour markets on both sides of the border between the US states of New Jersey, where the minimum wage had been increased, and Pennsylvania, where it had not.

Their research showed that, in that context, the minimum wage increase had no downward effect on the number of employees.

That finding went against the prevailing theory at the time, which assumed that an increase in the minimum wage would destroy jobs as it would make it more expensive for companies to do business.

- More school, more income -

Card also studied the relationship between immigration and the labour market using another case study: the 1980 settlement of tens of thousands of Cubans in Miami, Florida, who had been allowed to leave the island by President Fidel Castro.

The economist's work showed that this wave of new arrivals did not have a negative impact on employment.

Also collaborating with the late Alan Krueger, American-Israeli Joshua Angrist looked at the link between education and income.

He compared the time spent in the education system by people born in the same year according to their month of birth.

Those born at the beginning of the year -- who therefore had the opportunity to leave school a little earlier -- had on average a shorter education than those born later in the year.

They also had lower wages.

This allowed Angrist to determine that higher levels of education generally led to higher wages.

Dutch-American Guido Imbens subsequently worked with Angrist to refine the interpretation of those results.

1. Wanless D. Securing good health for the whole population. HM Treasury; London: 2004. [Google Scholar]

2. Foresight . Tackling Obesities: Future Choices. Department for Innovation, Universities and Skills; London: 2007. [Google Scholar]

3. Department of Health . Healthy Weight, Healthy Lives: a research and surveillance plan for England. Department of Health; London: 2008. [Google Scholar]

4. Kelly MP, Morhan A, Bonnefoy J, et al. Final report to World Health Organisation Commission on the Social Determinants of Health. National Institute for Health and Clinical Excellence, UK/Universidad de Desarollo; Chile: 2007. The social determinants of health: developing an evidence base for political action. [Google Scholar]

5. Davey Smith G. Commentary: Behind the Broad Street pump: aetiology, epidemiology and prevention of cholera in mid 19th century Britain. Int J Epidemiol. 2002;31:920–32. [PubMed] [Google Scholar]

6. West SG. Alternatives to randomized experiments. Curr Dir Psych Sci. 2009;18:299–304. [Google Scholar]

7. Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. British Medical Journal. 2008;337:a1655. [PMC free article] [PubMed] [Google Scholar]

8. Academy of Medical Sciences . Identifying the environmental causes of disease: how should we decide what to believe and when to take action. Academy of Medical Sciences; London: 2007. [Google Scholar]

9. Meyer BD. Natural and quasi-experiments in economics. Journal of Business and Economic Studies. 1995;13:151–61. [Google Scholar]

10. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalised causal inference. Houghton Mifflin Company; Boston, Mass: 2002. [Google Scholar]

11. West SG, Duan N, Pequegnat W, et al. Alternatives to the randomised controlled trial. Am J Public Health. 2008;98:1359–66. [PMC free article] [PubMed] [Google Scholar]

12. Ogilvie D, Mitchell R, Mutrie N, et al. Evaluating health effects of transport interventions: methodologic case study. Am J Prev Med. 2006;31:118–26. [PubMed] [Google Scholar]

13. House of Commons Health Committee . Health Inequalities. Third Report of Session 2008-9. Volume 1: Report, together with formal minutes. The Stationery Office; London: 2009. [Google Scholar]

14. Bonell CP, Hargreaves J, Cousens S, et al. Alternatives to randomisation in the evaluation of public health interventions. J Epidemiol Community Health. 2010 doi:10.1136/jech.2008.082602. [PubMed] [Google Scholar]

15. Melhuish E, Belsky J, Leyland AH, et al. Effects of fully established Sure Start Local Programmes on 3 year old children and their families living in England: a quasi-experimental observational study. Lancet. 2008;372:1641–7. [PubMed] [Google Scholar]

16. Gunnell D, Fernando R, Hewagama M, et al. The impact of pesticide regulations on suicide in Sri Lanka. Int J Epidemiol. 2007;36:1235–42. [PMC free article] [PubMed] [Google Scholar]

17. Kreitman N. The coal gas story. United Kingdom suicide rates, 1960-71. British Journal of Preventive Medicine. 1976;30:86–93. [PMC free article] [PubMed] [Google Scholar]

18. Wheeler B, Gunnell D, Metcalfe C, et al. The population impact on incidence of suicide and non-fatal self harm of regulatory action against the use of selective serotonin reuptake inhibitors in under 18s in the United Kingdom: ecological study. BMJ. 2008;336:542–5. [PMC free article] [PubMed] [Google Scholar]

19. Taylor S, Kingdom D, Jenkins R. How are nations trying to prevent suicide? An analysis of national suicide prevention strategies. Acta Psychiatr Scand. 1997;95:457–63. [PubMed] [Google Scholar]

20. Deeks J, Dinnes J, D’Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7 [PubMed] [Google Scholar]

21. Rubin DR. For objective causal inference, design trumps analysis. Annals of Applied Statistics. 2008;2:808–40. [Google Scholar]

22. Mackay D, Pell J, Irfan M, et al. Meta-analysis of the influence of smoke-free legislation on acute coronary events. Heart. 2010 (in press) [PubMed] [Google Scholar]

23. Pell J, Haw SJ, Cobbe S, et al. Smokefree legislation and hospitalisations for acute coronary syndrome. JAMA. 2008;359:482–91. [PubMed] [Google Scholar]

24. Sims M, Maxwell R, Bauld L, et al. Short term impact of smoke-free legislation in England: retrospective analysis of hospital admissions for myocardial infarction. BMJ. 2010;340:c2161. [PMC free article] [PubMed] [Google Scholar]

25. Clancy L, Goodman P, Sinclair H, et al. Effect of air pollution control on death rates in Dublin, Ireland: an intervention study. Lancet. 2002;360:1210–14. [PubMed] [Google Scholar]

26. Hedley A, Wong C, Thach T, et al. Cardiorespiratory and all-cause mortality after restrictions on sulphur content of fuel in Hong Kong: an intervention study. Lancet. 2002;360:1646–52. [PubMed] [Google Scholar]

27. Campostrini S, Holtzman D, McQueen D, et al. Evaluating the effectiveness of health promotion policy: changes in the law on drinking and driving in California. Health Promot Int. 2006;21:130–5. [PubMed] [Google Scholar]

28. Ludwig J, Miller DL. Does Head Start improve children’s life chances? Evidence from a regression discontinuity design. Q J Econ. 2007:159–208. [Google Scholar]

29. Hernan M, Robins JA. Instruments for causal inference. An epidemiologist’s dream?. Epidemiology. 2006;17:360–72. [PubMed] [Google Scholar]

30. Lim SS, Dandona L, Hoisington JA, et al. India’s Janani Suraksha Yonana: a conditional cash transfer scheme to increase births in health facilities: an impact evaluation. Lancet. 2010;375:2009–23. [PubMed] [Google Scholar]

31. Dusheiko M, Gravelle H, Jacobs R, et al. Discussion papers in economics. University of York; York: 2003. The effect of budgets on doctor behaviour: evidence from a natural experiment. [Google Scholar]

32. von Elm E, Altman D, Egger M, et al. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335:335–8. [PMC free article] [PubMed] [Google Scholar]

33. Fox MP. Creating a demand for bias analysis in epidemiological research. J Epidemiol Community Health. 2009;63:91. [PubMed] [Google Scholar]

34. Turner R, Spiegelhalter D, Smith G, et al. Bias modelling in evidence synthesis. J R Stat Soc Ser A Stat Soc. 2009;172:21–47. [PMC free article] [PubMed] [Google Scholar]

35. D’Agostino R. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med. 1998;17:2265–81. [PubMed] [Google Scholar]

36. Angrist J, Pischke J-S. Mostly harmless econometrics: an empiricist’s companion. Princeton University Press; Princeton: 2009. [Google Scholar]

37. Brookhart MA, Rassen JA, Schneeweiss S. Instrumental variable methods in comparative safety and effectiveness research. Pharmacoepidemiol Drug Saf. 2010;19:537–54. [PMC free article] [PubMed] [Google Scholar]

38. McClellan M, McNeil BJ, Newhouse JP. Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables. JAMA. 1994;272:859–66. [PubMed] [Google Scholar]

39. Stukel TA, Fisher ES, Wennberg DE, et al. Analysis of observational studies in the presence of treatment selection bias. Effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007;297:278–85. [PMC free article] [PubMed] [Google Scholar]

40. Herttua K, Makela P, Martikainen P. Changes in alcohol-related mortality and its socioeconomic differences after a large reduction in alcohol prices: a natural experiment based on register data. Am J Epidemiol. 2008;168:1110–8. [PMC free article] [PubMed] [Google Scholar]

41. Herttua K, Makela P, Martikainen P. An evaluation of the impact of a large reduction in alcohol prices on alcohol-related and all-cause mortality: time series analysis of a population-based natural experiment. Int J Epidemiol. 2011;40:441–54. [PMC free article] [PubMed] [Google Scholar]

Page 2

Examples of natural experimental studies to evaluate population health interventions

Population	Intervention	Comparator	Outcome	Analysis method	Results	Reference
Sri Lanka -whole population	Legal restriction on pesticide imports	Suicide ratespre-ban;method specificsuicide rates;rates of non-fatal self- poisoning	Overall suicidemortality andmortalitymortality fromsuicide by self-poisoning with pesticide	Graphicalanalysis of trends	Import bans reduced method-specific and overall suicide mortality	Gunnell et al 2007[16]
UK - 12-17 year olds	Restriction onprescribing SSRIs to people aged <18	Suicidalbehaviour pre- restriction	Hospitalisationsfor self-harm;suicide mortality	Joinpoint regression	Restriction on prescribing of SSRIs wasnot associated with changes in suicidal behaviour	Wheeler et al 2008[18]
Finland -men andwomen aged >15	Reduction in alcohol taxes	Alcohol relatedmortality before the tax change	Alcohol-related mortality	Poissonregression toobtain relativerates; timeseries analysis	Alcohol-related mortality increased, especially among the unemployed	Herttua et al 2008,[40]Herttua et al 2011[41]
Hong Kong- whole population	Legislation to restrict sulphur content of fuel	Mortality pre-restriction,overall and by district	All-cause andcause specific mortality	Poissonregression ofchange inseasonallyadjusted death rates	Cardiovascular, respiratory and overallmortality fell post-restriction; decline greater in districts with larger falls in SO2 concentrations	Hedley et al 2002[26]
Dublin –whole population	Ban on coal sales	Mortality pre-ban, and in the rest of Ireland	Non-trauma,respiratory andcardiovascular mortality	Interrupted time-series	Non-trauma, respiratory and cardiovascular death rates fell post-ban	Clancy et al 2002[25]
Scotland –patientsadmitted to 9 hospitals	Legislative ban onsmoking in public places	Hospitalisationspre-ban and in England	Hospitalisationsfor acutecoronary syndrome	Comparison ofnumbers ofadmissions pre and post ban	Admissions for acute coronary syndromefell among both smokers and nonsmokers post-ban	Pell et al 2008[23]
England –patients aged >17	Legislative ban onsmoking in public places	Hospitalisations pre-ban	Emergencyadmissions formyocardial infarction	Interrupted time-series	Small but significant fall in emergency admissions in the first year post-ban	Sims et al 2010[24]
India –pregnant women	Cash incentives to usea health facility to give birth	Districts withlow rates oftake up; birthsto women notreceiving payments	Use of healthfacilities; infantand maternal mortality	Matched andunmatchedcomparisons ofrecipient andnon-recipientbirths;difference-indifferencesanalysis ofdistrict levelusage and mortality rates	Higher rates of take-up of the incentiveswere associated with higher proportions ofbirths within health care facilities; use ofhealth care facilities to give birth wasassociated with fewer perinatal andneonatal deaths. There was a non- significant reduction in maternal deaths	Lim et al 2010[30]
England –general practitioners	Abolition of GP fundholding	Non-fundholdingpractices; pre-abolition admission rates	Referral forelective andemergency admissions	Difference-indifferenceanalysis ofreferrals fromfundholders and non-fundholders	Fundholders had lower rates of electivereferral while fundholding was in operationand their rates of referral increased morethan those of non-fundholders followingabolition. There was no difference inemergency admissions pre or post abolition.	Dusheiko et al 2003[31]
USA – lowincomefamilieswithchildren aged 3-5	Headstart - help withparenting, nutrition,health and social services and schooling	US countieswith povertylevels above thecutoff used toallocate helpwith accessingHeadstart funding	Mortality fromcauses ofdeathamenable toHeadstart services	Regressiondiscontinuitydesign,comparingregressions ofmortality onpoverty forcounties aboveand below the cutoff	For Headstart-related causes, there was adiscontinuity in mortality rates at thecutoff, but no difference in deaths fromother causes or among children too old to qualify for Headstart services	Ludwig andMiller 2007[28]
USA –patientsadmitted tohospitalwith acutemyocardial infarction	Invasive cardiactreatment(catherisation followed by revascularisation)	Non-invasivecardiac treatment	Long term (i.e.7-year) mortality	Instrumentalvariableanalysis usingregionalcatherisationrates as the instrument	Cardiac catherisation was associated withlower mortality; instrumental variableanalyses produced smaller estimates ofeffect than analyses (multivariate riskadjustment, propensity score-basedmethods) that only adjust for observed prognostic factors	Stukel et al 2007[39]

Q&A Why

Why are natural experiments important?

Page 2

Related Posts

Advertising

LATEST NEWS

Advertising

Populer

Advertising

About

Legal

Help

Social