Statistics logo
Department of Statistics & Actuarial Science
Simon Fraser University
Author: Carl James Schwarz   P.Stat.
Phone: (778) 782-3376
Office: K 10559

Professional Statistician (P.Stat.)
        More information on Accreditation of Statisticians in Canada


Reanalysis of data from published papers - cautionary tales

As part of my courses in Statistics for Ecologists and Resource Managers, I have students re-analyze data from published papers to see if they can recreate the results in the paper. The purpose of these exercises is to reinforce the principles of matching the analysis to the study design, and to illustrat the principles of Reproducible Research

How did I choose these paper? I'm an occasional reader of The Economist which now and then presents a short summary of an interesting result. I've been collecting a number of these, and so far, all of the paper cited by the Economist that I've examined made substantial errors in the statistical analysis of their results as noted below. Based on the observed data, there are two consistent hypotheses. (1) The conrrespondents who find these articles for the The Economist has an uncany knack for selecting papers with statistical flaws. If this hypothesis is true, then being cited by the Economist is a "kiss of death". (2) The incidence of sloppy statistical analysis in published research is so high, that any selection of papers invariably leads to papers with problems in the analysis. You be the judge.

Many students find that the results in published papers are not reproducible! In many cases, the errors are basic errors in data analysis (e.g. using the wrong statistics procedure) or errors in interpretation (e.g. p-values misinterpreted). In some cases, the errors are serious enough that the results in the published papers are questionable.

This web page is a collection of the results of such investigations. Your mileage may vary.

Examples of papers where data re-analysis is at odds with the results in the paper.

PaperCommentary and code
Gueguen, N. (2013).
Effects of a tattoo on men's behaviour and attitudes towards women:
An experimental field study.
Archives of Sexual Behavior, 42, 1517-1524.
DOI: 10.1007/s10508-013-0104-2.
Commentary on the paper
SAS code,       SAS output
Tsay, C.-J. (2013).
Sight over sound in the judgment of music performance.
Proceedings of the National Academy of Science, 10, 14580-14585.
DOI: 10.1073/pnas.1221454110.
Commentary on the paper
SAS code,       SAS output
Gino, F. and Mogilner, C. (2013).
Time, Money, and Morality.
Psychological Science 0956797613506438.
DOI: 10.1177/0956797613506438
Commentary on the paper
R code,       R output
Suarez-Rodriguez, M., Lopez-Rull, I. and Garcia, C.M. (2012).
Incorporation of cigarette butts into nests reduces nest
ectoparasite load in urban birds: new ingredients for an old
recipe?
Biological Letters, 9, 20120931.
http://dx.doi.org/10.1098/rsbl.2012.0931
Re-analysis of the paper
R code,   Result of R code
Egri, A., Blaho, M., Kriska, G., Farkas, R., Gyurkovszky, M., Akesson, S. and Horvath, G. (2012).
Polarotactic tabanids find striped patterns with brightness and/or polarization
modulation least attractive: an advantage of zebra stripes.
Journal of Experimental Biology 215, 736-745.
DOI: 10.1242/jeb.065540
Commentary on the paper
R code,       R output
Response by the authors

[ SFU Statistics and Actuarial Science Home | SFU Home]
Email comments or suggestions to Carl Schwarz (cschwarz@stat.sfu.ca)
© 2013 Carl James Schwarz Last updated 2013-12-.