What do free will, the effect of physical distance on emotional closeness and mate preference have to do with each other?
Each was the subject of a psychological study that researchers recently found to be either unreproducible or weaker in significance than originally believed.
This effort by psychologists to check the reproducibility of many psychological studies, all published in psychology’s largest journals, brought to light the fact that the effect of many of these studies was overstated by the studies themselves.
The Reproducibility Project, as it is called, was not an attack by a group of crusading psychologists bent on discrediting the statistical significance of over half of the studies they attempted to reproduce, but a group of volunteers who worked closely with the original authors of each study in order to most accurately replicate the design and methodology of each study.
The studies being reproduced also included larger sample sizes than the original study in order to have more statistical weight.
While this project is not the be-all and end-all of reproducibility in psychology, as evidenced by separate findings that putting eyeballs on political pamphlets in order to influence people to vote is likely less effective than previously believed, its findings are on a larger scale than previously realized.
As I poured through news articles on these findings, I found a number of possible explanations for the apparent failures of the scientific community at large to create reproducible research: lackluster training in experimental design, an emphasis on making a statement rather than presenting the details that led to such results and publications that do not report basic elements of their design and methodology.
While these explanations certainly contribute to the issue, I hold with those who find that the incentive system for publication is deeply flawed.
The agencies that fund research often encourage the scientific community to overvalue research published in more prominent journals, providing incentives such as promotion and tenure for their researchers to publish therein.
These incentives often force scientists to choose between the work they love and the work that can get them published.
Furthermore, programs such as the Reproducibility Project are few and far between — it is hard to find a journal that wants to publish works that point out the flaws in research.
I certainly appreciate the fact that not all work is equally significant, and that certain topics require further study than others.
But I can’t help but feel that if the systems that we have put in to place do not value the significance of reproducibility, then the systems need to change.
mccinkos@indiana.edu
