When scientists come up with an unexpected result, they need to repeat their tests to confirm their results. But if scientists can’t replicate each other’s experiments, they can’t verify results to make sure they’re true. This replication crisis could hurt the credibility of their work and put humans at risk.
Psychology researchers at IU say the solution requires understanding science better, rather than discrediting research altogether.
“Is there a vast proportion of claimed effects that can’t be reproduced with regularity?” John Kruschke, professor of psychology, said.
In August of 2015, psychology researchers could only replicate 39 of 100 prominent psychology studies, according to a report by the American Psychological Association. Scientists try to replicate each other’s results to confirm findings and understand their research better.
Though the replication problem is part of all areas of science, its effect in psychology could mean trouble for our own lives.
Replicating research requires confirming previous results, which is different from exploring new and interesting phenomena, Kruschke said. Researchers have to make sure they get a consistent relationship with what they study, said Mary Jean Amon, a postdoctoral researcher in developmental cognitive neuroscience.
If scientists can’t replicate results, they can’t completely believe their credibility.
“And for the studies with huge benefits, there’s a huge impact,” Kruschke said.
Policy, medicine and other fields depend on the credibility of this research. Kruschke researches statistical methods scientists use in their work, including how psychologists should better understand statistics to combat the replication crisis. All scientists need to have a better understanding of these statistical methods to understand their results, Kruschke said.
These statistical techniques would help psychology researchers confirm findings and replicate the same results of other scientists.
Since psychology results can vary significantly, it becomes difficult to get the same results in experiments and confirm findings.
“We are inherently studying flexible and adaptive systems,” Amon said.
Apart from statistics, the replication crisis is partly due to how complex psychology research is.
“The research process goes from narrowing a research question to experimental design to finding a method that controls for relevant contextual factors to prepping data appropriately to reporting results correctly,” Amon said.
The results of psychology research can change significantly with even minor factors, Amon said.
Even good science will show different effects on repeated attempts.
“Just because a study gets published doesn’t mean it’s true always and everywhere,” said Eliot Smith, distinguished professor of psychology.
To find out which of two championship baseball teams is better, they play multiple games against one another, Kruschke said.
Similarly, it takes multiple research projects on ongoing, thorough processes to study human psychology.
“The public should understand the difference between ‘one study found this’ and ‘many studies in many countries found this,’” Smith said.
Apart from issues scientists face, media and journalists are partly to blame as well, Smith said.
“For the headline ‘Red wine helps heart health,’ people need to understand the phenomena is highly variable,” Kruschke said.
The effects of wine on health are complicated, and they can’t be reduced to a simple newspaper headline, Kruschke said.
Even university press releases can exaggerate findings of studies, Smith said. Scientists do have incentives to publish significant results. Science journals, departments and labs often encourage researchers to publish often and as quickly as possible, Amon said. This means results can be skewed to show effects when there are none.
Despite this, the issue of the replication crisis is usually not a result of research fraud, Smith said.
“The issue doesn’t mean scientists are disingenuous or knowingly disguising the data necessarily,” Kruschke said.
Some false findings come from serious scientists earnestly trying to discover new things, Kruschke said. Even with the concerns, psychology research still has a lot of credibility and value.
“The replication crisis is serious problem, but most researchers go to great lengths to ensure they have valid results,” Amon said.
Scientists have taken up registered replication reports, Kruschke said. In this reports, several labs in various locations agree on a specific procedure for all of them to use in their work. With a large sample, random variation should disappear, Kruschke said.
The reports also ensure scientists can check and confirm each other’s results in their replication process.
These reports can help but they’re not a cure-all solution, Smith said.
Scientists should poke with data and look at things in new ways. Doing experiments strictly the same way might discourage that, Smith said. But general transparency of exact procedures when publishing makes it easier for other people to prove your results, Smith said.
“We’ll find more findings that fail to replicate but over time we’ll find more ways to do better science,” Amon said.
