Reproducibility of Psychology Studies Debate Continues

Researchers discussing something around a deskA new analysis published in Science suggests the field of psychology might not have the “reproducibility problem” discovered in a 2015 study.

The original study re-analyzed and repeated 100 studies published in three psychology journals. The 270 researchers involved in the project were able to replicate the results of just 39 studies. For an additional 24 studies, the results were “moderately similar” to the original research. Reproducibility—the ability of researchers who may have different beliefs or biases to replicate the original findings of a study—ensures the reliability of scientific research. Thus, the failure to replicate these studies was billed as a “crisis” in psychology.

Does Psychology Have a Reproducibility Problem?

The follow-up study, led by Harvard psychologist Daniel Gilbert, looked at the previous study’s attempt to replicate psychological research. The team found a number of problems with the initial study, including replication techniques that did not properly repeat the original research.

For example, to repeat a study that originally looked at American attitudes toward black Americans, researchers substituted native Italians. To replicate a study that required young children to locate items on a large computer screen, researchers instead gave older children the task of locating items on a smaller computer screen.

These apparently small differences in methodology can produce significant differences in research outcomes. In some cases, the new study argues, the original replication study relied on too few subjects for replication to either succeed or fail.

When Gilbert and his team corrected for the issues in the original study, different researchers were able to produce the same results as the original researchers in 85 studies, compared to 39 studies with the original replication study. According to Gilbert and colleagues, at least 34 of the original studies should have failed by chance alone, making the replication of 85 studies even more compelling.

The Debate Continues

As the ongoing debate about replication makes clear, science is a process of testing, re-testing, analyzing, and debating data. The new study does not necessarily overturn or disprove the original reproducibility study, just as the original reproducibility study did not conclusively prove that psychological research is deeply flawed.

The original authors of the 2015 reproducibility study have already published a rejection of the 2016 analysis. They point to a number of concerns with the new research, saying the team erroneously focused on the confidence interval (CI), one of five measures of reproducibility used in the original research.

A related study published in February in PLOS One was able to repeat the results of 75% of the studies in the original replication project. However, the evidence for the conclusions contained in the original study was often weak. This suggests more research is necessary to explore whether psychology has a reproducibility problem.

References:

  1. Bower, B. (2016, March 3). Psychology’s replication crisis sparks new debate. Retrieved from https://www.sciencenews.org/article/psychologys-replication-crisis-sparks-new-debate
  2. Estimating the reproducibility of psychological science. (2015). Science,349(6251). doi:10.1126/science.aac4716
  3. Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the Reproducibility Project: Psychology. PLOS One, 11(2). doi:10.1371/journal.pone.0149794
  4. Researchers overturn landmark study on the replicability of psychological science. (2016, March 3). Retrieved from https://www.sciencedaily.com/releases/2016/03/160303145733.htm

© Copyright 2016 GoodTherapy.org. All rights reserved.

The preceding article was solely written by the author named above. Any views and opinions expressed are not necessarily shared by GoodTherapy.org. Questions or concerns about the preceding article can be directed to the author or posted as a comment below.

  • 3 comments
  • Leave a Comment
  • Grace

    Grace

    March 9th, 2016 at 11:36 AM

    Oh golly sounds like this is a debate that no one is ever really going to win. We are going to tend to look at the evidence that supports what we choose to believe and that is the point that we will continue to try to stress to others. Most of the time it is all about what we believe to be true and settling in with that.

  • ernestine

    ernestine

    March 10th, 2016 at 3:24 PM

    But since its very inception psychology has been a field that has been much discussed and rarely agreed upon. Just like other scientific studies and really all fields of study there will be growing pains, and there will be times when you find success and then there are times that you have to try and try again. Any of the sciences are like this, I just think that there is probably a tendency among some communities to be loud about what part of this they want others to hear and keep quiet when there actually is reproduction of outcomes and success.

  • Leila

    Leila

    March 11th, 2016 at 12:01 PM

    I guess that your opinion is going to be heavily influenced by the way you feel about science in general.
    But my suspicion is that most of us outside of this field really are pretty indifferent about it.

Leave a Comment

By commenting you acknowledge acceptance of GoodTherapy.org's Terms and Conditions of Use.

* Indicates required field.

GoodTherapy uses cookies to personalize content and ads to provide better services for our users and to analyze our traffic. By continuing to use this site you consent to our cookies.