Avoiding a Witch Hunt: What is the Next Phase of our Scientific Inquisition?

by Jon Grahe

Earlier this week, I learned about another case of fraud in psychological science (Retraction Watch, 4.29.2014). The conclusions from the evidence in the case against him after an extended investigation are hard to ignore. The probability that the findings could have occurred by chance are so minute that it is hard to believe that they didn’t result from falsified data. In an email to the scientific community (Retraction Watch, 4.30.2014), the target of this investigation strongly asserted that he never faked any data, while assuring us that the coauthor target never worked on the data, it was all his. Some comments from the Retraction Watch post use the term “witch hunt.” It was the first term I used in response as well, suggesting caution before judgment. A colleague pointed out that the difference was that there were no witches, and that there are clearly dishonest scientists. I have no choice but to agree; I think a better analogy is that of the Inquisition. We are entering the era of the Scientific Inquisition. A body of experts (LOWI in this case) will use a battery of sophisticated tools to examine the likelihood that the findings’ irregularities occurred by chance. In this case it is hard to believe his denial, but thankfully I am not a judge in the Scientific Inquisition.

It was suggested that I should consider other similar inquiries, such as doping scandals; but I prefer the Inquisition. I invite the reader to consider other metaphors to guide us through this phase of our scientific correction. However, I would like to avoid scientific witch hunts where people are prosecuted and eventually terminated (professionally) even though they are innocent. Therefore, I think it important to consider the Inquisition as a metaphor. For a brief description, visit the Wikipedia entry. For our analogy, consider that there were actually a series of inquisitions. They started as trials to deter increasing secularism and heresy against the Catholic church in the middle age in the late 1100s. Given that my life is guided by science as much as a believer’s life might be guided by Catholicism or some other religion, I can think of no heresy more offensive than falsifying data and reports. While we are not nearly as centralized as the Catholic church, the only way to find academic heretics is through an inquiry such as the one used in this case. A long detailed report using many statistical procedures to determine likelihood of fake data, a series of meetings and hearings ending with committee recommendations. The major difference I see is the absence of torture, although maybe that was included and not reported.

If Phase I of our Scientific Correction was and continues to be recognizing the problem of publication bias and fraud, Phase II is the beginning of the correction. I will not waste space discussing the structural problems that lead to academic dishonesty as there are many voices that have already discussed them. Instead, I would like to consider Phase II: Beginning of Correction. Here again, I will ignore some very positive next steps such as changes in publication expectations, crowdsourcing science projects, and computer systems that make openness easier as we reach toward the process of bringing forth Scientific Utopia as envisions by Nosek and colleagues (Nosek & Bar Anan, 2012; Nosek, Spies, and Moytl, 2012). Changes in publications are announced by the journals and societies embracing them, and there are some early indications that crowd-sourcing science projects are successfully replicating many, if not all, published effects in psychology (Yong, 2013). Naturally, there will be disappointments there as well, but that is topic for a future blog. Instead, consider the Scientific Inquisition portion of Phase II.

Consider the fate of transgressors against science. Are you one? Did you fake data to get published? Did you change some data points to get published, but not many? Did you use a priori tests when you were in fact HARKing (Kerr, 1998)? Did you actively p-hack? Did you fail to retract? What is the minimal level of violation that should have consequences during the Scientific Inquisition? I would imagine that a number of data fakers are huddling in their houses hoping to escape reality. Detecting scientific fraud is the hot new field and our Scientific Correction requires justice. What is the necessary criteria for preponderance of evidence to be found guilty by the scientific community or a host institution?

One we determine guilt, how should we sanction dishonest scientists? Job loss, retraction of degrees, and social ostracism from the the field all seem reasonable consequences for egregious violations. Are there more severe sanctions we could consider? Public shaming at a professional conference might be more impactful to ward off other offenses. Maybe an offender should be exiled not only from academia, but also from his or her neighborhood so as to avoid spreading horrible ethics with the good people who live nearby.

The title of this blog invokes witch hunts because the Inquisition morphed across a series of trials aiming to stamp out different types of heresy. Regardless of the reader’s personal religious convictions, one might forgive church leaders at a time before science was firmly rooted for believing that their enemy, the devil, had possessed their targets of accusation: the heretic. Wikipedia describes a series of inquisitions, responding early to secular movements and progressing eventually to the prosecution sorcery and witchcraft. Wikipedia also quotes a source that reminds us that the purpose of the inquisition was to create fear in the public so as to avoid further violations of church law.

We have not remotely approached this level of dysfunction in our Scientific Inquisition, but we should be cautious. This change from a real threat to the church (secularism) to imagery and fantastical threats such as sorcery and demonic possession occurred later after it became clear that the tools and outcomes of the inquisition became useful for political gain and personal agendas. In the end, the power was overwhelming and the victims didn’t even need to appear guilty anymore. The inquisitors had a direct line to God, and accusations were sufficient for conviction.

Scientists have for decades engaged in slippery statistical practices in order to succeed in a field of biased publication expectations and promotion rewards. How many stepped past the line and engaged in outright fraud? Some estimates are quite high. How many tenured and established researchers are going to be found guilty of academic fraud? What judge or jury will determine their fate? Will they be objective and unbiased as they sanction transgressors? Will the Scientific Inquisition be an independent organization or organizations developed to root out academic fraud?

Ideally, they should also have a direct line to truth. Well not yet; the Inquisition didn’t turn evil immediately. Sure the methods of inquisition were brutal, but not particularly for the time, and they got worse as the spectacles progressed. It took a little over 70 years before the torture as part of the inquiry was no longer considered a sin or a crime, as long as it was part of the inquisition. The metaphor is less clear at this point, for it is not clear what is the appropriate modern, scientific version of torture. Instead, I hope to caution us to be wary of the Scientific Inquisition; the determiners of truth. They will still be human, and the tools they will use can still be manipulated.

This case suggests fraud, not false accusation. But there will be more accusations and the preponderance of evidence will not always be as clear. In the US judicial system the burden of proof in criminal cases is on the prosecutor and the preponderance of evidence for guilt is “beyond a reasonable doubt”. However, the burden is much less rigorous in civil cases and much closer to “at least 51 % responsible”. In other judicial systems, the burden of proof is on the accused. What will be the burden of proof in the next phase of the Scientific Inquisition? Who will shoulder the burden of proof? We already know that I will not sit on this board; I don’t have the will to punish. But I would like to call for leniency; forgiveness; empathy.

The Catholic church went to war against heretics; academics have new tools to go to war against academic heretics. However, instead of progressing through stages and ending with witch hunts, follow the ideal of pardons. At the end of bloody revolutions, it is ideal when the victor pardons the loser. Let us welcome our academic heretics to bring in and lay down their arms. If they agree to point to papers where they committed fraud before they are discovered, offer a pardon. With new publishing standards that value scientific openness, there will be fewer transgressions. Instead of crafting torture tools, focus on training new scientists how to use public dispensaries to share their data and materials. Transgressors know they will live in shame, and might be stripped of their academic accolades. They lost their way, but we also know that social forces impact behavior. Now that the environment is prohibitive of future academic heresy, the same transgressors might become zealots of good science. It would be the best way to gain any respect from their angry peers.