Questions and Answers about the Förster case

by Denny Borsboom, Han van der Maas, Eric-Jan Wagenmakers, Department of Psychological Methods, UvA

By now, everyone is probably familiar with the recent investigation of the work of Dr. Förster, in which the Landelijk Orgaan Wetenschappelijke Integriteit (LOWI) concluded that data reported in a paper by Dr. Förster had been manipulated. In his reaction to the newspaper article NRC Dr. Förster suggested that our department would be involved in a witch-hunt. This is incorrect.

However, we have noticed that there are many questions about both the nature of the case and the procedure followed. We have compiled the following list of questions and answers to explain what happened. If any other questions arise, feel free to email them to us so we can add them to this document.

Q: What was the basis of the allegations against Dr. Förster?
A: In every single one of 40 experiments, reported across three papers, the means of two experimental conditions (“local focus” and “global focus”) showed almost exactly opposite behavior with respect to the control condition. So whenever the local focus condition led to a one-point increase of the mean level of the dependent variable compared to the control condition, the global condition led almost exactly to a one-point decrease. Thus, the samples exhibit an unrealistic level of linearity.

Q: Couldn’t the effects actually be linear in reality?
A: Yes, that is unlikely but possible. However, in addition to the perfect linearity of the effects themselves, there is far too little variance in the means of the conditions, given the variance that is present within the conditions. In other words: the means across the conditions follow the linear pattern (much) too perfectly. To show this, the whistleblower’s complaint computed the probability of finding this level of linearity (or even more perfect linearity) in the samples researched, under the assumption that, in reality, the effect is linear in the population. That probability equals 1/508,000,000,000,000,000,000.

Q: But that’s just a probability. That something is improbable doesn’t mean it’s impossible, right?
A: That is correct. However, the probability of this happening is so small that it justified an investigation: it is roughly equivalent to winning the Dutch lottery on five consecutive drawings. That is also possible in theory, but it would certainly justify an investigation. Also, note that the probability is computed under the most favorable scenario for Dr. Förster: namely, the computation assumes that the linear version of Dr. Förster’s theory is perfectly true, and shows that under this assumption the effects are very unlikely. In reality, that linear version of the theory is highly unlikely to be true; at best, we could expect some ordinal theory to hold. That would lead the probability in question to become at least an order of magnitude lower. It is not an overstatement to say that, given realistic assumptions, Dr. Förster’s results are equivalent to winning the lottery on dozens of successive drawings. The datacolada website provides accessible illustrations of how unrealistic these effects really are.

Q: What about the statistical validity of the whistleblower’s method?
A: By now, the whistleblower's method has been thoroughly checked and stands undisputed. In his May 11 response, Dr. Förster lists some questions about the method but provides no compelling rebuttal. No one has proposed a plausible QRP or set of QRPs that could explain the results. Note that exactly the same patterns have been detected in 40 experiments in three papers, two of which single-authored by Dr. Förster. Different statistical methods may be applied, but, in our opinion, all of them will inevitably lead to conclusions that are qualitatively identical to those drawn by the LOWI.

Q: So what happened in the investigation?
A: The University of Amsterdam investigated the case for roughly a year. The relevant committee concluded that the data could not possibly have arisen from actual sampling, as stated in Förster’s papers, and thus endorsed the main conclusion of the whistleblower’s report.

Q: But I thought the University of Amsterdam concluded there was nothing wrong with the papers?
A: That is not correct. The University of Amsterdam committee endorsed the conclusion of the whistleblower that the data could not have arisen from standard sampling. However, although the committee considered the data to be practically impossible, they also concluded that data manipulation could not be considered proven, because the statistical expert they consulted said that the data might have arisen as a result of the use of Questionable Research Practices (QRPs). Thus, the committee stated they could not conclude that data manipulation had taken place, because they could not rule out the alternative explanation that Dr. Förster had engaged in QRPs. However, Dr. Förster denied having used these practices. Moreover, the whistleblower argued that no known set of QRPs, however excessively used, could ever lead to the patterns in Dr. Förster’s data, and took his complaint to the Landelijk Orgaan Wetenschappelijke Integriteit (LOWI). The LOWI agreed with the whistleblower and considered data manipulation proven.

Q: So this verdict was reached entirely on the basis of the analysis of the whistleblower?
A: No. During the investigation, the statistical expert consulted by LOWI had access to data files provided by Dr. Förster. In analyzing these, the expert discovered that the linearity of the manipulation did not show up within males or females separately. However, whenever the males deviated from linearity in the positive direction, the females showed a precisely equivalent deviation from linearity in the negative direction, so that the resulting overall means were perfectly linear again. It is hard to imagine how any actual experiment could have produced such an anomalous result.

Q: OK, so that’s a lot of statistical work. But was the verdict then reached entirely on the basis of a statistical calculation?
A: No. Dr. Förster proved unable to give basic details of where the data came from, who gathered them, or when they were gathered. All verifiable traces of the data gathering process had evaporated as a result of dumping of raw data to win office space, and lapses of Dr. Förster’s memory. The combination of the practical impossibility of the results, and Dr. Förster’s inability to provide any details about the origin of the data, led to the LOWI’s verdict that the data were manipulated and that scientific integrity was violated..

Q: Does that mean Förster has been found guilty?
A: The data have been manipulated, but the committee does not know by whom. However, the LOWI concludes that Dr. Förster, as lead researcher and first author, is nevertheless responsible. LOWI excluded the possibility that the many research assistants, who were involved in the data collection in these experiments and were typically unaware of the research hypotheses, could have created such exceptional statistical relations.

Q: But didn’t the LOWI investigate only one paper while the whistleblower’s report mentioned three studies?
A: Yes, LOWI investigated only the Förster and Denzler (2012) paper. Two other papers featured in the original complaint, single-authored by Förster, were not investigated.

Q: Does that mean that the data of the other two papers were judged to be trustworthy?
A: No. They were simply not investigated. We do not know why LOWI chose to investigate only one paper, but there does not appear to be a good reason why the verdict for the other two papers would be different.

Q: Do you know who the whistleblower is?
A: Yes. But because the whistleblower has asked to remain anonymous (which is the legal privilege of any whistleblower), we will not give information that would lead to his or her identification. Also, we cannot react to the accusation voiced by Dr. Förster in his letter to Retraction Watch, in which he stated that the case originated in our department. Note that this does not mean that Dr. Förster’s statement is correct.

Q: Did you or the whistleblower leak the information to the press that led to the preliminary exposure of the case in NRC Handelsblad?
A: No.

Q: In his letter to Retraction Watch, Dr. Förster says he is the victim of a witch-hunt. What do you think about that?
A: A witch-hunt could be defined as an organized systematic attempt to incriminate researchers on the basis of vague hints and weak circumstantial evidence for misconduct. In our view, the patterns in Dr. Förster’s data clearly and directly pointed to irregularities in the process of data gathering and data analysis. Thus, these patterns were not vague or circumstantial, and were sufficiently implausible to justify an investigation. As far as we know, the investigation stood on its own and was never part of a larger investigative process. In fact, we do not know of any other investigations into misconduct that ran while the case of Dr. Förster was being investigated. Also, we do not know of any investigations into misconduct that are currently taking place, and we do not know of any systematically attempts to search the literature for evidence of fraud. Hence, as far as we know, the investigation of Dr. Förster’s work was an isolated investigation that closely followed the official procedures as laid down in the integrity policies of the University of Amsterdam.

Q: What needs to happen now?
A: We recommend further investigation of the work of Dr. Förster to determine whether other papers show similar improbable data patterns that are too good to be true.