The APA and Open Data: one step forward, two steps back?

by Denny Borsboom

I was pleasantly surprised when, last year, I was approached with the request to become Consulting Editor for a new APA journal called Archives of Scientific Psychology. The journal, as advertised on its website upon launch, had a distinct Open Science signature. As its motto said, it was an “Open Methodology, Open Data, Open Access journal”. That’s a lot of openness indeed.

When the journal started, the website not only boosted the Open Access feature of the journal, but went on to say that "[t]he authors have made available for use by others the data that underlie the analyses presented in the paper". This was an incredibly daunting move by APA - or so it seemed. Of course, I happily accepted the position.

After a few months, the first papers in Archives were published. Open Data enthusiast Jelte Wicherts of Tilburg University immediately tried to retrieve data for reanalysis. Then it turned out that the APA holds a quite ideosyncratic definition of the word “open”: upon his request, Wicherts was referred to a website that presented a daunting list of requirements for data-requests to fulfill. That was quite a bit more intimidating than the positive tone struck in the editorial that accompanied the launch of the journal.

This didn’t seem open to me at all. So: I approached the editors and said that I could not subscribe to this procedure, given the fact that the journal is supposed to have open data. The editors then informed me that their choice to implement these procedures was an entirely conscious one, and that they stood by it. Their point of view is articulated in their data sharing guidelines. For instance, "next-users of data must formally agree to offer co-authorship to the generator(s) of the data on any subsequent publications" since "[i]t is the opinion of the Archives editors that designing and conducting the original data collection is a scientific contribution that cannot be exhausted after one use of the data; it resides in the data permanently."

Well, that's not my opinion at all. In fact it's quite directly opposed to virtually everything I think is important about openness in scientific research. So I chose to resign my position.

In October 2013, I learned that Wicherts had taken the initiative of exposing the Archives’ policy in an open letter to the editorial board, in which he says:

“[…] I recently learned that data from empirical articles published in the Archives are not even close to being “open”.

In fact, a request for data published in the Archives involves not only a full-blown review committee but also the filling in and signing of an extensive form: http://www.apa.org/pubs/journals/features/arc-data-access-request-form.pdf

This 15-page form asks for the sending of professional resumes, descriptions of the policies concerning academic integrity at one’s institution, explicit research plans including hypotheses and societal relevance, specification of the types of analyses, full ethics approval of the reanalysis by the IRB, descriptions of the background of the research environment, an indication of the primary source of revenue of one’s institution, dissemination plans of the work to be done with the data, a justification for the data request, manners of storage, types of computers and storage media being used, ways of transmitting data between research team members, whether data will be encrypted, and signatures of institutional heads.

The requester of the data also has to sign that (s)he provides an “Offer [of] co-authorship to the data generators on any subsequent publications” and the (s)he will offer to the review committee an “annual data use report that outlines what has been done, that the investigator remains in compliance with the original research proposal, and provide references of any resulting publications.”

In case of non-compliance of any of these stipulations, the requester can face up to a $10,000 fine as well a future prohibition of data access from work published in the Archives.”

A fine? Seriously? Kafkaesque!

Wicherts also notes that “the guidelines with respect to data sharing in the Archives considerably exceed APA’s Ethical Standard 8.14”. Ethical Standard 8.14 is a default that applies to all APA journals, and says:

“After research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release.”

Since this guideline says nothing about fines and co-authorship requirements, we indeed have to conclude that it’s harder to get data from APA’s open science journal, than it is to get data from its regular journals. Picture that!

In response to my resignation and Wicherts' letter, the editors have taken an interesting course of action. Rather than change their policy such that their deeds match their name, they have changed their name to match their deeds. The journal is now no longer an "Open Methodology, Open Data, Open Access Journal" but an "Open Methodology, Collaborative Data Sharing, Open Access Journal".

The APA and open data. One step forward, two steps back.