At the end of last year we posed some questions to David Mellor from the Center for Open Science about Registered Reports, Research Preregistration and Why Both Are Important.
Matthias Mittner and his co-authors, recently submitted a stage 1 Registered Report to European Journal of Neuroscience. You can read more about Registered Reports at the European Journal of Neuroscience here.
We asked Matthias to share his RR experience as an author with us.
A. I have been following the Open Science movement for several years now, ever since the first failed replications appeared in our field. The OSC article on the replicability of psychological science was certainly an eye-opener! But maybe the clearest moment for me, that clearly shows that we also face these problems in cognitive neuroscience was when some of my colleagues published a paper where they tried to replicate a set of brain-behavior correlations (Boekel et al., 2015, Cortex). At that time, there was no established RR format and the researchers therefore published their pre-registration document on a blog. This idea of publicly sharing your analysis plan really impressed me and I am glad that the RR format is being picked up by many journals today!
Q. What made you want to go through stage 1 peer review in addition to submission of the finished article?
A. When we decided that we wanted to conduct a pre-registered replication of a published study, I anticipated push-back from other researchers in case the replication should turn out to be unsuccessful. This was the fate of many other failed replications and also my colleagues publishing their brain-behavior correlations received some serious push-back in the form of published commentaries. Another concern was whether a failed replication would be publishable. While the situation is certainly improving in recent years, it is not hard to look for stories from scientists trying to publish negative results and being rejected in many places. The concept of in-principle-acceptance counteracts this problem and also has the advantage that we could formerly involve a number of scientists in the form of peer-reviewers. Our hope is that our study will provide a definite "last word" on the matter.
Q. There’s been a lot of discussion about how Registered Reports benefits science. But what’s in it for you personally as a researcher and author?
A. I think the most beneficial aspect for myself has been the projectability of this project. Once the in-principle acceptance was obtained, the execution of the study is specified in all details (including pre-written experimenter instructions, software and analysis scripts) and therefore requires only minimal supervision. Having a project that simply goes by itself is certainly a rewarding experience! It is also great to see how smoothly the coordination of the research with the three internationally distributed labs involved in our study has been.
Q. How did you find the stage 1 peer review (of your hypotheses and experimental procedures) compared to peer review of a finished study?
A. The experience was nothing less than amazing! The reviewers were incredibly helpful and contributed pages of comments and suggestions. One researcher even went so far as to run custom simulations, augmenting our original Bayesian power-analysis! Our protocol has really been significantly improved through that process, even though we had to bite the bullet of dropping an interesting condition in order to increase statistical power. I believe that the fact that the study has not yet been conducted makes the reviewers feel that they can really contribute to a novel piece of research - instead of simply having to evaluate it when a final paper is submitted for publication.
Q. Has the stage 1 peer review helped to improve the quality of your study?
A. It certainly did. All reviewers had interesting and valid additions and objections to our protocol and invested a lot of time and effort to improve our study.
Q. Hypothetically, how would you feel about publishing your stage 1 submission? Or pre-registering it in a registry where you can set the embargo?
A. In fact, our stage 1 submission has been public as an OSF repository since the day of in-principle acceptance. We also publish our incoming raw data and everything else in that repository.
Q. Your Stage 1 submission has been accepted in principle for publication following adherence to the study as you’ve outlined. Will you come back and publish the finished paper?
A. Definitely! Coordination and data-acquisition took a bit longer than anticipated but we are soon to write up and submit the final paper.
Wiley has a Registered Reports toolkit to help launch Registered Reports quickly. Please speak with your publisher so together we can make this option available for researchers in your communities.