At the Wiley Society Executive Seminar in Washington DC in June, many speakers touched upon the future of research, with topics ranging from how to build collaborative and interdisciplinary communities to how to measure the impact of research with new digital tools. The room buzzed with energy, especially during the spirited discussion provoked by a talk given by Brian Nosek from the Center for Open Science (COS).
Nosek’s talk explored the many ways in which scholars and scientists are discouraged from sharing their research information and how publishers, societies, and researchers themselves can help to change that. Nosek began by illustrating the gap between research values – transparency, reproducibility, and objectivity – and the reality of research practice. For example, surveys have shown that most scientists do want to share their research data, but most simply don’t follow through. Likewise, in a “publish or perish” knowledge economy researchers are incentivized to publish as much and as quickly as they can, leading to problems of quality in study design, data analysis, and a bias for publishing “exciting” results.
And Nosek didn’t stop at the need for more open data. Taking reproducibility as an example, he shared the results of a study conducted by the Center for Open Science showing the power of individual researcher decisions during data analysis: 29 research teams were asked to answer the same research question, and given the same exact data sets, they came up with 29 different conclusions.
Publishers and others are responding to this “reproducibility crisis” in a variety of ways. Nosek highlighted the TOP guidelines (Transparency and Openness Promotion) as one simple way societies and publishers can encourage more open research practices. The guidelines currently have 62 organizations – including Wiley – and 714 individual journal signatories. Nosek then shared an example of how the analysis bias might also be diminished by a shift in the peer review process being piloted by the COS project Registered Reports. Registered Reports signify real change that have the potential to transform the role publishing plays in the research process. Instead of peer review occurring after the study has been designed and conducted, Registered Reports move up the review process, so it takes place before the experiment is conducted or the paper is written. With this process, the publication workflow more closely matches the process for grant application.
The power of the Registered Report model – which Nosek is piloting as a submission option with over 20 journals – is that papers are accepted in principle, regardless of how significant the experimental outcome. This incentivizes strong study design, and discourages analysis choices which might make the results more “exciting.” In a research community where the importance of impact is at an all-time high, this focus on the research question rather than the outcome represents a fascinating shift in thinking.
When it was time for questions, multiple hands shot into the air. One audience member asked if peer review under the Registered Reports model risks biasing study design through strong reviewer recommendations? Nosek explained that it shouldn’t, as design still ultimately remains the decision of the research team conducting the study, though he acknowledged that the Registered Reports workflow is too new to know what its wider impact on the research process would be.
What was clear, however, was the interest and commitment of the 64 society leaders in the room to improve the publication process to encourage more data sharing and support reproducibility. Sara Rouhi, of Altmetric, declared that her “mind was blown” and, honestly, so was mine. Nosek presented powerful arguments for how open science can improve the research process and bring our community’s values in line with practice.
Sam Green is a member of the Society Marketing team at Wiley.
Image credit: avemario/Shutterstock