At the end of last year, we interviewed David Mellor about how Registered Reports support the “open research” agenda.
Following on from that we asked Jörg Schulz, Editor-in-Chief of the Journal of Neurochemistry (JNC) and Sendil Ethiraj, Alfonso Gambardella, and Constance Helfat, Co-Editors of the Strategic Management Journal (SMJ) how the Center for Open Science’s Open Science Badges might more broadly help connect research with the peer reviewed manuscript – in David’s words, helping to “demonstrate reproducible work by pointing to a body of work that contains all of the essential parts: preregistered plans, materials identified with persistent and unique IDs, datasets with clear descriptors, and code for running tests on those data”.
SMJ We think it is part of a package of initiatives designed to increase confidence in the data and methods used in research. It complements other initiatives such as encouraging replications of prior studies. In addition, by making unique data sets more widely available, Open Practice initiatives promote the reuse and augmentation of datasets in new research.
JNC The overall quality of data reporting is poor, which leads to the current “reproducibility crisis” –the outcome in a survey conducted by Nature, in which more than half of the interviewees agreed that there is indeed a crisis, represented by 70% of the interviewees failing to reproduce another scientist’s experiments. To improve the quality and trustworthiness of published data, thorough data reporting is essential and the availability of materials is a fundamental requirement. The open science badges, visual icons placed on publications, certify that an open practice was followed and signal to readers that an author has shared the corresponding research evidence, thus, allowing an independent researcher to understand how to reproduce the procedure.
The open data badge is awarded for making publicly available the digitally shareable data necessary to reproduce the reported results. All raw data and a data dictionary (for example, a codebook or metadata describing the data) with sufficient details and descriptions for an independent researcher to reproduce the reported analyses and results are stored in a repository.
The open material badge is earned by making publicly available the components of the research methodology needed to reproduce the reported procedure and analysis. Commercially available materials/animals/technique/software/equipment need to be described by persistent identifiers (Research Resource Identifiers (RRIDs) for example). Custom-made materials like antibodies or genetically manipulated animals have to be provided upon request.
Lastly the pre-registration badge is earned for having a preregistered study design. A pre-registered study indicates that research design, study material, planned sample size, outcome variables and predictor variables are pre-determined and stored in a public date-time stamped registration, like http://www.clinicaltrials.gov or http://www.osf.io/registries/.
Q. Why should researchers bother pursuing an Open Science badge?
JNC Researchers should be interested in the best quality and reproducibility of their research. Reproducibility leads to follow-up studies and consequently to citations and increased conspicuousness. Furthermore, funders and the public are questioning the value of research if there is low reproducibility and if the research results are rarely translated into clinical success. Badges are an incentive and a quality award, which will drive trustworthiness again.
SMJ The Open Data badge [GA-B3] recognizes researchers who make their data publicly available, providing sufficient description of the data to allow researchers to reproduce research findings of published research studies. Qualifying public, open-access repositories are committed to preserving data and keeping them publicly accessible via the web into perpetuity. SMJ is making the FIVES Project data repository (http://five.dartmouth.edu) available to either store the data and documentation or provide links on the FIVES website to the data and documentation on other open access websites.
Q. Isn’t it just another chore for researchers? What’s in it for them?
JNC No, it’s not a chore. Of course, applying for the open science badges requires the authors to spend extra time; however, this extra work helps certify that an open practice was followed. Researchers must upload all raw data and constructed data along with a clear description of the analysis. However, these materials and data are already available for the researcher at the time of submission and, thus, extra work is required only to upload the files to a repository. The identification of catalogue numbers and persistent identifiers for materials is time consuming as well. However, every researcher benefits from a detailed description of data and materials in each manuscript, because it saves a lot of time to search for the correct materials or protocols. The authors should put themselves in the position of the reader (and they are readers in return) to see and understand the benefits.
SMJ The badges provide an incentive for researchers to publicly disclose their data. Providing data in a form that is usable by others entails time and effort, and the badges provide public recognition of this effort. As we start out the practice, the incentives for researchers to adopt open practice initiatives are perhaps modest. However, over time, we expect that norms around Open Data practices will get established and there will be greater sharing of data.
Q. OK, so what are some of the other benefits?
SMJ Badges are likely to benefit journals in driving reputation. With time, more journals are likely to embrace such practices and it is possible that metrics such as the percentage of articles published in a journal that have the Open Data badge could assume importance. In the near term, it will help signal to stakeholders that the journal takes transparency in data and methods seriously, and the journal values availability of data for future research.
JNC Open Science Policies bring a lot of benefits for researchers. As mentioned above the immediate benefit is a badge award for the quality of the paper. In addition, making data, techniques and materials available helps to boost the researcher’s own research field, recognition and citation. However, it provides the authors with advantages as well: preparing research materials for sharing during the active phase of the study helps them keep track during the experiments and it eases the work for follow-up studies within the same laboratory. If the leading researcher of a certain study leaves the laboratory, all the knowledge about data and materials is stored and the risk of losing essential knowledge is decreased. At the same time, it is much easier for the leading researcher to respond to requests by other researchers. The accessibility of data and materials reflects the confidence of authors in their own research and this will drive other researchers to develop follow-up studies and cite the study. The recognition of institutions of open science practices is increasing. Sharing articles and data is increasingly becoming an expectation and in some cases, open science practice is an obligation to receive grants or even a new job. The pre-registration of studies especially helps the authors to protect themselves from flexible interpretation of results and from “cherry picking” to publish only positive results.
Q. So how did you implement open science badges for your journals?
JNC The Journal of Neurochemistry implemented the open science badges in January 2018. The authors can indicate during the submission process if they are interested in applying for the open science badges. For detailed information, we provide a summarizing info document. The authors have to fill and upload a disclosure form and the editorial office will evaluate the disclosure before issuing the badge, but does not do more than a cursory evaluation of the data, materials, or registration. Such a review might include: confirming that the provided link leads to the data, materials, or registration on a public, open access repository, and that the linked materials are related to the report. The authors are accountable to the community for disclosure accuracy. Regarding the open data badge, authors continue to avoid the extra work of uploading the raw data. If the practice of preparing the data for sharing is established during the study and the consciousness of the importance is part of the training of young researchers, the application for the badges will increase because it does not lead to extra work during the revision process. The open materials badge does not lead to extra work than what the Journal of Neurochemistry is already expecting from the authors. Therefore, we plan to persuade authors to apply for the open materials badge and hope that this will boost the interest to apply for the other badges as well. You can see here an example.
SMJ Recognizing the importance of research transparency and data sharing to cumulative research, SMJ encourages authors to share the data supporting the results in their study by archiving them in an appropriate public repository. In partnership with the non-profit Center for Open Science (COS), SMJ awards qualifying authors an Open Practice badge recognizing their contributions to the open science movement. As of October 2017, SMJ authors have an opportunity at the time of manuscript submission and again at the time of acceptance to inform themselves of this initiative and to determine whether they wish to participate. Applying and qualifying for the Open Data badge is not a requirement for publishing with SMJ, but this badge is further incentive for authors to participate in the open science movement and thus to increase the visibility and transparency of their research. Participating authors will be asked to complete a disclosure form after their manuscript is accepted.
Why Open Science Badges at Wiley?
Let’s start with three assumptions: #1 that transparency in research and its publication is a good thing; #2 that changing to new transparent practices is hard for us all whether we are researchers, peer reviewers, editors, or publishers; and #3 that we would all feel rewarded for that hard work if we were to celebrate transparent practices, no matter how small or large, when we see them. For example, we could celebrate researchers who take advantage of the new transparent choices our journals offer them, like data sharing and citation, and Registered Reports. This is where Open Science Badges come in. They give researchers a small but visible reward when they adopt a transparent practice, with a badge actually on their article. Open Science Badges signal to researchers that journals care about transparency and openness. We hope to encourage researchers to publish research with us that’s more transparent, and for Wiley to lead cultural change in research publishing.
And for Wiley journal editors, here’s our closing message. We have a toolkit to help you launch Open Science Badges quickly and easily. Please speak with your publisher and we’ll get things organized. Thank you.
Image Credit/Source: Cytonn Photography/Pexels