At the end of October, sixty-five people with an interest in peer review gathered in Brussels for the European meeting of the International Society of Managing and Technical Editors (ISTME). The theme of the two-day event was ‘Becoming more open’ but there were presentations and workshops covering many aspects of peer review management and new trends in scholarly publishing. With so many publishing professionals in the same place, there were also a lot of interesting conversations over tea and coffee between sessions.
There were presentations on topics that have been subject to great discussion and debate in recent years such as Copyright Reform and Open Access. However, there were three areas which seemed to have shown greatest recent progress which are outlined below.
Open peer review
Opening up processes was a key topic of discussion in multiple sessions, whether it was a plea for recognizing that peer review is not the same for all journals, opening up editorial office reporting, or analysing the benefits of open peer review.
In an open peer review panel, Phil Hurst, Publisher from the Royal Society, explained that open peer review translates not only into transparency, but into better reviews and reviewer recognition. Adrian Aldcroft, BMJ Open Editor, added that currently bias is a main disruptor in the review process, and while you would intuitively say ‘go double blind’, that is not the answer. Double blind review can contain hidden bias, and at least with open peer review any bias is upfront. In his experience, open peer review leads to higher acceptance rates due to more constructive reviews, so we can publish more. Dominika Bijos, Research Associate, sees open peer review as part of educating the next generation about how to judge science. It increases the awareness of what peer review means, how it works and what it is. For young researchers, providing good reviews gets you known with the editors. But they need guidance too; opening peer review shows the effect reviews have and the value of a reviewer’s contribution.
Open peer review trials done by BioMed Central showed that reviews returned are more substantiated with evidence and the number of submissions grew. However, the number of reviewers who agree to review goes down. Further disadvantages are that reviews take longer and that young researchers might be uncomfortable providing (critical) reviews of senior researchers. There is also scepticism from editors to be overcome, but open peer review doesn’t question their integrity, nor does it break any agreement of confidentiality.
Opening peer review is a good way forward, but although there seem to be no major adverse effects, it is not a simple switch. However, there is no need to jump to open peer review in one go; there are levels of open peer review, baby steps or bigger strides to take. It is worthwhile considering whether it is more important to know who reviewed a paper, or if is it more important that the content is available.
Editorial Office reporting
As part of ISMTE’s mission to provide education and training for those working in editorial offices, the Society’s Education Committee has been working with a statistician to develop best practice guidelines for reporting editorial office data. They are now available to members on the ISMTE website and attendees of the European meeting were given a sneak preview. Jason Roberts, from Origin Editorial and the founding President of ISMTE, explained that there is currently no standardization in peer review management reporting and little consensus on what parameters to use. The new resource details five essential reports for editorial board meetings and provides templates to help editorial staff report, analyse and present data in more meaningful and accurate ways. He talked through the most appropriate graphs to use for presenting numbers of submissions, the importance of giving three years of data to allow for comparisons, and the advantages of reporting turnaround times with a median and a coefficient of variation rather than a simple mean. Those who work in editorial offices are not always as mathematically literate as the editorial boards they report to, so this resource will be fantastically useful. We all left the workshop with the intention of revising our reporting templates!
This is a bottom-up network of scientists and stakeholders who are collaborating in an EU funded research programme with the aim of improving journal and grant peer review by building evidence-based models of best practice, implementing initiatives and improving collaboration between researchers studying peer review. The chair of PEERE, Professor Flaminio Squazzoni, was interviewed by Exchanges in 2015, and at the ISMTE conference Marco Seeber of Ghent University gave an overview of progress.
The most important piece of news was that data sharing arrangements have been agreed between major stakeholders, i.e., publishers. Data will be fully anonymised in compliance with all relevant legislation, but there will be sufficient information to help PEERE researchers discover inter- and intra-community links and explore the patterns of behaviour and incentives which motivate best practice. PEERE is also asking specific questions such as: ‘What is the effect of having more people review a paper?’; ‘What is the effect of double blind versus single blind peer review?; ‘How do junior researchers compare with experienced researchers as reviewers?’
Some trends are already emerging. For example, there appears to be little benefit in assigning more reviewers unless an unfair review is returned or a reviewer is late. This seems like a good argument for aiming to have that extra reviewer in the first place. In another study it emerged that a particular conference tended to display fewer posters from authors who were unknown to them even when they were experienced authors in other areas.
PEERE seems destined to provide extremely useful data for raising the fairness of peer review as well as its efficiency.
Like any conference, the most important aspect was the connections formed with colleagues both old and new as well as the exposure to new perspectives from fellow professionals working in related areas of peer review. However, EO Reporting was certainly the most useful hands-on part of the conference, open peer review perhaps the most open to debate and PEERE the area with greatest evolutionary potential.
Authors: Hannah Wakley (Senior Production/Managing Editor), Noel McGlinchey (Senior Editorial Assistant) and Iris Poesse (Associate Managing Editor).
Image Credit: Hannah Wakley