Elizabeth Moylan, Publisher at Wiley, talks to Michael Willis, Senior Manager in Wiley’s Content Review team, about the work he and colleagues have undertaken to explore what better peer review looks like.
Q. What inspired you to define a set of standards for ‘better peer review’ ?
A. The starting point was a question thrown out by a Wiley colleague: ‘is there a gold standard of peer review?’ That got us thinking about what good peer review looks like. I guess we all have our preconceptions of what good peer review looks like – it should be timely, ethical and fair - but we felt we needed to articulate the details more usefully and also help journals to improve in measurable, specific ways.
This in turn led to a project to define essential areas of best practice for peer review. We thought about different characteristics of the peer review process, and then we described the ways in which each of these might be manifested. Taking integrity as an example, and pertinent to the theme of this year’s Peer Review Week, a journal might achieve greater integrity in its processes by working towards greater geographical and gender diversity in its reviewer pool.
You can read more about our project in this blog post which we wrote soon after the project launched.
Q. How did you go about researching some of the issues in peer review?
A. Having defined our scope, we then published a survey seeking the views of editors, reviewers, authors, readers and the general public, asking them to share examples of good practice in peer review. We received 40 case studies which we grouped under the headings of integrity, ethics, fairness, usefulness and timeliness.
Simultaneously we also explored the current literature on peer review to identify major themes that recur, such as peer review models, transparency, ethics and bias.
Q. What did you conclude?
A. Looking at the responses we received from the survey, we realized that we could help journals identify their strengths and weaknesses in each of the five themes we identified, and so we devised a detailed checklist for journals to use. The checklist covers such aspects as author guidelines, reviewer guidelines, and internal editorial processes.
We wrote up our findings in this preprint and shared them through a poster at this year’s conferences of the International Society of Managing and Technical Editors. We’re hoping we can publish our work in due course, and we’re running a symposium on the topic at next year’s World Congress on Research Integrity.
Q. In many respects, you touch on issues broader than peer review, e.g. recommendations for policies regarding authorship, why is this?
A. That’s correct. By ‘peer review’ we encompassed the whole journal editorial process and not simply peer review in the strict sense, i.e. review by independent expert colleagues. So we looked at (among other things) the integrity checks undertaken by the editorial office, the process for handling submissions used by journal editors, and how editorial and peer review processes are described in author guidelines. This is illustrated in Figure 1.
Figure 1: The act of peer review in the context of the end-to-end peer review publishing process: a simplified view.
Peer review in the strict sense doesn’t happen in abstract; it’s part of a nexus of processes. You cannot separate editorial control from how external peer review operates, and all players in a journal team have differing (albeit interconnected) responsibilities within this. So we wanted to cast the net more widely than how peer review is typically studied.
Q. So what’s next?
A. We’re concentrating our efforts at the moment on developing and refining the self-assessment checklist which you can find at the end of the preprint. We want to help journal teams reflect on their current practices and consider if and how they can improve. We’ll road test the checklist on a few journals to see how they get on with completing it, and we’ll make improvements to it as necessary. Then we’ll scale up to implementing it on as many of our journals as possible in the coming months, and we hope we can share more information about this in due course.
Ultimately this whole project is about helping journals to aspire to ‘be better’, and we want to play a part in guiding and equipping journals to achieve this. Taking the theme of this year’s Peer Review Week as an example, we can help journals identify if they need to improve in the area of diversity, and our checklist can encourage them to aspire to be more diverse.
Looking to become a peer reviewer? Check out our hints and tips.
Are you an active peer reviewer? Join over 135,000 Wiley Peer Reviewers on Publons to help you record, verify and showcase your peer review contributions.