Expectations of peer review
The expectations of what peer review should or can achieve are neither stable nor generally agreed upon. The expectations differ through time, but also across and even within scientific disciplines. This wide variety of expectations of peer review also explains the wide diversity of peer review procedures, since diverse expectations ask for different approaches.
Horbach and Halffman (2018) give an overview of different expectations of peer review, and provide references for further reading. Some of the main expectations frequently mentioned in the literature are:
Selection and accreditation of ‘good’ science
The peer review system acts as a gatekeeper, deciding upon what gets incorporated in the scientific literature. Therefore, expectations of peer review are closely intertwined with expectations of the scientific literature. Some argue that all “valid” research, including all failed experiments and negative results should be published. Others argue that published research should not only be valid, but should also be relevant or novel, and thus worthy of researchers’ scarce reading time. Not only does peer review select, it also acts as an accreditation mechanism. A research article receiving the stamp ‘peer reviewed’, is generally assumed to be a piece of high-quality science.
Improvement of the quality and accuracy of submitted research
Besides deciding what research is “worthy” of publishing, many expect that the peer review process improves the quality of submitted manuscripts. Through their constructive feedback, reviewers can genuinely contribute to the work under review. Several surveys have indicated that indeed many authors feel that their manuscripts benefited from reviewers’ comments. Some argue that to meaningfully contribute to research quality as a reviewer, review should be performed earlier in the process. When review is performed before data collection, changes can still be made in the experimental design.
Providing fair and equal opportunities to all actors
The peer review process is in charge of what gets published and what does not. Since the course of careers often depends on the publication of peer reviewed articles, many expect the peer review process to be fair, and to offer equal opportunities to all actors. However, there are several indications of biases in peer review, making it harder for, among others, women, scholars from less prestigious institutes, and minority groups to publish their work. As a response to these biases, several journals experiment with degrees of blinding during peer review. However, others argue that openness and transparency are the way to a fair process. In addition, with open peer review, reviewers can be fairly credited for their work.
Active filtering of problematic research
There is a heated debate about the role of peer review in active filtering of problematic research. Some feel that the system is not capable of performing this task, and in fact was never meant to do so. Others consider the editorial process as the gatekeeper of science, and thus expect it to stop problematic research from getting published.
One of the arguments against the role of peer review in the active filtering of problematic research, is that reviewers can never be ‘at the scene of any crime’ and that, even with the provision of background material or raw data, one cannot expect reviewers to thoroughly check experimental results. However, the editorial process consists of more than just the work of reviewers, and recently new technologies such as image manipulation and text similarity scanners have emerged. These technologies offer possibilities for automated filtering of problematic research. If these technologies become more popular, automated filtering for problematic research could also become the standard.
