I suspect that you would be hard pressed to find a researcher who doesn’t believe that there are problems with peer review, at least with the unfavorable reviews of grant submissions and manuscripts that they have personally received, even if they are uncertain about the systematic nature of the problem.
Typical peer review model
Editors decide based on unknown study characteristics (likely some combination of novelty, controversy and potential hits but not methodological quality) to desk reject or send for peer review
If sent for external review, based on reviews received and editor’s prior beliefs, decision made to reject or accept, generally a conditional acceptance depending on requested revisions.
If the outcome is rejection, the authors generally receive either 1) a desk rejection notification with no reviewer comments or 2) a rejection letter with anonymized reviewer comments. However, in both cases, there is typically no information about the editorial decision-making process, other than perhaps a vanilla, non-informative “it doesn’t meet our priorities”.
The whole process lacks accountability from both reviewers and editors. Except for the BMJ (and perhaps others that I’m not aware of), journals still rely on anonymous reviews. It is difficult to believe that reviewer diligence would not be enhanced with signed reviews. The absence of any clarity on editorial decision making is not only unhelpful for authors seeking to improve the quality of future resubmissions but also enlarges the potential space for arbitrary decisions that are more influenced by novelty, controversy, and click bait rather than based on the study’s research question and methodological merits independent of results.
Are there other possible solution besides signed review reviews to increase the quality of the peer review system. Two other solutions that have suggested are
Request peer review be performed in the absence of the results) and discussion to encourage rewarding study design and analytical quality
Put more effort into post-publication review. Essentially this has been put forward as an efficiency argument whereby efforts are concentrated on papers that people care about.
Why these thoughts of peer review this Saturday morning. One answer is the role that inadequate peer review plays in allowing the publication of studies with faulty design, data (although in fairness hard to see how peer review can deal with this) and analytics that contribute to the reproducibility crisis and form the basis for many of my previous blog posts. Of course this is only one side of the peer review problem, a kind of type I error. What about the other side, the type II error where a paper or grant is refused by a questionable peer review process.
Yesterday I received a review from a respected Journal, at least as measured by their impact factor of 30.2 and according to their website 3.3 million annual article views and downloads. This is what has stimulated this post and I’m including a redacted version of a letter I sent to the editor in response to this review.
Dear Editor-in-chief of XXXX
I would like to provide you with some feedback regarding your peer review of my recently submitted manuscript ( XX-XXXX ). To be clear, my complaints have nothing to do with the final decision but are related to the process. As my manuscript involves a well performed pragmatic randomized clinical trial (RCT), I have no doubt it will eventually find a good home. Therefore, I am NOT seeking to appeal your decision. Nor am I expecting or desiring a reply to this letter, especially one containing “alternative” facts.
Issue #1 Unacceptable Delay:
The manuscript was submitted on July 8th and my rejection letter with two enclosed reviews received on September 27, a total of 12 weeks or 83 days to be exact. As I have experience as journal associate editor, I am aware of the difficulty in securing peer reviewers and receiving their timely reports. Nevertheless, tardy acceptance replies, and final reports are now successfully minimized by automated computerized managerial systems so that an associate editor with a minimally responsible attitude can typically, especially for a high impact journal like yours, secure reviewers within one week. Reviewers are now expected to return their reviews within 2 weeks and even allowing for an additional week that required another automated reminder, the review process is now at 4 weeks. So in my case, the journal editors then required a further 8 weeks to reach a decision? Seriously? This borders on total disrespect for the authors. No, it doesn’t border on, it is disrespectful!
Issue # 2 Quality of Review # 1:
They begin by mentioning a typo that was in the abstract. Is this really the most important element for this reviewer? Their second point is to flatly disagree with the study premise that controversy regarding different treatment options exists. They supply nothing to support their opinion and completely ignore the manuscript’s introductory paragraphs which explain in detail, with references, the existence and origins of the uncertainty. While I find these two points rather disappointing as an opening critique of our study, their third point is so egregious that it rendered me almost apoplectic. And here it is - “for my knowledge what is a “pragmatic” RCT?”. Bear in mind that the title of my manuscript includes the wording “a pragmatic cluster randomized controlled trial”. This raises several questions. First, while there is no shame in not knowing what a pragmatic RCT is, it does make one wonder about the appropriateness of selecting this reviewer. Second, if the reviewer doesn’t know about pragmatic trials, then they would seem to have two distinct courses of action. One would be to decline the review, and the second would be to spend about 15 minutes Googling and reading about the subject. Simply stating in your review “for my knowledge what is a “pragmatic” RCT?” is NOT a reasonable third option. At this point, I would have thought the editorial board might consider excluding this review from further consideration and replacing them with a third reviewer, if necessary. I won’t bother with the rest of this mundane review which includes questions that were well answered in the manuscript had it been read with the slightest attentiveness and otherwise irrelevant comments.
Issue #3 The editorial decision process.
I will first comment on the report from Reviewer 2. Lest you feel I am over valuing my own work, here is opening paragraph from Reviewer #2
“This is an elegant and innovative trial that provides valuable new evidence to the field. Its pragmatic design that leverages provincial electronic health records and time-clustered randomization strengthen the value of this work while minimizing logistical challenges and reducing costs. The use of a Bayesian analytical approach is commendable, as it allows incorporation of different prior beliefs (vague, skeptical, enthusiastic and summary) highlighting how these could impact the interpretation of the results with respect to clinically meaningful outcomes. The authors have done an exceptional job in making all these complex methodological aspects accessible to the reader through a clear and concise manuscript.”
Reviewer 2 goes on to make several suggestions that would improve the overall clarity of the writing that I am thankful for, and which have been incorporated in a revised version. Returning to the editorial process, I have already commented on the excessive delays, but I have concerns that extend beyond that issue. How, was a reviewer who has no knowledge of pragmatic trials selected to review one? How is a flat rejection arrived at when one review is obviously of ridiculously low quality and the other largely favorable? Is the editorial board unable to distinguish between the quality of the two reviews? The rejection letter makes no comment in that regard, not even the usual generic “cover-all your bases” reply “it doesn’t meet our current priorities” which conveniently absolves the editors from making any meaningful comments about their decision making.
I repeat I am not appealing your decision, and the manuscript is currently under review elsewhere. I offer these comments in the rather naïve hope of improving your peer review process.
Regards
Now that I have vented, it is time to get back to more enjoyable weekend activities.
Citation
@online{brophy2024,
author = {Brophy, Jay},
title = {Reflections on Peer Review},
date = {2024-10-05},
url = {https://brophyj.com/posts/2024-09-14-my-blog-post/},
langid = {en}
}