Communications of the ACM
Computing ethics
Where Review Goes Wrong
Credit: Scott Norris Photography
I am a researcher twice accused of professional misconduct in the publication process. The first incident happened when I was a junior professor submitting the definitive paper from my Ph.D. research to a journal. The second happened quite recently. Despite these accusations, I am a successful researcher and teacher. This column is my appeal to reviewers and editors for caution and moderation.
In the first instance I went through several rounds of reviews, revisions, and resubmissions. All but one of the reviewers accepted the paper, and the paper was eventually rejected. I resubmitted the paper to another journal. Unbeknownst to me, the reviewer who had previously rejected the paper was contacted as reviewer again. The result was that the editor, in an email sent to all reviewers, charged me with knowingly submitting a paper with incorrect results. It had never occurred to me that I was doing anything wrong. I felt scared, helpless, ashamed, alone, and confused. It took a while to dig out proof that I had checked the veracity of the paper. I forwarded to the editor all previous reviews and my responses. I also forwarded my email correspondences with a mathematics researcher who had helped me verify proofs and address the reviewer's concerns. The review process was restarted with the same set of four reviewers; as I expected, the paper was rejected. I rarely submitted to a journal again because I was terrified of being charged with trying to "shop" a rejected paper.
The second accusation of professional misconduct happened recently. Some months ago, my student and I submitted a paper to a conference. A couple of months later, we submitted new research—generated since the last submission—to a workshop connected to the conference. The conference paper was rejected; instead of getting reviews for the workshop paper, we were accused of unethical conduct for submitting a paper that had significant overlap with another paper in review, and for not citing the paper in review. No proof or examples of the overlap were presented. There was no attempt to contact us authors prior to the accusation and negative decision concerning our workshop submission. We tried to contest the decision by sending proof that the writing overlap between the two papers was 3% and by explaining that the shorter workshop paper presented a new algorithm developed and validated after submission of the earlier paper. The conference chairs and the program committee served as the judge and the jury; we had no arbitrators, no voice. We were punished without being given a chance to rebut.
Trying to publish a paper that was rejected earlier or trying to publish new results are part of the publication process, and researchers should not fear being charged with professional misconduct for doing their job. The review process is unbalanced with concentration of power in the hands of reviewers. In both incidents, when a reviewer charged wrongdoing, there was an immediate presumption of guilt followed by punishment. By the time we were contacted, our guilt and punishment was fait accompli. We felt helpless and wronged, with no possibility of our names being cleared of wrongdoing. When a reviewer suspects something amiss, it is important that editors contact the authors for an explanation. The final decision should reflect input from both sides. In order for authors to understand and accept a decision, they should feel that their voices were heard.
The review process should incorporate the ethos of research and the publication process. A review process is adversarial since reviewers are tasked with checking that a paper is correct, relevant, and original, while authors believe that their paper is correct, relevant, and original. Both reviewers and authors may make mistakes, but errors/misjudgments by reviewers can lead to punishment of authors. Therefore, it is important for editors/chairs to be impartial, which is not possible without author input.
A reviewer spends a few hours on a paper, while authors invest several years, so authors often understand their research more than the reviewer. When a paper is rejected at one venue, authors resubmit to another venue, hoping that a new set of reviewers accept their paper. Authors try to revise their paper based on reviewer comments, but it is sometimes impossible to address all the reviewers' concerns. Possible reasons for not addressing comments are: lack of time since the next conference deadline follows immediately, author fatigue after going through countless revisions, lack of resources to address comments, contradictory comments by various reviewers, to name a few. Therefore, reviewers who are reassigned papers they rejected elsewhere should not label authors as unethical if their comment is unaddressed in this new submission.
Sometimes rules are ambiguous and authors unintentionally break a rule. Apropos, rules on when and how to cite one's previous papers are contradictory: for double-blind, citing one's older papers is wrong; for single-blind, not citing one's older papers is wrong. When a paper is resubmitted, it is possible that authors forget to add/remove their paper citation. Sometimes, authors are simply embarrassed by their earlier paper and choose not to cite it. Sometimes, authors do not cite older papers since it appears as an attempt to increase citation count of their papers. Reviewers may attribute sinister intentions where none exists. Instead of charging authors with misconduct, asking authors for an explanation is reasonable. Editors/chairs may subsequently choose to eliminate the paper from consideration without charging authors with professional misconduct.
Reviewing is subject to error and bias.1,2 In both incidents, the reviewing was single-blind. The double-blind review process is vanishing, which is unfortunate. A reviewer might be biased by author names and affiliations, so removal of the double-blind process hurts researchers from lower-ranked institutions. Moreover, some conferences allow reviewers to resubmit their review after seeing other submitted reviews of the paper; this exacerbates the problem of biased review. Authors who are not part of the elite group have to scale an impossibly high bar to get their research published in reputed conferences/journals.
The review process is unbalanced with concentration of power in the hands of reviewers.
In both incidents, charges were brought by reviewers who had rejected an earlier submission by the authors. Before punishing us, the possibility of reviewer fatigue and bias should have been considered. A friend who is on several program committees laughingly mentioned that he rejected a paper, submitted to three different conferences, thrice. The authors of this paper may have given up without realizing the paper was reviewed by the same set of reviewers. This problem could be addressed by asking potential reviewers whether they have previously reviewed (and rejected) the paper, and if so, whether they could impartially review this new submission. The framing of these questions might help reviewers understand their biases. If impartiality is not possible, then one should recuse oneself from reviewing the paper.
Reviewing research papers is a difficult task, and I thank reviewers, editors, program chairs, and others involved in the process. It is arduous to read and comprehend technical papers that are likely written by young researchers who are learning to articulate their research. In recent years, paper submissions have spiked, so reviewers may be reading a large number of papers. Time pressure on reviewers has led to non-reviews, which provide little to no feedback. Sometimes, reviews are sprinkled with words such as "moronic," "stupid," or "myopic" that reflect the frustration of reviewers. One solution might be to increase the size of program committees by making them more inclusive and to reduce overlap in program committees. Reviewer bias could be reduced by bringing back double-blind reviews and by ensuring that, as a general rule, reviewers do not review papers they have earlier rejected at another venue.
Almost a decade after the first incident, the editor who accused me of misconduct sought me out and apologized. I thank this editor because his apology allowed me to evaluate and acknowledge the impact of that first wrongful accusation. This second accusation has little impact on my career. I am speaking up on behalf of young researchers who are just embarking on their careers. I appeal for rules and guidelines, which protect David and keep Goliath in check. I appeal for the psychology of research to be incorporated into the review process, for program committees to have more diversity and less overlap, for reviewers to understand their inherent biases, and above all, for chairs and editors to have a preponderance of evidence before charging authors with wrongdoing. After all, if Hardy had accused the unknown young mathematician who sent him well-known theorems as original work, one of the greatest mathematical geniuses, Ramanujan, would have been lost to the world.
References
1. Publication Ethics by Council of Scientific Editors, 2016; http://www.councilscienceeditors.org/resource-library/editorial-policies/white-paper-on-publication-ethics/
2. Kahneman, D. Thinking, Fast, and Slow. Farrar, Straus and Giroux, 2011.
Footnotes
I thank Rachelle Hollander for her guidance through the review process and for steering this column toward publication. I thank several anonymous reviewers for their comments and suggestions, which have greatly improved the column; special thanks to the reviewer who suggested the column title. Finally, I thank Cindy Childers, Larry Dowdy, Andras Fekete, Anu Mathai, and Royce Withey for their support and feedback on early versions of this column.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2017 ACM, Inc.
Comments
Gunnar Wolf
Thanks for the interesting read. As a newcomer to the academic publishing world (I dare not say "a young academician" anymore), I think there is an extra point you are missing here: Language. English is the lingua franca of scientific publishing, and while many of us also publish in other languages, English is clearly the way to reach most readers.
It is very clear to me as a Mexican to feel an "accent" when reading English papers submitted by people from cultures with a different language structure - Mainly Arabic, Chinese, Indian. I am sure that, no matter how careful I am when writing, my native Spanish thought structure can be felt in my writings; it is sometimes harsh to read a reviewer says (as it happened to me recently) that "the authors obviously did not take the time to proofread their draft".Our field is quite prone to intercultural blindness, and the language-imposed barrier of entry also takes its toll for those of us in non native English-speaking countries.
Igor Schagaev
Dear Elizabeth, all you have described is painful and ... true. But still is not the greatest pain.
How about behaviour of EC R&D system when they receive your project proposal, reject it and then resubmit though their friends very similar one to fund? My former colleague from China did spot quite a lot of thing like that. Or when MIT refer your paper and then easily copy a semantics of few paragraphs from it, i.e. - why worry we did cite you! Same is everywhere - India "researchers "do copy even pictures of your papers, simply putting their names instead of yours; - Russian "experts" re-patenting your patents claiming novelty after they have learned from you every bit for months.
Displaying all 2 comments