Communications of the ACM
Fixing the Process of Computer Science Refereeing
Last week at ECSS 2010, the annual meeting of Informatics Europe in Prague, we heard a fascinating keynote by Moshe Vardi, editor-in-chief of Communications of the ACM, on "The Tragedy of the Computing-Research Commons" . Professor Vardi talked about the importance of engaging in activities that benefit the community even if they bring no huge immediate reward to the individuals who participate in them. He lamented the degradation of the computer science culture, due among other causes to shoddy refereeing practices.
Lamenting about the reviewing process is common, and everyone has horror stories. Yet the simple solution is almost never considered: turn the default refereeing mode to open, rather than anonymous.
Some cases may still justify anonymity, but they should be the exception, calling for a specific justification. Refereeing should be what it was before science publication turned into a business: scientists giving their polite but frank opinion on the work of other scientists. Anonymity just encourages power games, back-stabbing and, worst of all, poor quality: since non one can call your bluff, you are not encouraged to be a good referee. Of course many people do an excellent job anyway, but they do not necessarily prevail: in the highly competitive world of computer science publications, conference publication in particular with its schedule pressures, one incompetent but arrogant negative review typically outweighs five flattering and carefully considered analyses.
By revealing who you are you force yourself to write reviews that you can defend.
More than two decades ago I started refusing to do anonymous reviews. This stance may not have only brought me new friends (which may not be a big deal, as I am not sure people who hate you because you found flaws in one of their papers are worth having as scientific friends), but it has certainly made me a better reviewer, and in fact did bring me some friends — people who are grateful for having gained new insights, positive or negative, into their own work.
A more complete discussion and rationale can be found in a Web page, to which I regularly refer editors asking for reviews. That text, written several years ago, is verbose and should be rewritten, but it does include the basic analysis.
The decision to perform open refereeing was personal and until now I have always refrained from proselytizing. Seeing the degradation in refereeing, however, I believe such reserve is no longer appropriate. Establishing open refereeing as the default strategy is the first step towards fixing the flawed culture of computer science refereeing.
Comments
Anonymous
Sounded good to me. I tried it just now at another journal. Sent a link to this blog and the editor did not want to change their policy. So I caved in.
Anon., but really Jared Bernstein
Anonymous
I'm bothered more by reviews which are anonymous to other reviewers, rather than to the authors. At the USENIX FAST program committee the other week the papers were blinded, but I knew that my reviews (and my identity as the author of those reviews) would be public to the rest of the committee members.
(the review system - hotcrp - was set to hide reviews on my assigned papers until I completed my own reviews, but that's just a safeguard against intellectual laziness and me-too-ism)
Frankly, I think the prospect of a crappy review being visible to an assembled panel of high-profile people in the field, who are going to interact with me face-to-face after seeing my review, is much more motivating than the prospect of my identity being revealed to the authors.
Peter Desnoyers, Northeastern University
Displaying all 2 comments