Communications of the ACM
Communications of the ACM
Technology Has Social Consequences
A conference program committee (PC) member received a paper for review. He distributed the manuscript to his research group to "solicit their opinions of the paper" and the group embarked on improving the results of the paper under review. The research group then submitted their own paper to another conference, their submission occurring three months before the first paper was to be presented at a conference. When eventually confronted (the short gap between the appearance of the two papers triggered questions), the PC member responded with "Was that wrong? Should I have not done that?" (The reader may want to search for this phrase on YouTube.)
Amazingly, this PC member was not aware that a conference paper submission constitutes privileged communication. In theory, reviewers should immediately "forget" what they have read. For reviewers to use such privileged material for their own work immediately creates a blatant conflict of interest. How did this PC member, a full professor in a respected university, not know such a fundamental rule of scholarly reviewing?
To understand how the ethics of program committees has declined, one must review the history of computer science program committees over the last 50 years. Until the mid-1990s, program committees met in face-to-face meetings. This had two significant consequences. First, PC members bore the cost of attending PC meetings, leading them to be careful with accepting PC service commitments. It was rare then for one to serve on more than one PC per year. Second, junior PC members had a chance to interact intensively with senior PC members. There was nontrivial social pressure on junior PC members to demonstrate their competence in PC meetings. Thus, PC service provided important socializing experience, where unwritten norms and customs were learned by observation.
With the emergence of the WorldWide Web in the mid-1990s, physical PC meetings suddenly seemed wasteful, as it became possible to conduct virtual meetings without incurring travel expenses and headaches. Conference-management software tools emerged and many communities abandoned physical meetings in favor of virtual ones. I was very much in favor of this change back then! It took, however, a few years for the adverse consequences of this change to become visible.
Economists would tell you that a commodity priced too low would end up being overconsumed. PC service is a commodity with positive utility. Our community views PC service as a form of professional recognition; in fact, it is one of the few markers of professional recognition available to junior researchers. Since the "cost" of PC service has dropped with the switch to virtual meetings, "consumption" has gone up. Indeed, it is quite common today to see researchers serving on several PCs per year.
Of course, one cannot expect the same level of effort from someone who serves on one PC per year as compared to someone who serves on multiple PCs per year. Indeed, in the 1980s it was typical to see every submission read by five to six PC members, today the norm is often three to four reviewers for submission. Furthermore, these reviewers are often not PC members but "subreviewers." In fact, while the concept of subreviewer was originally developed with the purpose of bringing additional expertise to PCs, today it is viewed as an opportunity to train junior researchers in the art of paper reviewing. Thus, the role of a PC member seems to have evolved from that of a reviewer to that of "review orchestrator."
What has been the outcome of this development? Many of us are quite familiar with this outcome. The quality of conference reviewing has declined and the selection process has become far more random. Two years ago, I wrote in this space about "Conferences vs. Journals in Computing Research." The declining quality of conference reviewing was one of triggers that spurred me to write that editorial.
The loss in quality of conference reviewing is just one result of the move to virtual PC meetings. Another outcome is the loss of socialization that took place in PC meetings. It is this lost socialization that contributed to a senior researcher being ignorant of one of the most basic rules of scholarly reviewing.
We all know that technology has social consequences. This applies to us as well. The switch to purely virtual meetings did not serve us well. Many communities have already realized that and are combining virtual and physical meetings to merge the strengths of both formats. Such practices, I believe, should be widely adopted. Technology can be managed!
Moshe Y. Vardi, EDITOR-IN-CHIEF
©2011 ACM 0001-0782/11/0500 $10.00
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected] or fax (212) 869-0481.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2011 ACM, Inc.
Comments
Anonymous
Dear Moshe,
First of all, many thanks and deep appreciation for putting this serious problem on *the* table of the community, CACM.
The problem is indeed very serious. I'll allow myself to add my 2 cents on why we all agree to serve on multiple PCs, with the situation getting worse with every year.
While WWW allowed for virtual PC meetings, Latex (and to some extent Word) allowed for preparing an X-page paper in virtually no time. As a result, the number of conference submissions continuously and rapidly grows. While most of these submissions would be "summary rejected" at any reasonable journal, in most (all?) conferences they are put through a reviewing process, wasting time of 3-4 reviewers. As a result, the demand for PC members keeps growing, while naturally the pool of qualified reviewers per topic remains more or less of the same size (or at least grows much slower).
Understanding of this situation, makes each of us dealing with the same dilemma every year: joining more than one PC means everything that you point on in this and previous articles, but refusing to help means leaving some PC chairs (that we usually know and appreciate) working with a ... less professional team. And this is a serious dilemma.
The only partial solution I see to the overall problem (and I dedicated quite a lot of thought to it in the last couple of years) is to (re-?)introduce a mechanism of "summary reject" to the conferences. If only a small portion of submissions will require a careful reviewing of details and actual writing of the review, then only a fraction of the current PCs will be needed, and PC chairs will get legitimacy to demand high quality of reviewing.
With warmest regards!
Carmel Domshlak (Technion, Israel)
Anonymous
Carmel, there is some truth to what you say, but I am not convinced that it is paper proliferation that drives conference proliferation. It sometimes seems the other way around. I have seen many conferences that the number of PC members is larger than the number of submissions :-)
Moshe Vardi
Anonymous
Moshe,
Point taken ;), but at least in part, this is yet another side of the same coin. PC chairs are worried these days not to get the most qualified PC members if these will be asked for too much work. And so they expand the PC ...
But of course, everything I wrote was to add my 2 cents, not to replace with them your $2 ;) Again: many thanks for the article!
Carmel
Anonymous
See Jeff Naughton's ICDE'10 slides at
http://lazowska.cs.washington.edu/naughtonicde.pdf
Anonymous
Dear Moshe,
Thank you for bringing up the issue. I fear that your observations are quite correct.
These implications of technology to the conference culture further exacerbates the ills that you have already discussed in past articles.
I would like to add another example to the discussion.
Proceedings papers are necessarily limited in size, and often omit improtant information such as proofs, experimental data, and so on.
It used to be a common practice to require appendices to be submitted together with the conference paper, so that, at least, the material would be in the hands of the referees. As you remarked, technology affects practices: I have the impression that it is a growing practice, at least in certain conferences, that papers point the reader to web resouces for missing information, which seems a most natural use of the current technology.
But technology strikes again: I have heard on few occasions that reviewers are known to avoid looking those web resources up, assuming that visitors to the web site might be tracked, and fearing for their anonymity. The natural development: submissions referring to web resources that do not even exist (and the reviewers not noticing it). It happenned.
Journals should not have this problem. First, the length of an article's text is not so severely restricted. Secondly, digital libraries enable the addition of electronic appendices to journal articles, which could include, for example, experimental data. I believe that this is an important consideration for journals to fulfill their roles in archiving research. Web resources are notoriously volatile, even if at the time of submission everything was proper.
The issue of ensuring that a reviewer has access to all the infomation they need is, however, pertinent both to journals and conferences, and I think that it should be considered by journal editors / conference committees.
Amir Ben-Amram
Anonymous
Dear Moshe,
Thank you very much for writing about the declining of ethics of program committees. There is another ethics problem which is the result of the one you mentioned. If a person is a member of programming committees at two - three four closely related conferences and is considered to be the most suitable person for reviewing papers in some branch of computer science then, to very high extent, he rules by himself the fate of all publications in this area because, with the very high probability, the same person will be the member of the editorial board of the specialized magazine (transactions).
Another example of the declining ethics is the rewarding of the programming committee members with the Best Paper Award. This became the practice on programming conferences, while in all other areas of human activities it's unthinkable to be a judge in the competition where you participate.
My third remark is not on your text but on the first comment from Carmel Domshlak. The number of the conference submissions is not the result of using faster printing equipment but the result of evaluation of the scientific results by the number of publications. For the previous generation of scientists it would be a very good result of the long and very successful 40 or 50 years career in the University to have 70 80 publications (I am not writing about extraordinary cases). Now you can look through the web sites of contemporary assistant professors who are 35-40 years old and after 10-15 years of career they demonstrate you a list of 80 - 100 or more publications. Does anyone really think that there are new ideas in all these papers? "Something is rotten in the state of Denmark."
Thank you for your article,
Sergey Andreyev
Displaying all 6 comments