Communications of the ACM
Home/Magazine Archive/September 2015 (Vol. 58, No. 9)/The Pros and Cons of the 'PACM' Proposal: Point/Full Text
Point/Counterpoint
The Pros and Cons of the 'PACM' Proposal: Point
Credit: iStockPhoto.com
Make no mistake, computer science research is a resounding success. Its research advances, peer reviewed and published in conferences, created new billion-dollar industries, changing how we do science, business, government, entertain ourselves, and communicate.
While many universities accommodate the rigorous, conference publication culture of CS, others do not. Our practices differ from other sciences that only peer review articles in journals, using conferences for unreviewed talks, posters, and articles.
I recommend the ACM Publications Board proposal because it articulates best principles for research publication—rigorous peer reviewing and no page restrictions—yet is flexible enough to accommodate our conference culture. Adopting it will further improve our process and recognize our archival quality conferences as such, harmonizing computer science with the international science community.
Computer scientists use a conference reviewing process with annual deadlines to combine peer review with timely publication (typically five to seven months from submission to publication). Compared to journal reviewing, I believe our conference process has distinct advantages that this proposal maintains.
"I recommend the proposal because it articulates best principles for research publication, yet is flexible enough to accommodate our conference culture."
ACM conferences are organized by Steering Committees, with general and program chairs and SIG representatives. Program chairs generally serve once for one year. They select Program Committee (PC) members, who each review 10 to 30 submissions and act as editors, choosing accepted papers. Because of increasing numbers of submissions, many conferences have an additional committee with fewer responsibilities. They each review 1 to 10 papers and do not attend the PC meeting. When well deployed, these reviewers deliver highly expert reviews. The most important job of program chairs and journal editors is the same—to find expert reviewers, going beyond the committee and associate editors as necessary. While individual reviews are blinded, the committees are publicly acknowledged. Reviewers are known to each other, increasing accountability.
Because reviewers and leadership change regularly and every submission is reviewed, no one editor or associate editor exerts influence and scientific biases for many years or by desk rejecting submissions. Consequently, conferences are more likely to include diverse problems and approaches.
A disadvantage of conference reviewing is page-limiting submissions to control reviewer workload, which may cause omission of material, such as proofs and methodology. With this proposal, reviewers can judge page-limited submissions, but require additional material, reviewing it or not, as appropriate. Concision is a virtue that reviewers may also require. With one to three months to revise accepted papers (and more for rejected submissions), I think the result will be articles with appropriate content without overburdening reviewers.
Conferences often produce more reviews than journals, providing authors richer feedback from diverse points of view. Submissions to conferences, such as SIGGRAPH, SOSP, ICSE, and OOPSLA, receive four to six reviews in rounds—papers that remain under consideration are assigned more reviewers. This process limits reviewer workload, while accommodating the growing numbers of submissions many conferences are experiencing.
The conference process fosters our research communities; a benefit I believe is underappreciated. For instance, in-person PC meetings structure research discussions on problem selection, contributions, approaches, methodology, and other scientific values. Committees include and train fresh Ph.D's. Every researcher, from a first-time PC member to the most famous in the field, has an equal opportunity to express scientific viewpoints. A common experience after your first PC meeting, which I distinctly remember, is that young researchers feel respected. What better way to welcome researchers to our scholarly community?
Program committee meetings also build cross-institution relationships at all levels. Reviewing alone in your office does not foster community.
One remaining issue is citations. Many ACM conferences have very highly regarded and established brands. Unfortunately, the ACM Digital Library ambiguously specifies how to cite some of them. For example, see the Export Formats for PLDI, ISCA, and older SIGMOD papers. They have two DOI identifiers, one conference and one SIG Notices. ISI indexes SIG Notices, so some authors prefer them. This duplication is a historical accident due to SIGs that made conference proceedings issues of their SIG Notices as a member benefit, producing duplicate, ambiguous, and non-branded bibliographic entries. Worse, some conference names were not standardized from year to year. I strongly recommend that ACM work with the SIGS to develop citation formats that ensure the bibliographic record avoids confusion, the new citation clearly features conference brand names, and consistently apply this citation format retrospectively and thenceforth.
©2015 ACM 0001-0782/15/09
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected] or fax (212) 869-0481.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2015 ACM, Inc.
Comments
Rajesh Purohit
Yes agreed to this view as conferences are to be encouraged as well.
Scott Whitmire
Interesting comments, but the main thrust seems to be that the conference-preference currently in place in the ACM is preferable because of the program committees. Having participated in a conference program committee for over a decade, I will agree that the PC format for reviewing submissions is better than the typical peer-review process used by most journals.
However, the conference format suffers from one major issue: the limited agenda space. Time limits imposed by the conference format necessarily limit the number of submissions that can be selected. Larger conferences can select more submissions, but necessarily limit the exposure of each submission. The result is that valuable research is not published, or if it is, not widely seen. Publication in a journal can remedy both of these issues.
Conferences are good for fostering dialog in situations where problems have not been solved and researchers submit position papers or preliminary research. In these cases, the dialog is the goal, not the publication.
Journals also have an advantage of a stable brand and more permanence as a place to find research. I've been on both sides of the research problem and find that locating relevant research in a conference proceeding is much more difficult than a journal. That may be caused by the way libraries select their subscriptions, but it does affect the ability to find information.
A compromise appears to be in order here. If the program committee concept is a superior peer-review process, why not use it to select submissions for journals?
Conferences and journals are both necessary, and I am in favor of the proposal, but the arguments given in the article are not necessarily relevant to the issue. It is not an either-or-situation.
Kathryn McKinley
Thank you for your comments! While Journals and Conferences could have similar reviewing processes, currently they do not.
In theory, both Journals and Conferences are not limited in what they accept, but in practice they are, because being "selective" is one metric of quality and the best venues are selective. If there is valuable research in a submission, it is easy for journals and conferences to include it. Conferences do have more strict physical limits (presentation slots vs pages on the web). However, many have added parallel sessions to address this problem and are adding more. Furthermore, conferences could simply publish more papers and not give all of them a speaking slot.
Kathryn S. McKinley
Principal Researcher Microsoft
Displaying all 3 comments