Communications of the ACM

Departments

Reboot the Computing-Research Publication Systems

By Moshe Y. Vardi
Communications of the ACM, January 2021, Vol. 64 No. 1, Page 7
10.1145/3437991
Comments (1)

A year ago, I proposed that ACM establish a policy change for its conferences, requiring that authors of accepted papers may opt out from in-person involvement and contribute instead by video.^a "Be careful what you wish for," says an idiom. By mid-March 2020, conferences were forced to virtualize due to COVID-19. It is now clear that conferences will continue to be virtual at least until the middle, if not the end, of 2021, and perhaps even beyond that. While I am happy this will prevent adding tens of thousands of tons of CO₂ to the atmosphere, the virtualization of conferences sharpened my conviction that the computing-research publication system is badly broken and is in need of a serious reboot.

How did we get here? Back in the 1960s, journals were slow, while the field was new and needed to move fast. Conferences offered a solution: Present a preliminary version in a conference with a six-month submit-decide-present cycle, get feedback and credit, and then publish an archival version in a journal. Program committees then did not review papers; they selected papers for the program.

Over time, the selection process became a review process: decision notices became decision notices with some feedback, feedback evolved into reviews, and reviews eventually led to rebuttals. Furthermore, conferences have evolved from being a venue for preliminary publication to, in practice, a venue for archival publication. Also, conference-program time pressure, led to selectivity, which led to prestige. As a result, computing-research conferences today are clearly the preferred venue for archival publication.

But conferences were never designed to provide a venue for high-quality archival publication, since they lack the appropriate editorial process, whose essential element is iterative improvement. It is not uncommon for published journal papers to go through two and even three versions before the reviewers and editors are satisfied. Conferences that run "review experiments," such as NIPS 2014^b and ESA 2018,^c have concluded there is a huge element of randomness in conference editorial decisions. "The reputation of the peer-review process is tarnished," concluded Hannah Bast. Practically everyone in computing research complains about the "reviewers," but the reviewers are us! If everyone is unhappy, then the problem must be systemic.

Probably the strongest arguments in favor of conferences are their predictability, in terms of submission and decision timing, and their function as venues for community building. But the promised predictability is an illusion, as papers bounce from conference to conferences until they finally find a home. And when every paper has to be published in a conference, conferences are attended mostly by junior researchers, while 20–30 years ago they offered a truly representative sample of their communities, where junior and senior researchers mingled.

A natural reaction to loss is to recreate the familiar. When a conference was "a journal that meets in a hotel," space restriction forced a time restriction, so a conference typically lasts 2–4 days. We kept this tradition in virtual conferences, which still meet for 2–4 days, even though the original reason is gone. But if we have learned anything over these past months it is that spending a day in virtual space is quite difficult and screen fatigue is a real phenomenon. Yet, while we all long for COVID-19 to be over, the climate-change threat is looming larger, and I suspect that virtual conferences are here to stay.

So, we seem to be stuck with a dysfunctional, antiquated publication system. It is time to end the debate about journals and conferences. Let us design a new publication system, something we, as computing professionals, should know how to do. We should collect system requirements, design the system, implement prototypes, experiment, and iterate. The publication system is our system. We are in charge! Technology opens new avenues, but we must be imaginative and not be bound by the dogmas of the dysfunctional past.

If we have learned anything from COVID-19 it is that dealing with major societal challenges requires collective action. The U.S., with its tradition of "rugged individualism" and under meager federal leadership, is handling the pandemic quite poorly. But enabling collective action is exactly why we have established professional societies. They must lead the way.

It is a cliché that everyone wants change, but no one wants to change. Let us collectively agree to change. We deserve a publication system that meets the needs of science, of scientists, and of society. Let us reboot the publication system for computing research. ACM is launching a Presidential Task force on Future Formats for ACM Conferences. It is a start!

Back to Top

Back to Top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.

---

Comments

---

Displaying 1 comment