Communications of the ACM

As the computer science field has evolved, so should the methods for disseminating computing research results.

Filippo Menczer

I could not agree more, as I already mentioned in response to Moshe Vardi's editor's letter.

As program chair of an ACM conference I can attest to the many flaws
of the conference review process, in spite of everyone's best efforts.
Time constraints, concentrated load, difficulty of holding a relaxed
rebuttal phase, competition among authors and referees, pressure to go
with average ratings, financial concerns (attendance), and other
factors --- all bias and weaken the process in important ways.

And as an interdisciplinary researcher, I experience first-hand how
our conference-driven publication practices hurt us in terms of
impact, reach, and visibility. Computing journals have very low impact
factors and very long dissemination times, compared to other science
disciplines. Conferences hurt the viability of journals because top
researchers are busy refereeing conference papers. The time
constraints of conferences make us submit papers in a rush before they
are ready, or impose long delays. And competition for the top
conferences, as has been noted by others, means that many good papers
do not get the attention they deserve.

I propose a simple solution: abolish conference proceedings. Then
papers will be submitted to journals instead. Journals will receive
more and better papers. Refereeing resources will shift naturally from
conferences to journals. As a result, journals will gain impact,
improve quality, and speed up their processes. With our full
attention, they will become viable once again and the review process
will be more rigorous, effective and timely. Deadlines will no longer
be concentrated and we can submit better work, revise it until it is
ready, and profit immediately and directly from reviewer's feedback --
the same referee can judge improvements to a paper. I could go on but
it seems the many advantages are obvious.

We would still hold conference, of course. In many cases where
conferences and journals are nicely aligned, presentations can be
selected and invited among the best papers published in the previous
year. For newer areas and groundbreaking work, a conference or
workshop can still accept submissions --- but it should not publish
proceedings; publication is the job of journals.

ACM should take the lead in such a transition because it publishes the
proceedings of most top computing conferences, as well as many of the
top computing journals. Therefore ACM has everything to gain from
leading the way. ACM is also the only body that could successfully
shepherd such an undertaking.

The switch would not be easy, but with careful planning we could
manage a phased transition over a few years and catch up to the rest
of the scientific community.

Filippo Menczer
Program Chair, Hypertext 2009

Barry Jay

I also agree. Recently a journal editor asked me to expand an otherwise acceptable paper since "it could have been published in a conference". As a first step, it would be good if journals were to encourage original submissions, as well as those that have survived conferences.

My own, light-hearted, take on the difficulties of reviewing to a deadline are in "How to reject a paper you don't like, but don't know why" available at www-staff.it.uts.edu.au/~cbj/Publications/paper_rejection.pdf.

Lance Fortnow

Thanks for the support. The discussion continues on my weblog, at http://blog.computationalcomplexity.org/

Daniel Tunkelang

It's nice to see this piece joining a growing chorus questioning the way we conflate the distinct concerns of disseminating knowledge, establishing professional reputation, and building community. This problem is not unique to computer science, but we are certainly in a position to lead by example in addressing it.

In an age where distribution is nearly free, I agree that we should move the filtering role from content publishers to content consumers. There's no economic reason today why scholarship (or purported scholarship) shouldn't be published online. Of course, the ability to publish digital content for free (or close to free) does not imply anyone will (or should) read what you write. The blogosphere offers an instructive example: the overwhelming majority of blogs attract few (if any) readers. I suspect that the same holds true for arXiv.org. Of course, peer-reviewed content may not fare that much better, particularly given the proliferation of peer-reviewed venues. Regardless, it makes no sense for publishers to act as filters in an age of nearly-free digital distribution.

That brings us to the question of how researchers should establish their professional reputation--and, in the case of academics, obtain tenure and promotion. Today, they have to publish in peer-reviewed journals and conferences. Even if we accept the weaknesses of the current peer-review regime, we should be able to separate content assessment from distribution. The peer-review process (and review processes in general) should serve to endorse content--and ideally even to improve it--rather than to filter it.

Finally, conferences should primarily serve to build community. I find the main value of conferences and workshops to be face-to-face interaction, and I've heard many people express similar sentiments. Part of the problem is that so few presenters at conferences invest in (or have the skills for) delivering strong presentations. But more fundamentally it's not even clear that the presentations are the point of a conference--after all, an author's main motive for submitting an article to a conference seems to be getting it into the proceedings.

Here are some questions I'd like to suggest we consider as a community:

What if presentation at a conference were optional, and an author's decision to present had no effect on inclusion in the proceedings? Would there be significantly fewer presentations? Would those fewer presentation be of higher quality?

What if the process of peer-reviewing conference submissions required the submission of presentation materials rather than (or in addition to) a paper? Would the accepted presentations be of higher quality? Would researchers invest more in presentation skills? What would happen to strong researchers without such skills?

Can we update the traditional conference format to foster more productive interaction among researchers? For example, should we have more poster sessions and fewer paper presentations?

I'd love to see the computer science community take the lead in evolving what increasingly feel like dated procedures for disseminating knowledge, establishing professional reputation, and building community. I've tried to do my small part, co-organizing workshops on Human-Computer Interaction and Information Retrieval (HCIR) that emphasize face-to-face interaction and organizing the SIGIR 2009 Industry Track as a series of invited talks and panels from strong presenters. But I'm encouraged to see "establishment" types like Moshe and Lance leading the charge to question the status quo.

David Karger

You've described a problem that is dauntingly large and hard to solve. We can make progress by tackling pieces.

The database community has one piece of the answer: their premier conference, VLDB, is now driven by a journal submission model (http://www.jdmr.org/). Authors submit to the journal, and a year's worth of journal submissions are presented at each year's conference. The above url links to an excellent discussion of and rationale for their procedure.

You also highlight the way the proliferation of conferences has led to a breakdown in their value as tools for networking and drawing the community together. For these, I propose a simple solution: colocation. Take all those little conferences, and hold them all at the same time under one roof. Let them keep their independence, but share their coffeebreaks and banquets. Most importantly, allow anyone paid up at one conference to attend all of them---after all, no matter how many conferences someone "attends" at the same time, they can only consume one human-being's worth of resources. The federated computing conference does this in large, but it would be equally affective for subfields of computer science to bring together all their sub-subfields.

Anonymous

I really like David Karger's suggestion -- this way I have one place to send my students to where they will hear a large collection of interesting talks and hopefully deliver talks on their own research as well. Right now, if a student's paper is rejected from a "top" conference, I have to pay for them to go there to listen to interesting results and also pay for them to go to another conference where they might be presenting a paper.