Communications of the ACM

Viewpoint

Are CS Conferences (Too) Closed Communities?

By Jordi Cabot, Javier Luis Cánovas Izquierdo, Valerio Cosentino
Communications of the ACM, October 2018, Vol. 61 No. 10, Pages 32-34
10.1145/3209580
Comments

Publication in top conferences is a key factor, albeit controversial,^3,4 in the dissemination of ideas and career promotion in many areas of computer science. Therefore, it is a major goal for every CS researcher. However, many researchers believe publishing in a top conference is something reserved for the established members of the conference community. For newcomers, this is a tough nut to crack. Indeed, when talking with fellow researchers the assumed unspoken truth is always the same: If you are not one of "them," you have no chance to get "in" on your own.

If this were true, it would imply that senior researchers wishing to change fields during their research career may have a difficult time doing so. And the impact would be even more dramatic for junior researchers: they could only access top venues by going together with their supervisor, limiting their options to make a name for themselves—exactly the opposite of what evaluation committees typically require from candidates. Indeed, candidates are supposed to show their ability to propose and develop valid research lines independently of their supervisor, even better if it is in a slightly different research field and hence in a different community.

But is it true that conferences are closed communities? Or is it just a myth spread by those that tried and failed? And if so, how do we change this situation (and do we really need to change it)? Our goal in this Viewpoint is to shed some light on these issues.

Back to Top

Looking at the Data

To assess whether it is actually true that newcomers have a difficult time getting their papers accepted, we have evaluated the number of newcomer papers (research papers where all authors are new to the conference, that is, none of the authors has ever published a paper of any kind in that same conference) in 65 conferences. The list of selected conferences corresponds to the list of international CS conferences in the CORE ranking,^a 2015 edition, Computer Software category, for which we were able to find available data in the DBLP dataset, the well-known online reference for computer science bibliographic information. The choice of CORE as ranking system is based on its widespread use.

We have analyzed the conferences using a seven-year window (that is, an author is considered new to a conference if he or she has not published in that conference in the last seven years). We only count full papers in the main research track (since getting short papers, posters, demos, and so forth is typically easier but it barely counts toward promotion).

Results show that newcomers' papers are indeed scarce. Most conferences (88%) show a percentage of newcomer papers under 40%. This value is significantly lower in top conferences, with a median value of 14%. As specific examples, well-regarded conferences show the following values: ICSE (5%), OOPSLA (13%), ICFP (11%), RE (6%). We may be tempted to quickly dismiss these numbers by attributing the low percentage of newcomers papers to a lack of newcomer submissions. While it is true that CS communities are shrinking (at least based on ACM tables for SIG memberships), which could imply that the "newcomers pool" is smaller, our analysis suggests that newcomer paper submissions represent at least one-third of the total number of submissions.^b

Additionally, for each conference, we have also calculated the number of semi-newcomer papers. A semi-newcomer is a researcher that has never published in the main track but that has published before in other tracks (for example, a demo or a poster). Data indicates publishing a paper as a semi-newcomer is also difficult but slightly easier than doing so as a complete newcomer. If you want to be part of a given community, it seems to pay off to first participate in that community via lesser competitive tracks or collocated satellite events. And the good news is that, unsurprisingly, newcomers have reasonable chances of success to get papers accepted in those satellite events. Our data indicates the percentage of newcomer papers in satellite events is over 30% in most conferences and it frequently goes up to 50% and over. Clearly, satellite events play a positive role in the growth of the community. The full data is available, including all conferences values and the corresponding boxplot distributions based on the conference rankings.^c

Back to Top

Opening Up Conferences

We believe the data confirms CS conferences^d behave as closed communities. Most likely, some readers believe this is exactly how things should be and that newcomers must first learn the community's particular "culture" (in the widest sense of the word, including its topics of interest, preferred research methods, social behavior, vocabulary, and even writing style) either by simply attending the conference or warming-up publishing in satellite events, before being able to get their papers accepted in the main research track.

---

Satellite events play a positive role in the community.

---

We dare to disagree and argue that the situation is getting to a point in which is worth discussing how to change course. The overall presence of newcomers decreases over time.² Besides, increasing travel and economical restrictions make it difficult to follow the (so far) "easier" path to enter the community, for example, many outsider researchers will not get funded to attend a satellite event, preventing them from learning the ropes of that particular community.

While closed communities have indeed some positive aspects (for example, a particular focus, a heritage to build upon, sense of security, and so forth) we believe they are now becoming too closed. In our opinion, a healthier number for conferences would be having at least 25% of newcomer papers in each edition. This would ensure a continuous influx of fresh ideas and new members in the community among other benefits of open communities such as better diversity and inclusiveness. While junior researchers co-authoring a paper with their supervisor for the first time (in fact, the most common path to enter a top conference) could be considered new members as well, we argue that conferences must also make the effort to open up to complete outsiders (including junior researchers trying to start independent research lines in a new field, senior researchers moving to a new research interest, industrial researchers trying to disseminate their results ...) able to bring a completely fresh perspective to the community.

The main challenge in opening up conferences comes from the fact that we do not really know the reasons why these numbers are so low. Do some potential newcomers refrain from submitting in the first place? Do they get rejected more often than established authors? If the latter, are they being fairly rejected because their papers do not follow the right structure, process, or evaluation standards? Or is there a positive (unconscious) bias toward known community members during the review phase?

Narrowing down a root cause—or causes—requires much more conference data to be publicly disclosed for analysis. We hope this is a direction we will follow as a community. In the meantime, we would like to suggest a few ideas we think are worth pursuing and that, most likely, should be combined in order to tackle this multifaceted challenge:

• Open the review process. More and more conferences are adopting a double-blind review model to avoid bias. Its usefulness to avoid author identification seems to be confirmed⁶ but it is probably still fairly easy to spot whether the authors are at least members of the community so bias is not completely out of the question. We could go even further and aim for triple-blind reviews or, alternatively, open reviews (where reviewers sign the reviews and/or reviews are later released publicly).
• Identify and promote research topics with a lower entry barrier for newcomers either because they are new topics, and therefore not many people in the community work on them, or because they require less advanced skills/infrastructure.
• Increasing acceptance rates to have more slots available. This has been proposed as a solution to the randomness of the peer-review system.⁸ We could even decide to reserve a few slots for newcomer papers. Obviously, this goes against the traditional conference publication model and could trigger cascade effects on the role of conferences but there is already a part of the community that challenges the idea that very low acceptance rates are indeed good for us. ICSE'17 conference went to the extreme of limiting the number of papers to be submitted by a single author (restriction dropped in 2018 since the community felt it strongly discouraged collaboration). Given that newcomers typically submit far fewer papers, this could help prevent established researchers filling so many slots. An interesting experience nevertheless worth being reevaluated in the future (even if with different "parameters").
• Adopt more journal-like review systems. Introducing revision cycles in a conference could help newcomers to fix obvious but easy-to-correct mistakes that would otherwise force a paper rejection. Even better, a rolling deadline, allowing submissions all year-round (VLDB-style) would avoid paper acceptance to be decided on the basis of the paper itself and not related to the others in order to avoid over the limit acceptance rates.
• Start mentoring programs where young researchers can pre-submit their work and get some advice (typically from former PC members) before the actual submission. While mentoring may have a limited success in getting the newcomers' papers in immediately, it could have a positive long-lasting effect in speeding up the newcomer learning.
• Draw ideas from other domains where they may face similar problems. For instance, in the open source community, many projects struggle to attract new contributors and have come up with proposals to attract more people.⁷ Examples (adapted to our field) would be to have a dedicated portal for newcomers clearly explaining how papers in the conference are evaluated, showing examples of good papers (in terms of style and structure), listing typical mistakes first submitters do based on the experience of PC members, and so forth. And, importantly, encouraging them to keep trying if they are not initially successful—they may not be aware senior researchers also get many papers rejected.

Despite the number of works analyzing co-authorship graphs, newcomers metrics have been mostly ignored in previous research works. M. Biryukov et al.¹ study individual newcomer authors, B. Vasilescu et al.⁹ and J.L. Cánovas et al.² calculate just a coarse-grained newcomers value as part of a larger set of general metrics. We hope to trigger additional research and, especially, general discussions around the trade-offs of closing/opening up more of our research communities⁵ with this Viewpoint.

We are aware this is a challenging process due to the leadership role many conferences play in our research system. And we acknowledge opening up a conference is, in fact, an act of generosity. Unless we avoid the zero-sum game of the current publication model (with a somehow fixed number of slots to keep acceptance rates low) any explicit action to increase newcomer participation implies decreasing our own chances to get published. Still, we believe the newcomers' problem cannot be swept under the carpet any longer if we want to ensure we keep a vibrant and growing community in our research area.

Back to Top

Back to Top

Back to Top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.