Publication in top conferences is a key factor, albeit controversial,3,4 in the dissemination of ideas and career promotion in many areas of computer science. Therefore, it is a major goal for every CS researcher. However, many researchers believe publishing in a top conference is something reserved for the established members of the conference community. For newcomers, this is a tough nut to crack. Indeed, when talking with fellow researchers the assumed unspoken truth is always the same: If you are not one of "them," you have no chance to get "in" on your own.
If this were true, it would imply that senior researchers wishing to change fields during their research career may have a difficult time doing so. And the impact would be even more dramatic for junior researchers: they could only access top venues by going together with their supervisor, limiting their options to make a name for themselves—exactly the opposite of what evaluation committees typically require from candidates. Indeed, candidates are supposed to show their ability to propose and develop valid research lines independently of their supervisor, even better if it is in a slightly different research field and hence in a different community.
But is it true that conferences are closed communities? Or is it just a myth spread by those that tried and failed? And if so, how do we change this situation (and do we really need to change it)? Our goal in this Viewpoint is to shed some light on these issues.
To assess whether it is actually true that newcomers have a difficult time getting their papers accepted, we have evaluated the number of newcomer papers (research papers where all authors are new to the conference, that is, none of the authors has ever published a paper of any kind in that same conference) in 65 conferences. The list of selected conferences corresponds to the list of international CS conferences in the CORE ranking,a 2015 edition, Computer Software category, for which we were able to find available data in the DBLP dataset, the well-known online reference for computer science bibliographic information. The choice of CORE as ranking system is based on its widespread use.
We have analyzed the conferences using a seven-year window (that is, an author is considered new to a conference if he or she has not published in that conference in the last seven years). We only count full papers in the main research track (since getting short papers, posters, demos, and so forth is typically easier but it barely counts toward promotion).
Results show that newcomers' papers are indeed scarce. Most conferences (88%) show a percentage of newcomer papers under 40%. This value is significantly lower in top conferences, with a median value of 14%. As specific examples, well-regarded conferences show the following values: ICSE (5%), OOPSLA (13%), ICFP (11%), RE (6%). We may be tempted to quickly dismiss these numbers by attributing the low percentage of newcomers papers to a lack of newcomer submissions. While it is true that CS communities are shrinking (at least based on ACM tables for SIG memberships), which could imply that the "newcomers pool" is smaller, our analysis suggests that newcomer paper submissions represent at least one-third of the total number of submissions.b
Additionally, for each conference, we have also calculated the number of semi-newcomer papers. A semi-newcomer is a researcher that has never published in the main track but that has published before in other tracks (for example, a demo or a poster). Data indicates publishing a paper as a semi-newcomer is also difficult but slightly easier than doing so as a complete newcomer. If you want to be part of a given community, it seems to pay off to first participate in that community via lesser competitive tracks or collocated satellite events. And the good news is that, unsurprisingly, newcomers have reasonable chances of success to get papers accepted in those satellite events. Our data indicates the percentage of newcomer papers in satellite events is over 30% in most conferences and it frequently goes up to 50% and over. Clearly, satellite events play a positive role in the growth of the community. The full data is available, including all conferences values and the corresponding boxplot distributions based on the conference rankings.c
We believe the data confirms CS conferencesd behave as closed communities. Most likely, some readers believe this is exactly how things should be and that newcomers must first learn the community's particular "culture" (in the widest sense of the word, including its topics of interest, preferred research methods, social behavior, vocabulary, and even writing style) either by simply attending the conference or warming-up publishing in satellite events, before being able to get their papers accepted in the main research track.
Satellite events play a positive role in the community.
We dare to disagree and argue that the situation is getting to a point in which is worth discussing how to change course. The overall presence of newcomers decreases over time.2 Besides, increasing travel and economical restrictions make it difficult to follow the (so far) "easier" path to enter the community, for example, many outsider researchers will not get funded to attend a satellite event, preventing them from learning the ropes of that particular community.
While closed communities have indeed some positive aspects (for example, a particular focus, a heritage to build upon, sense of security, and so forth) we believe they are now becoming too closed. In our opinion, a healthier number for conferences would be having at least 25% of newcomer papers in each edition. This would ensure a continuous influx of fresh ideas and new members in the community among other benefits of open communities such as better diversity and inclusiveness. While junior researchers co-authoring a paper with their supervisor for the first time (in fact, the most common path to enter a top conference) could be considered new members as well, we argue that conferences must also make the effort to open up to complete outsiders (including junior researchers trying to start independent research lines in a new field, senior researchers moving to a new research interest, industrial researchers trying to disseminate their results ...) able to bring a completely fresh perspective to the community.
The main challenge in opening up conferences comes from the fact that we do not really know the reasons why these numbers are so low. Do some potential newcomers refrain from submitting in the first place? Do they get rejected more often than established authors? If the latter, are they being fairly rejected because their papers do not follow the right structure, process, or evaluation standards? Or is there a positive (unconscious) bias toward known community members during the review phase?
Narrowing down a root cause—or causes—requires much more conference data to be publicly disclosed for analysis. We hope this is a direction we will follow as a community. In the meantime, we would like to suggest a few ideas we think are worth pursuing and that, most likely, should be combined in order to tackle this multifaceted challenge:
Despite the number of works analyzing co-authorship graphs, newcomers metrics have been mostly ignored in previous research works. M. Biryukov et al.1 study individual newcomer authors, B. Vasilescu et al.9 and J.L. Cánovas et al.2 calculate just a coarse-grained newcomers value as part of a larger set of general metrics. We hope to trigger additional research and, especially, general discussions around the trade-offs of closing/opening up more of our research communities5 with this Viewpoint.
We are aware this is a challenging process due to the leadership role many conferences play in our research system. And we acknowledge opening up a conference is, in fact, an act of generosity. Unless we avoid the zero-sum game of the current publication model (with a somehow fixed number of slots to keep acceptance rates low) any explicit action to increase newcomer participation implies decreasing our own chances to get published. Still, we believe the newcomers' problem cannot be swept under the carpet any longer if we want to ensure we keep a vibrant and growing community in our research area.
1. Biryukov, M. and Dong, C. Analysis of computer science communities based on DBLP. Lecture Notes in Computer Science (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 6273 LNCS (2010), 228–235.
2. Cánovas Izquierdo, J.L., Cosentino, V., and Cabot, J. Analysis of co-authorship graphs of CORE-ranked software conferences. Scientometrics 109, 3 (Dec. 2016), 1665–1693.
3. Franceschet, M. The role of conference publications in CS. Commun. ACM 53, 12 (Dec. 2010), 129.
4. Freyne, J. et al. Relative status of journal and conference publications in computer science. Commun. ACM 53, 11 (Nov. 2010), 124.
5. Gebert, D. and Boerner, S. The open and the closed corporation as conflicting forms of organization. J. Appl. Behav. Sci. 35, 3 (Sept. 1999), 341–359.
6. Le Goues, C. et al. Effectiveness of anonymization in double-blind review. Commun. ACM 61, 6 (June 2018), 30–33.
7. Steinmacher, I. et al. A systematic literature review on the barriers faced by newcomers to open source software projects. Inf. Softw. Technol. 59 (2015), 67–85.
8. Vardi, M.Y. Divination by program committee. Commun. ACM 60, 9 (Aug. 2017), 7.
9. Vasilescu, B. et al. How healthy are software engineering conferences? Sci. Comput. Program. 89, PART C (2014), 251–272.
b. This calculation requires access to the set of papers submitted and rejected. Since this data is not publicly available, this analysis was only done on the four conferences for which one of the authors acted as PC-Chair.
d. At least in the subarea we have evaluated (computer software category) but we believe results can be generalized to other areas.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.