Communications of the ACM
The ASPLOS 2020 Online Conference Experience
ASPLOS, the ACM Conference on Architectural Support for Programming Languages and Operating Systems, is an annual, ACM-sponsored conference that brings together roughly 300-350 researchers and students from the computer architecture, programming languages, and systems research communities. The 25th ASPLOS was scheduled in Lausanne, Switzerland, on March 16-20, 2020, with two days of tutorials and workshops and three days of dual-track conference sessions. With a 36% increase in paper submissions this year and the resulting record-sized program of 86 papers, we anticipated an increased attendance of 400-500 participants.
Throughout the registration period, the ASPLOS organizers were aware of the COVID19 outbreak in China, in particular because of registration cancellations from participants who were unable to travel because of the COVID19 virus. The initial cancellations came from China, followed by South Korea, and finally Italy. On Friday, February 28, the Swiss government banned all gatherings of 1,000 or more people, forcing the cancellation of the Geneva International Motor Show, a huge event that brings in a sizeable fraction of the hospitality revenue in that city.
On Saturday, February 29, the ASPLOS General Chair and ASPLOS Steering Committee decided to cancel the meeting and move the conference online. We had three reasons for this decision. It was clear at that point that attendance would be poor, as countries and, even more significantly, companies were starting to impose travel restrictions that would prevent attendees from flying. We also believed that it would be likely that the Swiss government would impose stricter restrictions on public gatherings by the time the conference occurred. In fact, on the first scheduled day of the conference, the Swiss government banned gatherings of 10 or more people. And, finally, we strongly felt that cancellation was the responsible course of action, to avoid exposing attendees to the virus and avoid spreading it further.
This document discusses what we did to quickly move the conference online and the responses of the online attendees. We hope that our decisions can help future online conferences provide a better experience for the attendees.
Online Plan
As part of the cancellation, we notified all authors and registered attendees (as well as the broader public through social media and email distribution lists) that we were cancelling the conference and taking the following steps:
1. Attendee registrations fees were refunded. Students who received travel support were assured that they would be reimbursed for their non-refundable expenses.
2. We outlined our plans to move the conference online during the same time and to open it to everyone without charge. Authors were asked to prerecord their talks, and we provided a Slack workspace as the online conference. More details below.
3. The workshops and a co-located symposium (VEE) were given the opportunity to make their own plans and to use the conference workspace if desired.
Our plan for an online ASPLOS was a pragmatic one because of the strict time constraints (two weeks) and a worldwide distribution of authors, some of whom were already under confinement, and all of whom would soon be stressed and upset.
One very positive outcome of opening the conference was that over 1,000 people participated, more than twice the number expected in Lausanne. Several attendees noted that they could not have attended the conference because of financial or employment constraints, and were very pleased to be able to participate remotely.
A key decision was to ask authors to record their presentations in advance, rather than using a videoconferencing system to present them live. We saw two compelling reasons for this choice. First, authors were spread around the world and might not be able to arrange their schedule or find adequate facilities for presenting at an unusual hour. In addition, authors might not have an adequate network to reliably present a live talk. Pre-recording the talks solved both problems. The recorded talks were hosted on the SIGARCH YouTube channel, which made them publicly visible and easily viewed.
We asked authors to record their presentation "picture-in-picture," with a small image of themselves talking superimposed on the presentation itself. Creating videos in this format proved difficult for a number of authors, who did not have adequate software or cameras to make a high-quality presentation. Poor audio quality was also a significant problem.
As discussed below, opinion on these presentations was mixed. Some felt that they were superior to conference talks, while others missed the spontaneity of a live presentation. One session at the conference was broadcast live and was well-received, but four authors were unable to contribute for various reasons. There was a broad consensus that recording talks and making them available for subsequent viewing is beneficial—but live presentations can also be recorded.
The other key decision was to use Slack to host the conference. We considered other alternatives (Microsoft Teams, Discord, Mattermost, etc.) but felt that Slack had the advantages of being familiar to many people, running on all three desktop OSes, and being a hosted service with a generous free tier. The decision proved to be a good one, as Slack provided the necessary functionality and we were able to host the entire conference in a free instance.
The shortcoming of Slack, or at least the way in which we used it, was in the organization of the sessions and talks, which was overwhelming for newcomers and made it difficult to find a topic or specific presentation. Figure 1 shows the Slack layout. The 28 sessions from the program were each assigned a channel, and the presentations from the session were loaded into the channel. Each session had a session chair assigned as a moderator. ASPLOS attendees could view the conference talks at any time, in any order, and they could pose questions to the authors in the appropriate channel. Some channels had lively discussions spread over several days, while other sessions had no interactions. In general, the questions and the authors' answers were better formulated and more complete than questions in a live conference, and the successful session had a lively dialog that illuminated different aspects of a paper. Moreover, all discussions, except those that moved to a private chat, were visible to all attendees.
We also used channels for chair announcements, announcements of awards, diversity and inclusion, employment opportunities, and a "hallway" for informal conversations.
Figure 1. ASPLOS Slack Channel
The principal difficulty with our use of Slack was the long list of sessions (left hand side of Figure 1), which made navigation and discovery of a specific talk difficult. As an alternative interface, we added links to the channels to the online ASPLOS schedule, which provided attendees with a more complete and understandable overview of the program and facilitated discovery of a specific paper or author.
An alternative proposed was to list only a day's worth of sessions at a time, as at a physical conference. However, an enforced timeline meshes poorly with conference attendees' asynchronous interaction with an online conference, and it is challenging to define when a day begins and ends in a 24-time-zone world.
If future conferences decide to use Slack, it would be worth investigating its APIs, which permit modifications to the appearance of the Slack app. A conference-schedule-like front page to the app might alleviate these difficulties.
Another alternative is to restructure the sessions into fewer, general topics. ASPLOS could have supported 6-8 broad topics. Each topic would have its channel, and instead of posting the all talks in a channel at the beginning of the conference, they could be released gradually over the duration of the conference. In this scenario, a group of moderators would be helpful for getting a discussion started and keeping it going. We believe that it would also be helpful to stage a few live discussions over the period of the conference. It would be good to pick times so that people in different regions could participate, say times when the Americas, Europe, and Asia participate, respectively.
Survey Results
We conducted a survey of conference attendees by asking them to fill out a 28-question anonymous survey; 87 attendees (out of 1,028 Slack users) responded to the survey. The survey had four parts: attendee background, ASPLOS 2020 attendance, ASPLOS 2020 experience, suggestions for future online conferences.
Attendees
Of the 87 respondents to the survey, 59% were Ph.D. students, 21% were professors, and the remaining 20% were primarily postdocs (6%), industrial researchers (5%), and software/hardware designers (5%). More than half (56%) of attendees resided in North America, 28% in Europe, and 16% in Asia. However, when asked where they obtained their secondary education, the result was more balanced, with 35% in North America, 30% in Europe, and 32% in Asia. Ninety-three percent of respondents were fluent in English.
When asked, 6% said they had never attended a conference, 48% had attended 1-5, 22% had attended 6-10, and 24% had attended more than 10. Surprisingly, 67% had never attended ASPLOS before, and only 31% had attended 1-5 previous ASPLOS conferences. However, 59% had co-authored a paper submitted to any ASPLOS conference.
Attendance
Figure 2 reports how many talks a participant viewed. The majority viewed 1-5 presentations, with a significant fraction viewing 6-10 presentations, and only 14% viewing more than 11 talks. We are unaware of similar statistics for live conferences, but it appears that attendees this year focused on a small number of topics close to their research interests. We hypothesize that this effect can be explained by the fact that participation did not involve travel and therefore did not require full-time dedication to ASPLOS. It is also possible that attendees were juggling attending the conference with other activities prompted by the COVID19 pandemic.
Figure 2. Conference "participation."
We also asked attendees to compare the online talks to presentations at a live conference. Figure 3 reports the results (1 is better, 5 is worse). Forty-one percent felt that the online talks were better, 33% that they were similar, and 25% that they were worse than conventional presentations. Respondents particularly liked being able to view talks on their own schedule and to pause, jump around, replay, and accelerate a talk. The primary shortcoming noted was the low audio quality of some talks (video was not cited as a problem). Several attendees asked for additional shorter videos, such as the 1-minute highlight videos used at some conferences, so that they could take a quick look at a presentation. Several people suggested that recorded videos might allow more diverse and innovative presentations, including, for example, demonstrations.
Figure 3. Evaluation of ASPLOS talks (1 is better, 5 is worse).
Experience
We asked an open-ended question about what was missing from the conference. The responses focused on three points. Many responses asked for live ("Zoom-like") presentations, even while recognizing the difficulty of bridging time zones. Other attendees bemoaned the lack of personal interaction and commented that discussions in the channels were not as rich and enjoyable as person-to-person interactions at a conference. Finally, there were complaints that the Slack interface made it difficult to navigate the content.
Figure 4. Conference experience (1 is Enjoyed, 5 is Did not enjoy)
Despite these difficulties, attendees reported that they enjoyed "attending" ASPLOS (Figure 4), with 57% positive, 22% neutral, and 21% negative. The positive aspects cited were the ability to attend a conference that would normally not be accessible, the ability to view presentations at a time and place of one's choosing, and the ability to pose questions and see other peoples' discussion. The negative aspects were a lack of the excitement and intensity of a conference, the sparsity of comments and interaction on some channels, the lack of opportunities to network with other people, and a missed opportunity to travel. In general, attendees rated the online conference similar to an actual conference (Figure 5). When asked what was a "pleasant surprise" about the online conference, attendees cited the ability to attend the conference at no cost, the ability to view and review talks, and the live broadcast of the WACI (Wild and Crazy Ideas) session.
Figure 5 Comparison with typical conference (1 is Better, 5 is Worse).
Suggestions for Improvement
When asked if there should be more online conferences in the future (Figure 6), the response was strongly positive. Only 21% of responses were opposed to this change.
We asked for suggestions for organizers of future online conference and received many helpful replies:
- Organize the presentations better, with a smaller number of channels.
- Have a mixture of "live" sessions and pre-recorded talks.
- Increase moderation of channels and encourage moderators to act to increase interaction among participants.
- Find ways to increase engagement among participants, both the chatter that occurs at conferences as well as the excitement and interactions.
- Introduce a sense of "pace" into the experience so that that attendees focus on a small number of areas/topics/talks at a time.
- Keep it free.
We also asked which aspects from the online conference should be adopted by real-world conferences and received strong support for providing video recordings of the talks, as well as Slack channels for discussions.
Figure 6. Opinion of online conferences (1 is Definitely yes, 5 is Definitely no).
Overall, the conference organizers are pleased with the outcome of the conference. While it clearly does not compare to the social experience of spending three or more days with colleagues, ASPLOS 2020 did provide authors with an opportunity to present their work to a larger-than-normal community, as well as an opportunity to pose and answer questions about research.
As more conferences move online, both because of the COVID travel restrictions and concerns about excessive travel, it appears possible to develop a new model of technical information exchange that retains many of the benefits of our familiar one, while at the same time becoming more inclusive and welcoming to a larger technical community. Hopefully ASPLOS's experience is a first step on this path.
Full survey results are available at: https://docs.google.com/spreadsheets/d/1LMjKjy0t8vtEWNZgG3CVZv7VTBPw8JJqtjZ2Idd55tU/edit#gid=1961410688
James Larus is dean of the school of computer and communication sciences at École Polytechnique Fédérale de Lausanne (EPFL), Switzerland; he was general chair of ASPLOS 2020. Luis Ceze is a professor on the Computer science and engineering faculty of the Paul G. Allen School of Computer Science & Engineering of the University of Washington, and was a program chair for ASPLOS 2020. Karin Strauss is a principal research ranager for Microsoft Research, and an affiliate professor in the department of computer science and engineering at the University of Washington; she served as a program chair for ASPLOS2020.
No entries found