Communications of the ACM

Opinion

It Is Time to Let Go of 'Virtual Reality'

By Rick Skarbez, Missie Smith, Mary Whitton
Communications of the ACM, October 2023, Vol. 66 No. 10, Pages 41-43
10.1145/3590959
Comments

It seems as if the list of terms used to describe immersive experiences—here we use "immersive" very broadly to refer to computer systems that do not use a traditional 2D display plus keyboard and mouse interface—multiplies daily. At the IEEE Virtual Reality conference held in March 2023, one needed only to look at the titles of the workshops to see a representative sample: The terms virtual reality (VR), augmented reality (AR), eXtended reality (XR), inter-reality, immersive environments, and metaverse all appeared. (We acknowledge that the "metaverse" generally refers to some or all of a cluster of technologies—not only VR/AR, but networked sociality, interoperability, NFTs, and more—but note it normally assumes the use of AR/VR for presentation.) The workshops at the 2022 ISMAR Symposium added mixed reality (MR) and cross-reality (CR). This proliferation of terms is not without consequences: Researchers may be unable to find relevant research, their own research may have limited impact, or there may be disagreement over what is or is not in scope for a given publication venue; practitioners may not be able to find relevant research and may not be able to market their products or solutions effectively; the average end user is left to scratch their head in confusion and, too often, throw their hands up in frustration. These harmful consequences undercut the progress and potential of this research community and contribute to the simmering "VR is dead" discourse.

---

Augmented reality—and virtual reality—are subtypes of mixed reality.

---

We believe there is a way out of this unfortunate situation. As it stands, there is a large and increasing number of competing and overlapping fiefdoms. We propose to unite them under one organizing principle, to the benefit of all. In Skarbez et al., we argued the term "mixed reality" should include all of what has traditionally been carved out as "virtual reality."³ This column seems to us the natural consequence of the argument begun in the previous work. Note that while we specifically argue for the utility of mixed reality as the organizing and unifying concept, we believe the principle—the field would be better served if we could harmonize our discordant voices—is more important than "winning" the terminology war.

Back to Top

All Technology-Mediated Realities Are Mixed Reality

The term "mixed reality" dates to 1994, when Paul Milgram and Fumio Kishino introduced the reality-virtuality continuum to the literature in their landmark paper "A Taxonomy of Mixed Reality Visual Displays."² In that paper, Milgram and Kishino proposed that mixed reality is the area of the reality-virtuality continuum between—not including—unmediated "real reality" and entirely mediated "virtual reality." In the traditional formulation, mixed reality comprises augmented reality as well as augmented virtuality.

In our 2021 article, "Revisiting Milgram and Kishino's Reality-Virtuality Continuum," we argued everything traditionally called virtual reality—and so excluded from the mixed reality portion of the continuum—should rightly be included under the heading mixed reality.³ To summarize our argument, we believe it is impossible to avoid conflict between sensory information originating in the real environment and sensory information originating in a/the virtual environment with anything short of "Matrix-like VR," which is to say a "perfect"—and entirely hypothetical—brain-computer interface. Every so-called "virtual reality" experience is actually a mixture of virtual (most commonly computer-generated visual and auditory stimuli, but other sensory modalities are also employed) and real (for example, the sensation of the floor under one's feet, the feeling that gravity is down), and hence, mixed reality.

A similar argument can be made that augmented reality is also subsumed by mixed reality. (This view was already held by Milgram and Kishino,² and is, we believe, the view held by the majority of researchers in the augmented reality space.)

Back to Top

Statement of Our Position

We have previously argued to redefine mixed reality as, "[an] environment … in which real-world and virtual-world objects and stimuli are presented together within a single percept. That is, when a user simultaneously perceives both real and virtual content, including across different senses, that user is experiencing mixed reality."³ Note that we mean mixed reality spans not only the AR and VR spaces, but also any technology-mediated experience that presents virtual objects to the user. This may be reasonably construed to contain, for example, mobile AR, visualization, virtual human applications, and even computer games. We do not regard this as a problem, but rather a feature, for reasons we explain here.

All other "x-reality" terminology should be phased out in published work in favor of "mixed reality." The varied verbiage discussed earlier creates confusing, arbitrary, and often false distinctions among experiences, technologies, research topics, research methods, and research communities. If these currently divided communities can adopt shared terminology, communication among the communities will, we hope, become easier, clearer, and, as a consequence, more frequent.

A standardized language for the description of mixed reality systems should be developed and adopted. We acknowledge the fact that the existing terms do have some descriptive power, despite the challenges. For example, it can generally—but not always—be assumed that a technology described as "virtual reality" seeks to replace sensory input originating in the real world with computer-generated sensory input delivered via immersive displays, while a technology described as "augmented reality" seeks to supplement the real world with virtual content.¹ We believe such descriptive shorthand can be preserved and even enhanced with careful consideration.

---

The existing terms do have some descriptive power, but this loss can be overcome with better reporting guidelines.

---

To this end, we do not now recommend specific guidelines, as this would likely only further muddy the terminological waters. We instead propose that the research community should develop and adopt reporting guidelines for mixed reality research, inspired by the PRISMA guidelines. Specific items to be considered should include, at a minimum, which senses are receiving computer-generated stimuli and which are not, whether those senses are isolated from the real world—as in VR—or able to perceive the real world—as in AR, the specific operating characteristics of each of the sensory displays, and the interaction techniques supported by the system.

Back to Top

Practical Implications of These Recommendations

Here, we attempt to envision the effects these changes will have on the broader research community. While we believe the outcomes will largely be positive, we acknowledge there are some potential negative outcomes. We hope these can be mitigated by appropriate leadership and guidance; for example, the previously mentioned reporting guidelines for mixed reality research.

Potential positive outcomes. The first positive outcome that we foresee is an increase in effective communication among researchers, practitioners, and end users. As it stands, a researcher or developer might have to search over a half-dozen relevant terms to find all the important work on a given topic. With the proposed changes, a search for "mixed reality" would return all the potentially relevant work, which could then be narrowed down based on clearly reported data organized via the reporting guidelines that further classify the system, experience, or experiment.

A second positive outcome might be these changes in terminology would enable journals, conferences, and other outlets to reorganize along more principled lines. Referring again to the IEEE VR 2023 conference, papers sessions were organized around topics as disparate as the engineering and construction of mixed reality displays, research studies on human perceptual abilities, and the use of computer vision techniques in mixed reality applications. It can be difficult to draw a line around "VR research" without encircling nearly the whole of computing and incorporating psychology and human factors engineering to boot. While the same would be true of "MR research," this could again be remedied with improved reporting guidelines.

---

It can be difficult to draw a line around "VR research" without encircling nearly the whole of computing and incorporating psychology and human factors engineering.

---

A final benefit would be the elimination of arbitrariness in deciding what is, or is not, in scope for a given venue or topic. As it stands, there is a great deal of latitude to argue whether or not a given system, topic, or project "is VR": Virtual human research? Immersive analytics? Computer vision? Machine learning? We propose that "mixed reality" could provide a home for all of these research topics, which could then be evaluated fairly on their merits.

Potential negative outcomes. The most significant negative outcome we foresee is the loss of the previously mentioned useful shorthands. While distinguishing AR from VR is clearly imperfect, those categorizations do provide some information very efficiently. We believe that improved guidance with respect to reporting MR research is an absolute requirement to overcome this loss of descriptive power.

A second problem is, simply, that change is often awkward. This is especially true for participants who are new to the space. (As in, "I just learned about VR, and now you are telling me it is the wrong name!") Unfortunately, this awkwardness is unavoidable. However, the status quo is also awkward: To have to learn the distinctions between AR, VR, MR, XR, and others. Until a term wins out, there is bound to be confusion and awkwardness. That said, this was a motivating factor in our recommendation of mixed reality as the umbrella term. It is already known to the community, and has a (relatively) well-accepted definition.

Our perspective. A note at this point: All of the authors identify as Caucasian, all of us share an engineering background, and all of us have spent significant time doing MR research in academia, industry, or both. As such, we likely share some biases in how we think about and approach this work. That said, our desire is to create a more welcoming environment for researchers and practitioners, not less. The terminological silos currently in place create opportunities for gatekeeping: As in, "Is this really VR [or AR, or metaverse … ] research?" With a bigger tent, more—and more types of—work could be included.

Back to Top

Conclusion

We began with a provocative declaration: It is time to let go of "virtual reality." We hope it is now clear that we only mean the term virtual reality—as well as the terms augmented reality, extended reality, and their kin. These terms give the illusion of specificity, but closer reading of many papers using these terms shows conflation and disagreements between terminology in different papers. Even we (the authors) have changed our perspective on the appropriate terminology over time. Regardless of the exact language that may be adopted, standardized reporting guidelines will push the field toward more rigorous science and more effective communication. Nomenclatural confusion has dogged our field for far too long. "Virtual reality" must die so virtual reality can live.

Back to Top

Back to Top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.

---

No entries found