Reality-Virtuality Continuum
The Reality-Virtuality Continuum is a conceptual framework describing the spectrum of possible environments between the entirely physical world and entirely synthetic virtual environments. Proposed by Paul Milgram and Fumio Kishino in their 1994 paper A Taxonomy of Mixed Reality Visual Displays, it established the vocabulary that the XR field has used ever since to classify and reason about mixed-reality systems.1
The 1994 Paper
Milgram, then at the University of Toronto, and Kishino at ATR Communication Systems Research Laboratories in Kyoto, were motivated by a practical problem: the field lacked a consistent vocabulary for classifying the rapidly multiplying variety of display systems that blended real and virtual content. "Augmented reality," "virtual reality," and "mixed reality" were being used inconsistently across research groups. Their paper proposed a taxonomy to resolve this.1
The core contribution was a single axis — the Reality-Virtuality Continuum — running from Real Environment at one end to Virtual Environment at the other. Between these poles lies what Milgram and Kishino termed Mixed Reality (MR): any display that combines real and virtual content. They subdivided the MR zone into two regions:
- Augmented Reality (AR): primarily real, with virtual content overlaid — the real world is dominant
- Augmented Virtuality (AV): primarily virtual, with real content incorporated — the virtual world is dominant
The paper situated this continuum specifically within the context of visual displays. The authors noted that the full space of mixed-reality environments was at least three-dimensional (extent of world knowledge, extent of presence metaphor, and extent of rendering), but that the single-axis RV continuum was sufficient for classifying display taxonomies.1
The Continuum
Real ←————————————— Mixed Reality ————————————————→ Virtual
Environment Augmented Augmented Virtual
Reality (AR) Virtuality Environment
(AV)
Real Environment is the unmediated physical world — what a person sees without technological intervention.
Augmented Reality adds virtual content to the real world. The real world remains dominant; digital elements are overlaid, anchored, or composited into the physical scene. Systems like HoloLens, ARKit, and the passthrough cameras of modern headsets all sit in the AR zone.
Augmented Virtuality adds real content into a synthetic virtual world. This includes applications such as video conferencing windows embedded in a VR environment, or live camera feeds of a user's hands rendered inside a fully synthetic scene. Many modern video-passthrough headsets in MR mode occupy this zone — the synthetic environment is primary, and the real world is piped in as a feed.
Virtual Environment replaces the real world entirely with a synthetic one. The user receives no unmediated sensory input from the physical environment. Room-scale VR on headsets like the Meta Quest (with passthrough disabled) is the consumer form.
Influence and Extensions
Ronald Azuma's influential 1997 survey A Survey of Augmented Reality added a functional definition of AR to complement Milgram's display taxonomy: AR systems (1) combine real and virtual objects in a real environment, (2) run interactively and in real time, and (3) register virtual and real objects with each other spatially.3 This definition proved more useful for practitioners because it described what AR does rather than merely where it sits on a display spectrum.
Milgram published a follow-up in 1995 extending the taxonomy to additional dimensions — including the extent to which a system models the real world and the degree to which virtual objects are rendered to match real-world lighting — capturing distinctions the single continuum could not.2
Steve Mann proposed Mediated Reality as an alternative framing that treated all perception as mediated by some combination of real-world and computational input, and extended the space to include systems that subtract or modify real-world content rather than only adding virtual content.5 This captured phenomena like filtering out real objects or dimming parts of the visual field, which the AR/AV taxonomy did not accommodate.
A 2019 CHI paper by Speicher, Hall, and Nebeling surveyed how researchers across 68 papers had operationalised the terms AR, VR, and MR, finding widespread inconsistency even among papers that cited Milgram-Kishino — reflecting that the vocabulary, though foundational, has never achieved canonical stability in practice.4
Relevance Today
The Reality-Virtuality Continuum remains the field's most cited conceptual framework. The term "XR" (Extended Reality) itself — which names the full space between the poles — is a direct heir to it. Product categorisations by Apple ("spatial computing"), Meta ("mixed reality"), and Microsoft ("holographic computing") each implicitly occupy a position on the continuum, even when vendors avoid the academic vocabulary.
The framework's limitations are equally well understood: it captures display typology but not interaction modality, sensory breadth, user agency, or degree of world knowledge. A sophisticated AR system with real-time surface reconstruction, occlusion, and relighting sits at the same point on the continuum as a simple screen overlay of a GPS marker — yet the two experiences are radically different. The continuum is a starting point, not a complete description.
See also: History of XR · Presence & Immersion · Passthrough · Microsoft HoloLens · ARKit · OpenXR