History of Extended Reality
The history of extended reality spans more than sixty years, from analog mechanical simulators through to neural networks that reconstruct geometry from a single photograph. It is a history of ideas consistently ahead of the hardware required to realize them, punctuated by moments when engineering caught up — and the field lurched forward.
1962–1987: The Visionaries
The modern concept of immersive synthetic experience predates digital computers. In 1962, cinematographer Morton Heilig built the Sensorama, a coin-operated arcade cabinet that delivered a pre-recorded motorcycle ride through Brooklyn using stereoscopic film, binaural audio, vibrating seat, wind, and even scent.2 It was a closed, passive experience — no real-time computation, no interaction — but it established that multi-sensory immersion was possible and worth pursuing.
The computational foundation arrived in 1968 when Ivan Sutherland and his student Bob Sproull built the Sword of Damocles at Harvard and the University of Utah.1 It was the first head-mounted display linked to a computer: a mechanical contraption so heavy it had to be suspended from the ceiling, which rendered simple wireframe rooms that updated as the user moved. The display was crude by any modern standard, but Sutherland had already written the paper — The Ultimate Display (1965) — that described what such a device should eventually become: a room in which the computer controls the existence of matter, where a chair is solid enough to sit on.
Through the 1970s and 1980s, the research community built the field's vocabulary. Myron Krueger's Videoplace (1974–1985) was an environment-scale interactive system that tracked users' silhouettes in real time and let them interact with projected graphics and remote participants — an early argument that XR did not require a headset.24 In 1985, Jaron Lanier and Thomas Zimmerman co-founded VPL Research, the first company to sell commercial VR products: the DataGlove, the EyePhone HMD, and the RB2 (Reality Built for Two) multi-person system.3 Lanier coined the term virtual reality and articulated the philosophical stakes of the technology in ways that still echo in the field.
1991–2001: Theory, CAVE, and Early AR
The early 1990s produced two foundational systems that remain in use conceptually today.
In 1992, Carolina Cruz-Neira, Daniel Sandin, and Thomas DeFanti at the University of Illinois at Chicago unveiled the CAVE (Cave Automatic Virtual Environment) at SIGGRAPH.5 Rather than a headset, the CAVE projected stereoscopic imagery onto the walls, floor, and ceiling of a room-sized cube, tracked the primary user's head, and rendered perspective-correct views for everyone inside. It became the dominant installation format for scientific visualization and industrial VR through the 2000s.
That same year, Louis Rosenberg at the U.S. Air Force Research Laboratory developed Virtual Fixtures — the first functional augmented reality system.6 The system overlaid computer-generated imagery onto a teleoperator's view of a real robot arm, using the overlay to constrain and guide movement. Rosenberg demonstrated that AR overlays could improve human performance in physical tasks, a result that would drive decades of industrial AR research.
The theoretical framework that unified the field came in 1994, when Paul Milgram and Fumio Kishino published A Taxonomy of Mixed Reality Visual Displays.4 Their Reality-Virtuality Continuum described a spectrum from unaugmented physical reality at one end to fully synthetic virtual reality at the other, with augmented reality and augmented virtuality occupying the mixed-reality middle ground. Nearly every subsequent categorization of XR technologies has referenced or extended this model.
Also in 1994, Sega deployed the VR-1 — a large-scale VR arcade attraction that stands as one of the most technically accomplished immersive systems of the decade.26 Developed collaboratively by Sega's AM3, AM4, and AM5 divisions alongside British VR company Virtuality, and built on the AS-1 motion simulator platform jointly created with filmmaker and special-effects pioneer Douglas Trumbull, the VR-1 opened at Yokohama Joypolis on July 20, 1994 before expanding to SegaWorld London and Sega World Sydney.27 Its centrepiece was the Mega Visor Display (MVD) — a 640-gram head-mounted display with a 60° horizontal field of view, 756×244 pixel resolution, and head tracking that updated a 360° view of the virtual environment in real time, driven by Sega's Model 1 and Model 2 arcade hardware. Up to 32 riders across four hydraulic motion pods experienced a synchronized first-person rail shooter while the platform physically tilted and elevated beneath them. Where most 1990s VR systems were academic or industrial, the VR-1 demonstrated that serious immersive hardware could be deployed commercially at scale — and be well-received by the public.27
The first commercial consumer home attempt at VR arrived and failed spectacularly. Nintendo's Virtual Boy (1995) used a parallax barrier display and a monochrome red LED array to produce stereoscopic images in a tabletop viewer.8 It caused headaches, sold fewer than 800,000 units, and was discontinued within a year — a cautionary lesson about shipping before the technology was ready that the industry would repeat.
Also in 1999, Hirokazu Kato released ARToolKit, a free C library for marker-based augmented reality.7 By detecting printed square markers with a standard webcam, ARToolKit let developers overlay 3D content onto video in real time without any specialized hardware. It became the de facto platform for AR research and education through the 2000s and remains influential.
2002–2009: Mobile Sensors and the Road to Depth
The mid-2000s brought the components that would make AR practical at consumer scale, though no one had assembled them yet. Smartphones arrived with accelerometers (2006), GPS, and eventually gyroscopes and compass sensors — enough to anchor digital content to physical orientation. Early mobile AR browsers like Wikitude (2008) and Layar (2009) overlaid information onto the phone camera feed by combining GPS location with device orientation. The results were coarse, but the concept of a world-anchored digital layer accessible to anyone with a phone was established.
In parallel, the depth-sensing technology that would transform XR was being built. PrimeSense, founded in Tel Aviv in 2005, developed a structured light depth sensor on a chip: a projector emitted an infrared dot pattern; a receiver camera captured the deformation of that pattern across scene geometry; the chip computed depth in real time from the distortion.11 Structured light had existed in industrial machine vision for years, but PrimeSense made it small and cheap enough for consumer electronics.
2010–2013: The Kinect Revolution
On November 4, 2010, Microsoft launched Kinect for the Xbox 360 at a retail price of $149.9 Under the hood was PrimeSense's structured light depth sensor — the same technology — packaged with a color camera, a four-microphone array, and Microsoft's skeleton tracking software. For the first time, a living room device could see the human body in three dimensions: not just pixels, but joints, limbs, and poses, updated at 30 frames per second.11
Within 60 days Kinect had sold 8 million units. The Guinness World Records certified it as the fastest-selling consumer electronics device in history.10 The SDK release for Windows in 2012 opened the technology to developers and researchers, triggering an explosion of experiments in gesture interfaces, robotics, medical imaging, and spatial scanning that had nothing to do with games.
Kinect 2, released with the Xbox One in 2013, replaced structured light with time-of-flight (ToF) depth sensing. Where structured light inferred depth from pattern deformation, ToF directly measured the round-trip travel time of emitted infrared pulses — each pixel in the depth image represented a measured distance rather than a calculated one. The result was higher accuracy at greater range and better performance in lit environments.
In 2013, Occipital launched the Structure Sensor on Kickstarter — a depth sensor designed to clip onto an iPad.12 Where Kinect was tethered to a console, Structure was portable and developer-friendly, aimed squarely at room scanning, 3D reconstruction, and mobile AR. It brought depth sensing into applications — surgical navigation, robotics, object capture — that a game peripheral could not reach.
2013–2016: The VR Renaissance and Arrival of Mixed Reality
In August 2012, 19-year-old Palmer Luckey raised $2.4 million on Kickstarter for the Oculus Rift, a wide-field-of-view VR headset that shipped developer kits in 2013.13 The campaign reframed VR as a consumer technology after years of it being confined to research labs and military training systems. Facebook acquired Oculus for $2 billion in 2014, signaling that the largest social platform on earth believed presence was the next computing interface.
Valve contributed room-scale VR and the concept of inside-out positional tracking — tracking the headset's position in space using cameras on the headset itself, rather than external base stations (though early Vive used external beacons). The HTC Vive shipped in April 2016 alongside the consumer Oculus Rift, and within months room-scale VR was the standard for high-end headsets.
On January 21, 2015, Microsoft announced HoloLens at the Windows 10 event.14 Where VR replaced the world, HoloLens projected holographic content onto see-through waveguide optics, anchoring virtual objects to real space. The device combined a depth camera, four environment-understanding cameras, an inertial measurement unit, and a dedicated holographic processing unit running spatial mapping, hand gesture recognition, and gaze tracking simultaneously — all on a self-contained, untethered headset. The Developer Edition shipped in March 2016 at $3,000, the first device to deliver practical mixed reality outside a laboratory.
Pokémon GO, released in July 2016, demonstrated AR at a scale no lab could simulate: 500 million downloads in two months, overlaying creatures onto the camera view of hundreds of millions of phones simultaneously.15 It proved the mass-market viability of AR as an entertainment medium, even in its simplest markerless form.
Location-Based VR: The Venue as Interface
The same 2015–2017 window that produced room-scale VR and ARKit saw the emergence of a distinct industry that drew a direct line from Sega's VR-1: location-based VR entertainment (LBV). Where consumer headsets asked users to clear a living room, LBV operators built dedicated physical venues — converted warehouses, mall units, theater lobbies — engineered from the ground up so that every real surface, prop, and physical sensation corresponded to something in the virtual world. The insight was the same one Sega had demonstrated in 1994: immersive VR works best when the physical environment is designed to complement the virtual one, and that blurring the line between reality and simulation requires controlling both sides of it.
The VOID, founded in 2015 in Pleasant Grove, Utah, called its approach "hyper reality." Players wore backpack PCs and HMDs while moving through physical sets precisely aligned to virtual geometry — a real wall corresponded to a virtual wall, a real heat source corresponded to virtual fire. Haptic vests delivered vibration and pressure feedback. The VOID partnered with Disney and Lucasfilm on Star Wars: Secrets of the Empire in 2017, deploying at Downtown Disney, the Oculus at the World Trade Center in New York, and Westfield centres internationally.30 The company's economics collapsed during the COVID-19 pandemic in 2020, when Disney severed its licensing agreement and locations shuttered.30
Zero Latency, founded in Melbourne, Australia in 2015, pioneered warehouse-scale free-roaming VR — players navigated open floors of up to 400 square metres with no physical constraints, tracked by ceiling-mounted cameras.32 The format proved durable; Zero Latency expanded to over 50 venues across 20 countries.
Nomadic, founded in San Rafael, California in 2015 by veterans of Disney, Industrial Light & Magic, and the gaming industry, approached the problem as a modular content platform.29 Rather than building bespoke venues, Nomadic designed reconfigurable set pieces and haptic props that could be installed in existing mall units or cinema lobbies — lowering the capital barrier for operators while keeping the physical-virtual alignment that made the medium work. Participants wore backpack PCs and headsets, interacting with physical objects that matched their virtual counterparts. Nomadic raised $6 million in seed funding in June 2017 from Horizons Ventures, Maveron, Presence Capital, and Vulcan Capital.28
Sandbox VR, launched in Hong Kong in 2017, brought full-body motion capture suits to multiplayer LBV — every player's avatar reproduced their actual movements — and attracted a $68 million Series A from Andreessen Horowitz in 2019.31 The company survived the pandemic and expanded via franchise agreements into Europe and North America through 2023.
The LBV wave demonstrated both the potential and the structural vulnerability of venue-based XR: the experiences were genuinely compelling in ways that living-room headsets could not match, but the fixed-location model was fragile against disruptions to foot traffic. The COVID-19 pandemic closed or permanently shuttered a significant portion of the industry. Those that survived — Sandbox VR, Zero Latency — tended to be the operators with the most flexible real estate footprints and the most replayable content libraries.
Apple released ARKit in 2017 as part of iOS 11 — a framework that used the iPhone's camera, IMU, and Apple's visual-inertial odometry to track flat surfaces and place virtual objects on them.16 Overnight, hundreds of millions of existing iPhones became capable AR platforms. Google shipped ARCore in 2018 with equivalent capabilities for Android. Between them, ARKit and ARCore established the mobile phone as the dominant AR delivery platform.
HoloLens 2 arrived in February 2019, replacing the first generation's air tap gesture with full articulated hand tracking — the device could see and interpret individual finger joints — alongside eye tracking for foveated rendering and a significantly wider holographic field of view.25 OpenXR 1.0, the Khronos Group's cross-vendor standard for XR runtime APIs, was published the same year, beginning the long process of unifying a fragmented ecosystem.17
In the creative sphere, John Gaeta — the VFX supervisor who won the Academy Award for The Matrix (1999) — had moved into immersive media, serving as Chief Creative Officer at Magic Leap and later co-founding WEVR, where he applied cinematic language to volumetric and VR experiences. His work foregrounded the question of how human presence and spatial storytelling translated across the reality-virtuality spectrum.
2020–2022: The End of Specialized Depth Hardware
The decade of structured light and time-of-flight sensors represented a necessary stage — active depth sensing was the only way to get reliable geometry from the real world at interactive rates. But it imposed hard constraints: power consumption, cost, form factor, and outdoor usability all suffered. The shift that followed made most of those constraints irrelevant.
Neural Radiance Fields (NeRF), published by Mildenhall, Tancik, Barron, Ramamoorthi, and Ng at ECCV 2020, demonstrated that a neural network trained on a set of 2D photographs of a scene could learn its complete 3D volumetric representation — enough to render photorealistic novel viewpoints that had never been photographed.18 NeRF did not require depth sensors; it inferred geometry entirely from color images and known camera poses. The quality far exceeded anything structured light or ToF could produce for appearance-accurate scene capture.
MiDaS (Mixed Dataset Depth Estimation), developed at Intel Labs by Ranftl, Lasinger, Hafner, Schindler, and Koltun, attacked a different part of the problem: estimating dense depth from a single image, with no camera motion, no stereo pair, no active illumination.19 By training on a diverse mixture of datasets, MiDaS learned generalizable monocular depth estimation that transferred to images it had never seen. This had been a hard problem for decades — a single image is geometrically ambiguous — but large-scale neural training finally made it tractable.
The Meta Quest 2 (2020) brought inside-out tracking at $299, using four fisheye cameras and a neural tracking pipeline to maintain six-degrees-of-freedom position without any external hardware. Apple's iPhone 12 Pro (2020) added a LiDAR scanner — a solid-state ToF array — enabling ARKit's Scene Geometry API to reconstruct room meshes in real time on a phone, another step toward making the physical world directly machine-readable.
2023–Present: ML Depth and Spatial Computing
In 2023, 3D Gaussian Splatting emerged from INRIA as a real-time alternative to NeRF-style scene representation.20 Rather than querying a network at every rendered pixel, Gaussian Splatting represented scenes as millions of small 3D Gaussian primitives, each with position, scale, rotation, opacity, and spherical harmonic color coefficients. The result was photorealistic novel-view synthesis at interactive frame rates — scenes captured from video, rendered in real time, without any depth sensor.
Depth Anything (2024), from researchers at Hong Kong University and TikTok, established a new state of the art for monocular depth estimation by scaling training to over 62 million unlabeled images alongside labeled data.21 The resulting foundation model estimated depth with a degree of accuracy and generalization that made previous specialized approaches look brittle. For most practical XR and robotics applications, the question had shifted from can we measure depth without hardware? to how precise does the measurement need to be?
The Meta Quest 3 (2023) shipped as the first consumer headset with full-color mixed reality passthrough — two 18 pixels-per-degree color cameras compositing the physical environment into the display at near-real-time latency.23 Virtual objects could now be placed into a photorealistic view of the real world and occluded behind real furniture, not just in research demos but in a $499 consumer device.
Apple Vision Pro, launched in February 2024 at $3,499, represented a full convergence of the preceding decades.22 It combined a LiDAR scanner, two main cameras, four world-facing cameras, six microphones, eye tracking, hand tracking, and a neural engine running depth estimation, scene reconstruction, and rendering simultaneously. Its visionOS environment replaced windows, menus, and mice entirely with spatial interfaces anchored to the room. The display system used micro-OLED panels at 3660×3200 per eye with foveated rendering — rendering full detail only where the eye was looking. The form factor remained too heavy for extended use, the content ecosystem was thin at launch, and the price excluded most consumers — but the technical achievement was unambiguous: the XR stack that had been assembled piece by piece over sixty years had finally been integrated into a single shipping product.
The current state of the art sits at the intersection of these threads: foundation models that estimate geometry from images, Gaussian scene representations that render captured reality in real time, and headsets cheap enough (Quest 3) or capable enough (Vision Pro) to place persistent virtual objects in the physical world for ordinary users. The active depth sensors — structured light, time-of-flight, stereo rigs — that drove the field from 2010 to 2020 are being displaced at consumer scale by ML-inferred depth, retaining roles primarily in robotics, industrial metrology, and applications demanding millimeter precision.
The hard problems that remain — full-room persistent anchoring, socially acceptable form factors, natural haptic feedback, solving the vergence-accommodation conflict — are the same problems researchers were describing in the 1990s. The components to address them now exist. Assembly is the current work.
See also: Reality-Virtuality Continuum · Sega VR-1 · Nomadic · Microsoft Kinect · Microsoft HoloLens · Structure Sensor · Neural Radiance Fields · 3D Gaussian Splatting · Depth Anything · Apple Vision Pro · OpenXR