Scene Reconstruction

Scene reconstruction in XR is the process of building a three-dimensional model of the real physical environment — a mesh, point cloud, or volumetric representation — that the XR system can use for spatial understanding, occlusion, physics simulation, content placement, and navigation. Without some model of the world's geometry, virtual content cannot interact coherently with the physical space it is placed in.

Why Scene Reconstruction Matters

An XR system that only tracks position and orientation can place content in the world, but cannot reason about the world's shape. A virtual ball dropped in a room with no scene model falls through the floor; a hologram placed on a table clips through its surface; a user walking in VR passes through real walls unseen. Scene reconstruction makes the physical environment a first-class participant in the XR experience.

Depth-Sensor Approaches

KinectFusion (Microsoft Research / ISMAR 2011) was the first real-time dense scene reconstruction system to reach wide awareness.⁴ It used the original Kinect's structured-light depth sensor to integrate per-frame depth images into a volumetric TSDF (Truncated Signed Distance Function) grid, producing a dense mesh of the environment in real time as the user moved the sensor. KinectFusion demonstrated that a consumer depth sensor plus a GPU could reconstruct a room-scale environment at interactive frame rates — a result that previously required laboratory hardware.

HoloLens spatial mapping builds on this lineage. As the user wears and moves through a space, HoloLens continuously integrates depth data from its time-of-flight sensors into a spatial mesh anchored to the world. The mesh is available to applications as a collision surface, a physics substrate, and an occlusion mask.¹ HoloLens spatial maps are persistent: the device remembers the mesh of a familiar space and restores it on subsequent visits.

ARKit Scene Reconstruction (iOS 13.4+, LiDAR devices only) exposes a real-time mesh of the environment captured by the device's LiDAR scanner. The mesh updates continuously at walking speed, distinguishing floors, walls, ceilings, and furniture through semantic classification.²

Structural Understanding

Beyond raw geometry, higher-level scene understanding extracts semantic structure from the environment:

Apple RoomPlan (2022) uses the LiDAR scanner and cameras together to produce a parametric model of a room: walls, windows, doors, and furniture represented as structured data rather than raw mesh triangles.³ This enables applications that need to reason about room layout — furniture placement, navigation, floor plan generation — without parsing raw mesh geometry.

ARCore Scene Semantics (2023) classifies each pixel in the camera feed with a semantic label (sky, building, road, tree, person) using an on-device ML model, enabling outdoor AR experiences to reason about what type of surface content is being placed on.

Neural Scene Representations

Traditional mesh reconstruction represents scenes as explicit triangle geometry. A newer generation of neural scene representations encodes environments implicitly in neural network weights:

NeRF (Neural Radiance Fields, 2020) represents a scene as a continuous function mapping 3D coordinates and view direction to colour and density, trained on a set of photographs.⁵ NeRF renders photorealistic novel views of captured scenes but was originally slow to train and slow to render, limiting real-time XR use.

3D Gaussian Splatting (SIGGRAPH 2023) represents scenes as millions of 3D Gaussians — ellipsoidal primitives with position, scale, rotation, and colour — that can be rasterised at 100+ FPS on consumer GPUs.⁶ Gaussian splatting matches NeRF's visual quality while enabling real-time rendering, making it a compelling candidate for photorealistic environment reconstruction in XR pipelines.

Both approaches share a limitation for dynamic scenes: they reconstruct a static snapshot of the environment. Incorporating moving objects or updating the representation as the world changes remains an active research challenge.

Uses in XR

A scene reconstruction serves multiple downstream consumers in an XR application:

Occlusion: virtual objects are hidden by real surfaces (see Occlusion in XR)
Physics: virtual objects collide with and rest on real surfaces
Placement: applications snap content to detected planes, surfaces, or furniture
Navigation: autonomous agents or spatial audio systems reason about room layout
Portals and openings: applications detect doorways and windows to create spatial transitions