SLAM

SLAM — Simultaneous Localization and Mapping — is the computational problem of building a map of an unknown environment while simultaneously tracking the device's position within that map. It is the foundational technology beneath almost every modern XR headset and mobile AR framework: the reason a virtual object stays fixed to a real surface as you walk around it is that SLAM is continuously estimating where the camera is relative to the geometry it has observed.

The Problem

A camera or sensor moving through an environment faces a chicken-and-egg problem: to know where it is, it needs a map; to build a map, it needs to know where it is. SLAM algorithms resolve this by treating both the map and the trajectory as unknowns to be jointly estimated from sensor observations. The theoretical foundations were established in robotics by Smith and Cheeseman in 1986, who framed the problem in terms of Bayesian probability and spatial uncertainty.¹

Visual SLAM

Visual SLAM (vSLAM) uses cameras as the primary sensor rather than lidar or sonar. The algorithm extracts feature points from camera frames — corners, edges, or learned descriptors — and tracks them across frames. The motion of feature points between frames encodes information about the camera's motion (by the geometry of optical flow) and about the 3D structure of the scene (by triangulation from multiple viewpoints). By accumulating these observations, the system builds a sparse 3D map of feature points and a trajectory estimate.

The key challenges are loop closure (recognising a previously visited location and correcting accumulated drift) and scale ambiguity in monocular systems (a single camera cannot determine absolute scale without additional information).

PTAM and the AR Transition

Parallel Tracking and Mapping (PTAM), published by Klein and Murray at ISMAR 2007, was the first SLAM system fast enough to run in real time on a laptop and was specifically designed for AR rather than robotics.² PTAM separated the tracking thread (running at frame rate, estimating camera pose from the current map) from the mapping thread (running more slowly, refining the map by bundle adjustment). This split enabled real-time AR overlay on a tracked scene for the first time on consumer hardware — PTAM is the direct ancestor of the world-tracking pipelines in ARKit and ARCore.

ORB-SLAM

ORB-SLAM (2015) became the most widely used monocular SLAM framework in academic research, using ORB (Oriented FAST and Rotated BRIEF) features for their speed and invariance to rotation and scale.³ Its successors (ORB-SLAM2, ORB-SLAM3) extended it to stereo and RGB-D cameras and added inertial measurement unit (IMU) fusion. IMU fusion — combining camera observations with accelerometer and gyroscope readings — dramatically improves robustness in fast motions and textureless environments where visual features are sparse.

Visual-Inertial Odometry (VIO)

Most consumer XR systems use Visual-Inertial Odometry (VIO) rather than full SLAM: they track camera pose accurately over time (odometry) without necessarily maintaining a globally consistent map. VIO fuses camera frames with IMU measurements, achieving low-drift pose tracking suitable for room-scale AR. Apple describes ARKit's world tracking as VIO; it builds a sparse point cloud of the environment, tracks features across frames, and fuses with the iPhone's IMU to maintain continuous 6-degrees-of-freedom pose estimation.⁴

Microsoft HoloLens extends this with active depth sensors: the device's time-of-flight and structured light sensors continuously scan the room and build a dense mesh of the physical geometry, enabling precise surface-aware placement of holograms and mesh-based occlusion.⁵ This denser representation is sometimes called spatial mapping or scene reconstruction rather than SLAM, but the underlying tracking problem is the same.

SLAM in Modern Consumer Headsets

Every inside-out tracked XR headset — Meta Quest, Apple Vision Pro, HoloLens, PlayStation VR2 — runs a SLAM-derived pipeline to track its 6DOF pose. The specific implementations are proprietary, but share the common architecture: visual feature extraction from multiple fisheye cameras, IMU fusion, sparse map maintenance, and loop closure to prevent long-term drift. The seamless experience of virtual objects staying fixed in physical space is SLAM operating successfully; an object that "drifts" away from its anchor is SLAM accumulating uncorrected error.

See also: Tracking · Spatial Anchors · Scene Reconstruction · ARKit · ARCore · Microsoft HoloLens