Depth Anything

Depth Anything is a family of monocular depth estimation models developed by Yang et al., first published in January 2024 (CVPR 2024) and updated as Version 2 in June 2024. It estimates depth from a single RGB image or video frame without any dedicated depth sensor, using a large-scale training approach that combines a small labeled dataset with 62 million unlabeled images to learn depth representations that generalise across scenes, lighting conditions, and camera types. Depth Anything represents the current state of the art in general-purpose monocular depth estimation and has direct applications for AR on any camera-equipped device.¹

The Monocular Depth Problem

Recovering 3D structure from a single 2D image is an ill-posed problem — infinitely many 3D scenes could produce the same image. Human vision solves this through learned priors: the appearance of surfaces, typical object sizes, perspective foreshortening, atmospheric haze, and occlusion relationships all carry depth information. ML models for monocular depth estimation learn the same priors from large datasets of images paired with ground-truth depth measurements.³

The challenge is generalisation: models trained on images from a specific sensor or environment (indoor rooms, autonomous driving scenes) perform poorly on out-of-distribution inputs. Depth Anything's key contribution is addressing this generalisation problem through scale.

Architecture and Training

Depth Anything uses a DINOv2 vision transformer as its encoder — a large self-supervised vision model trained on 142 million images that learns rich, generalisable visual representations without task-specific labels.⁴ This encoder feeds into a depth prediction head fine-tuned for monocular depth.

The training pipeline combines:

Labeled data: 1.5 million images with precise ground-truth depth from LiDAR sensors (indoor and outdoor, multiple datasets)
Unlabeled data: 62 million internet images with pseudo-labels generated by a teacher model, providing scene diversity far beyond what any labeled dataset could supply¹

This semi-supervised approach gives Depth Anything breadth across scene types — natural landscapes, urban environments, indoor spaces, close-up objects — that previous models achieved only within specific domains.

Depth Anything V2

Version 2 (June 2024) replaced synthetic pseudo-labels in the unlabeled training set with higher-quality synthetic data from photorealistic rendering engines, significantly improving prediction quality at fine detail boundaries — thin structures, transparent surfaces, reflective materials.² V2 also provides metric depth variants (predicting absolute distances in metres rather than relative depth ordering) trained with scale calibration for specific camera configurations.

Outputs: Relative vs. Metric Depth

The base Depth Anything models produce relative depth — a depth map where values encode the order of surfaces from near to far, but not their absolute distance. This is sufficient for most AR use cases that need to know which is in front (occlusion, collision) but not exact distances. Metric depth variants provide absolute distance estimates, enabling use cases like object sizing, spatial measurement, and AR placement at specific real-world distances.

Impact on AR Without Depth Sensors

Depth Anything enables occlusion, surface detection, and depth-dependent rendering effects on any device with a camera — no LiDAR, no structured-light sensor, no ToF required. For the roughly 3 billion smartphones without hardware depth sensors, it provides the depth layer needed for realistic AR compositing at a quality level approaching purpose-built sensors.⁵

This was the technological trajectory described in the history of XR — from the Kinect's specialised structured-light hardware through the Structure Sensor's mobile depth hardware to ML models running on unmodified consumer cameras. Each step reduced the hardware requirement for depth-aware XR, democratising capabilities that were previously gated on specialised equipment.

The Monocular Depth Problem

Architecture and Training

Depth Anything V2

Outputs: Relative vs. Metric Depth

Impact on AR Without Depth Sensors

References