Occlusion in XR

Occlusion in XR is the correct depth-ordering of virtual and real objects: a virtual cube placed behind a physical table should be hidden by the table, just as a real object would be. Without occlusion, virtual objects appear to float on top of everything in the scene regardless of their intended position, which immediately breaks the illusion of co-presence and makes spatial placement feel unconvincing.

Occlusion is one of the hardest unsolved problems in practical AR. It requires the system to know the depth of every real-world surface in real time — information that was historically expensive to obtain, and that is still imperfect on current consumer hardware.

The Occlusion Problem

In a virtual-only scene, occlusion is trivial: the renderer knows the depth of every object because it created them, and standard depth buffering handles ordering automatically. In AR, the renderer must composite virtual objects over a live camera feed of the real world. The camera feed has no inherent depth information — it is a flat image of a scene where depth is unknown unless separately measured or estimated.

The consequence is that naive AR rendering places all virtual content in front of all real content, regardless of geometry. A virtual character standing behind a sofa appears to pass through it; a holographic label placed next to a physical tool appears to hover in front of it from any angle.

Approaches to Real-World Depth

Spatial mesh occlusion uses a pre-built mesh of the environment from SLAM-based scene reconstruction. HoloLens renders the spatial mesh as an invisible depth-occluder: virtual holograms are hidden by mesh surfaces that spatially correspond to real walls, floors, and furniture. The limitation is that the mesh is coarse (typical resolution 2–5 cm), captures only static geometry, and does not include people or moving objects.³

Depth sensor occlusion uses a ToF or structured-light depth camera to capture a per-frame depth image of the scene. This depth image is used as a real-time depth buffer against which virtual objects are tested. iPhone 12 Pro and later include a LiDAR sensor; ARKit exposes this as sceneDepth on supported devices.⁴ Apple Vision Pro's depth sensors provide per-frame depth, enabling the hand occlusion model that makes virtual objects realistically hide behind the user's hands. Meta Quest 3's structured depth projector similarly enables hand-occlusion in passthrough mode.

Person and body segmentation uses ML models to isolate people in the camera frame and occlude virtual content behind them. Apple's ARKit People Occlusion (iOS 13+) uses semantic segmentation to generate a per-frame matte separating people from the background, enabling accurate occlusion of virtual objects by human bodies without requiring depth hardware.¹ This works on any device with Neural Engine support and is robust to movement, but applies only to people and not arbitrary real-world objects.

ML depth estimation — using neural networks to infer depth from monocular RGB (see Depth Sensing) — enables approximate occlusion on any camera, but estimated depth maps are not precise enough for sharp occlusion edges and introduce artefacts at object boundaries.

Hard vs. Soft Occlusion

Hard occlusion applies a binary depth test: a virtual pixel is either fully visible or fully hidden. This is perceptually correct for opaque objects but produces sharp, aliased edges where the real object's depth mask meets the virtual content — edges that are clearly artificial when the depth mask is noisy or misaligned.

Soft occlusion blends the transition at the boundary, feathering the virtual content in a narrow region around the real object's edge. This hides depth map noise and produces visually smoother compositing, at the cost of introducing a subtle transparency halo that is not physically accurate.⁵ Most production AR systems use some form of soft occlusion at boundaries.

Hand Occlusion

The most commercially important occlusion case is hand-virtual object occlusion: ensuring that virtual content placed in the user's hand-space is correctly hidden when the real hand passes in front of it. This is critical for believable hand interaction — without it, picking up a virtual object shows the hand visibly clipping through it.

Meta Quest Pro, Quest 3, and Apple Vision Pro all implement hand occlusion using depth sensor data or the inside-out depth from stereo passthrough cameras. Meta's implementation uses the structured depth projector; Apple's uses the system depth sensors feeding into the R1 chip pipeline.²

See also: Rendering & Display · Scene Reconstruction · Passthrough · SLAM · Hand Tracking · Depth Sensing