Face Tracking

Face tracking in XR reconstructs the geometry, pose, and expressive state of a user's face in real time, enabling the system to animate a virtual avatar that mirrors the user's expressions, drive eye contact in social VR, and support biometric authentication. It is distinct from eye tracking (which tracks only the eye's direction of gaze) in that face tracking captures the full surface of the face — including mouth, brows, cheeks, jaw, and nose — and maps muscle activations to expression coefficients.

Techniques

Monocular RGB face tracking uses a standard front-facing camera to estimate head pose and facial landmarks (the 468-point mesh used by MediaPipe Face Mesh, for example). This is the technology behind Snapchat lenses, TikTok filters, and ARKit's face tracking on iPhones with a front camera.³ The iPhone's TrueDepth camera (used for Face ID) adds a structured light depth sensor and infrared camera to this, producing a precise 3D face mesh — the basis for Animoji and Memoji.³

Inside-headset face tracking must work without a clear view of the face. Solutions include:

• Downward-facing cameras inside the headset pointed at the lower face (cheeks, nose, mouth) — used in Meta Quest Pro⁵
• Electromyography (EMG) sensors in the headset foam that read muscle activation through skin contact — an emerging technique
• IR LEDs and sensors positioned near the eyes and cheeks — used in the Pico 4 Enterprise face tracking accessory

Blendshapes and FACS

The standard output format for face tracking in XR is a set of blendshape weights — floating-point values (0–1) representing the activation of each facial action. These map to the Facial Action Coding System (FACS), developed by Paul Ekman, which describes facial expressions in terms of 44 Action Units (AUs) corresponding to discrete muscle movements.⁴ A blendshape weight for "jawOpen" at 0.8 means the jaw is 80% of the way to fully open; "browInnerUp" at 0.3 means a mild brow raise.

Game engines (Unity, Unreal) consume blendshape weights directly on avatar meshes, allowing real-time expression mirroring without requiring the developer to understand the underlying tracking mathematics.

Apple Vision Pro: Persona

Apple Vision Pro's Persona feature creates a photorealistic digital avatar of the wearer, built from a scan taken during device setup.¹ During use, the headset's internal IR cameras track the lower face (visible below the lenses) and eye movement; this drives the Persona avatar in FaceTime calls and spatial video conferencing. Because the headset covers the eyes, the eye animations are inferred and stylised rather than directly captured.

The result is a semi-realistic avatar that moves with the user's expressions — blinking, smiling, speaking — while avoiding the full uncanny valley of a photographic face because the rendering is slightly idealised. EyeSight (the outward-facing display showing approximate eye position to bystanders) addresses the social isolation problem from the other direction: making the wearer legible to people physically present.

Meta Codec Avatars

Meta's research division has pursued Codec Avatars — photorealistic neural avatars driven by headset-internal sensors — since 2019.² The approach uses a variational autoencoder trained on dense multi-camera captures of a person's face. At runtime, a compact latent vector (the "codec") encodes the current expression state; decoding this vector produces a photorealistic mesh and texture. The codec can be transmitted over a network far more efficiently than raw video, enabling high-quality telepresence at modest bandwidth. As of 2024, Codec Avatars remain a research prototype not yet available in consumer Meta headsets, but they demonstrate the long-term direction: social VR in which participants appear as themselves rather than as cartoon avatars.²

See also: Tracking · Eye & Gaze Tracking · Body & Skeleton Tracking · Apple Vision Pro · ARKit

References

1. Use Persona in FaceTime on Apple Vision Pro — Apple Support(accessed May 1, 2026)
2. Codec Avatars — Meta AI Research(accessed May 1, 2026)
3. Tracking and Visualizing Faces — ARKit, Apple Developer Documentation(accessed May 1, 2026)
4. Emotions Revealed — Paul Ekman, 2003(accessed May 1, 2026)
5. Meta Quest Pro — Meta(accessed May 1, 2026)