Eye and Gaze Tracking

Eye and gaze tracking in XR uses infrared cameras and corneal reflection to determine, in real time, exactly where within a display the user is looking. The gaze vector — the direction from each eye's pupil through the optical axis — is combined with scene geometry to identify a 3D fixation point in the environment. This information drives two distinct applications: foveated rendering (rendering at full resolution only where the eye is looking) and gaze-based interaction (using eye direction as an input signal).

How It Works

Consumer XR headsets use near-infrared (NIR) illumination: small LEDs or laser emitters inside the headset project infrared light onto the eye, invisible to the user. NIR cameras (typically two or four per eye) capture the resulting reflections. The key features extracted are the Purkinje images — reflections off the anterior cornea and lens — and the pupil centre. By tracking how these features move relative to each other as the eye rotates, algorithms compute gaze direction to within approximately 0.5–1.5 degrees of visual angle in well-calibrated consumer systems. Research-grade trackers (Pupil Labs Neon, SMI) reach 0.1–0.3 degrees.⁵

A calibration procedure — typically asking the user to look at a sequence of fixation points — aligns the tracker's output to the specific geometry of the user's eyes and headset fit. Some systems skip explicit calibration by using personalized models built during initial headset setup; Apple Vision Pro's Optic ID enrollment serves this dual purpose.³

Hardware Integration

Tobii (founded 2001, Stockholm) was the first company to ship commercial eye trackers for consumer use and pioneered integration into VR headsets, partnering with HTC for the Vive Pro Eye (2019) and Vive Focus 3.¹ Tobii's Eye Tracking SDK became the dominant development interface for gaze-based XR applications.

Microsoft HoloLens 2 (2019) integrated eye tracking as a core input mechanism — the first mainstream MR headset to do so at launch.² HoloLens 2's eye tracking serves both interaction (gaze-directed commands) and calibration (fitting the holographic display to each user's inter-pupillary distance).

Apple Vision Pro (2024) uses eye tracking as its primary navigation mechanism: the user looks at an interface element to direct attention, then performs a hand pinch to activate it. This architecture — gaze for targeting, hand gesture for confirmation — keeps interaction natural and eliminates the need for a cursor.³ Vision Pro's eye tracking also drives Optic ID, an iris-based biometric authentication system.

Meta Quest Pro (2022) added eye tracking to the Quest line, enabling foveated rendering and avatar eye animation. The Quest 3 omitted eye tracking at its $499 price point, though it remains in the Pro tier.

Foveated Rendering

The primary technical application of eye tracking in rendering is foveated rendering: the GPU renders the region around the current fixation point at full resolution while reducing resolution in the periphery, where the eye's spatial acuity drops sharply. The human fovea — the central 2–5 degrees of vision — resolves fine detail; beyond this, acuity falls off rapidly and the visual system uses peripheral vision primarily for motion detection rather than detail.⁴

Because XR headsets must render millions of pixels per frame at 90+ Hz, foveated rendering can reduce GPU workload by 30–60% without perceptible quality loss — savings that translate into either better visual quality at the same power envelope, or the same quality at lower power. Without eye tracking, static foveated rendering (which reduces peripheral resolution regardless of gaze direction) provides partial gains; dynamic foveated rendering (which follows gaze in real time) provides the full benefit but requires tracking latency below approximately 5 milliseconds to avoid visible artefacts at gaze boundaries.

Gaze as Input

Beyond rendering, gaze direction is a rich input signal. In XR interfaces, gaze can:

• Pre-select: highlight UI elements the user is looking at, reducing the motor action needed to formally select them
• Scroll: move content in the direction of gaze in document or list views
• Aim: direct a pointing ray for far-field interaction (particularly useful for hands-free or accessibility use cases)
• Infer intent: analytics systems can determine which objects in a scene attracted attention and for how long

The primary design concern with gaze as input is the Midas Touch problem: if looking at something triggers it, every glance becomes an inadvertent command. Current systems address this by separating the gaze targeting signal from the confirmation signal — look to target, hand pinch or dwell to confirm.

See also: Tracking · Hand Tracking · Foveated Rendering · Gaze-Dwell Selection · Apple Vision Pro · Microsoft HoloLens