Hand Tracking

Hand tracking in XR is the real-time detection, reconstruction, and skeletal mapping of a user's hands using cameras — allowing bare hands, without controllers, to serve as the primary input device. A hand tracking system must locate the hands in 3D space, identify individual finger segments and joints, and output a skeletal model fast enough to drive interaction at interactive frame rates.

Skeletal Model

The standard representation of a tracked hand is a skeletal model of 21 keypoints: the wrist, the metacarpophalangeal (MCP), proximal interphalangeal (PIP), and distal interphalangeal (DIP) joints of each finger, plus the fingertip — five chains of four points plus the wrist. Each keypoint has a 3D position; the full model provides orientation, bend state, and spread of each finger. From this representation, gesture classifiers can reliably detect pinch (thumb-index contact), grab (all fingers curled), point (index extended), and more complex poses.

History

Leap Motion (founded 2010, San Francisco) shipped the first consumer hand tracking peripheral in 2013 — a USB sensor that sat on a desk and tracked hands above it using two infrared cameras and structured light. The original device was designed for desktop use; Leap Motion later pivoted to VR, shipping the Leap Motion Controller as a clip-on for headsets and signing an integration deal with HP for the Reverb G2 Omnicept.¹ The company rebranded as Ultraleap in 2019 following a merger with Ultrahaptics, a mid-air haptics company, combining hand tracking with ultrasonic tactile feedback.

Microsoft HoloLens included basic hand gesture recognition (air tap) from its first generation in 2016, using depth cameras to detect a single pointing gesture. HoloLens 2 (2019) replaced this with full articulated hand tracking — 25 joints per hand — using a purpose-built time-of-flight depth camera running a neural network pipeline entirely on the device's Holographic Processing Unit.³ This was the first mainstream MR headset to support hands as a first-class input without controllers.

Meta shipped hand tracking as an experimental feature on the Quest in 2019, using the headset's four existing inside-out tracking cameras rather than dedicated hand sensors.² The system ran a convolutional neural network that processed the fisheye camera feeds to locate and reconstruct hand pose in real time. Meta's approach demonstrated that separate depth hardware was not required — RGB cameras plus a capable neural network were sufficient.

Apple Vision Pro (2024) uses hand tracking as its only primary input method — there are no controllers at all.⁵ The system tracks hands using the headset's main cameras and an on-device neural engine, detecting pinch gestures (thumb-index contact) as the primary selection mechanism and supporting complex multi-finger gestures for system navigation. The absence of controllers was a deliberate design choice: controllers add weight, require pairing, need charging, and constrain the user to specific hardware affordances.

Google's MediaPipe Hands (2019) brought accurate real-time hand tracking to mobile phones using a single RGB camera — no depth sensor, no IR, just the standard back camera.⁴ The system used a two-stage pipeline: a palm detection model to locate the hand in the frame, followed by a hand landmark model that regressed 21 3D keypoints from the cropped hand region. Running at over 30 fps on a mobile CPU, MediaPipe Hands democratised hand tracking to billions of existing devices.

Interaction Design

Hand tracking changes the interaction vocabulary significantly compared to controllers. Controllers provide haptic feedback on button press; bare hands do not — the system must provide visual or audio feedback to confirm selections. The most common patterns are:

Pinch-to-select: touching thumb and index fingertip together, used by Apple Vision Pro as the primary activation gesture. Requires precise fingertip localisation.

Ray casting from hand: extending a ray from the hand or index finger toward distant UI panels, selecting where the ray intersects. Used by Meta Quest and HoloLens 2 for far-field interaction.

Near-field direct manipulation: reaching out and touching virtual objects directly when they are within arm's reach. Requires sub-centimetre accuracy and low latency to feel natural.

Grab and release: closing the fist to grab a virtual object, opening to release — mirrors the real-world affordance of picking things up.

The main challenge of controller-free hand tracking interaction is the absence of physical confirmation. When pressing a real button, the button stops moving and provides tactile resistance. With bare hands, the only confirmation that a gesture was registered is visual or audio feedback from the system — making feedback design critical to avoiding accidental activations and missed inputs.

Skeletal Model

History

Interaction Design

References