Body and Skeleton Tracking
Body and skeleton tracking reconstructs the full-body pose of a person in real time — the 3D position and orientation of major joints (hips, spine, shoulders, elbows, wrists, knees, ankles, and head) — enabling XR systems to animate full-body avatars, drive inverse kinematics chains, and understand a user's physical posture and movement. It is significantly harder than hand tracking because the body is larger, more occluded, and involves a kinematic chain with more degrees of freedom.
The Kinect Breakthrough
The defining moment for consumer body tracking was the Microsoft Kinect (2010). Its skeleton tracking algorithm, published by Shotton et al. at CVPR 2011, used a random decision forest trained on millions of synthetic depth images to classify each pixel of the depth frame as belonging to a body part — left forearm, right shoulder, head, and so on.1 From the body-part classification, joint positions were estimated by finding modes in the distribution of candidate joint locations.
The result ran at 30 frames per second on the Xbox 360's CPU and produced a 20-joint skeleton for up to two people simultaneously — including head, neck, shoulders, elbows, wrists, hands, spine, hips, knees, ankles, and feet.2 This was the first time body skeleton tracking had been available at consumer price and with a developer SDK, and it triggered a wave of gesture interfaces, health applications, robotics research, and spatial scanning tools that had nothing to do with gaming.
Kinect 2 (2013) upgraded to time-of-flight depth sensing and a 25-joint model, adding joints for the hand tip, thumb, and foot, and supporting up to six bodies simultaneously.
Optical Markerless Approaches
Parallel to depth-based tracking, computer vision researchers developed methods to estimate body pose from ordinary RGB cameras — no depth sensor, no markers, no suits.
OpenPose (Carnegie Mellon, 2017) used a multi-stage CNN to detect 2D body keypoints from RGB video, extended to 3D by triangulating across multiple cameras.3 It became a widely used research tool for human motion analysis.
MoveNet (Google, 2021) brought single-camera body pose estimation to mobile devices, running at 30+ fps on a phone CPU with 17 keypoints.4 This enabled body tracking applications on standard smartphones without any additional hardware.
In XR Headsets
Modern consumer XR headsets face a fundamental challenge with body tracking: the headset's cameras look outward from the head, not downward at the body. Most of the body is below the camera's field of view.
Current consumer headsets (Meta Quest, Vision Pro) provide upper-body reconstruction: by fusing head position (from 6DOF headset tracking), hand positions (from hand tracking), and statistical body models, the system infers the likely position of shoulders, elbows, and spine — even though it cannot see them. The hips and legs are typically estimated from the standing position and stepping motion rather than directly observed.
Meta Quest Pro added downward-facing cameras that provide partial torso and hip visibility, improving the quality of the inferred pose.5 For full lower-body tracking, external accessories like Vive Trackers (strapped to feet and hips) provide ground truth that is fused with headset data.
Full-body inside-out tracking — reconstructing the complete body from only headset-mounted cameras, with no accessories — remains an active research problem. Neural network approaches trained on paired headset-camera / ground-truth-mocap data have shown promise, but production-quality full-body tracking from a headset alone is not yet available in consumer devices as of 2026.
Inverse Kinematics
The output of body tracking feeds Inverse Kinematics (IK) solvers, which compute joint angles that produce a desired end-effector position. In XR avatar animation, IK takes the tracked positions of the head and hands (the sparse set of reliable observations) and solves for the elbow, shoulder, and spine angles that place the arms in a physically plausible configuration. The challenge is that many joint configurations can produce the same hand position — IK solvers must choose among them using physical plausibility constraints, learned pose priors, or both.
See also: Tracking · Hand Tracking · Microsoft Kinect · SLAM · Face Tracking