ARKit

ARKit is Apple's augmented reality framework for iOS, iPadOS, and visionOS. Introduced at WWDC 2017 with the release of iOS 11, ARKit gave developers access to the iPhone and iPad's cameras and motion sensors for world-tracking AR without requiring dedicated depth hardware — making Apple's installed base of hundreds of millions of devices the largest AR-capable platform in the world overnight.¹

Core Tracking

ARKit's foundational capability is Visual-Inertial Odometry (VIO): fusing data from the device's camera with the accelerometer and gyroscope (IMU) to estimate the device's position and orientation in 3D space in real time. The camera provides visual landmarks; the IMU provides high-frequency motion data between camera frames. Together they produce a 6DoF pose estimate that is stable enough to anchor virtual content convincingly to the real world as the user moves.²

ARKit performs this tracking without GPS or external markers — it builds a map of local visual features from the camera feed and maintains position within that map, a technique related to SLAM.

Plane Detection and Surface Understanding

From ARKit 1.0, the framework detects horizontal planes (floors, tables) and later added vertical planes (walls, doors). Applications receive ARPlaneAnchor updates as new surfaces are found and refined, enabling virtual objects to be placed on real surfaces and appear to rest naturally.

Scene Reconstruction (ARKit 3.5+, LiDAR devices): On iPhones and iPads with the LiDAR Scanner, ARKit generates a real-time mesh of the environment through scene reconstruction, enabling per-object occlusion, physics interaction, and more precise placement.⁴

People Occlusion and Motion Capture

ARKit 3 (iOS 13, 2019) introduced two capabilities that significantly advanced human-integrated AR:

People Occlusion uses a segmentation model to identify people in the camera frame and occlude virtual content behind them — so a virtual object placed behind a person is correctly hidden by that person. This runs on devices with the A12 Bionic chip or later, without LiDAR.³

Motion Capture detects and tracks human body pose from the RGB camera in real time, providing a skeleton that applications can use to drive avatar animation, create body-reactive experiences, or analyse movement. ARKit 3 could simultaneously track multiple people and blend front/rear cameras.

Depth API

ARKit 4 (iOS 14, 2020) introduced the Depth API for LiDAR-equipped iPhones (iPhone 12 Pro and later), exposing a per-frame scene depth map at camera resolution. Applications use this for precise occlusion, real-time surface detection, and mesh generation at resolutions far above what plane detection alone provides.⁴

Face Tracking

ARKit uses the TrueDepth camera (the structured-light sensor in the iPhone front camera notch) to track faces in 3D with 52 blendshapes — weighted coefficients corresponding to facial muscle action units. This powers effects like Animoji, FaceID, and developer-accessible face-reactive AR experiences.²

Reality Composer and RealityKit

ARKit is the tracking and sensing layer; RealityKit (2019) provides the rendering engine, physics simulation, and entity-component system designed to work with ARKit data. Reality Composer and Reality Composer Pro are no-code and pro-code tools for building AR scenes using ARKit and RealityKit.⁵ On visionOS, ARKit underpins the spatial tracking of Apple Vision Pro, with new capabilities for detecting room boundaries, floor detection, and hand skeleton tracking.

See also: Standards & SDKs · ARCore · WebXR · Scene Reconstruction · Occlusion in XR · Apple Vision Pro · SLAM