Ray Casting

Ray casting in XR interaction is the technique of projecting an invisible ray from a point and direction — typically from the user's hand, controller, or gaze — and detecting what that ray intersects in the virtual or physical scene. The intersected object, surface, or UI element becomes the target of the user's interaction. Ray casting is the far-field interaction primitive of XR: it extends the user's reach beyond arm's length, making it possible to interact with UI panels, virtual objects, or real surfaces anywhere in the environment without moving toward them.

How It Works

At each frame, the system computes a ray origin (the hand position, controller tip, or eye position) and a ray direction (the pointing direction of the hand or finger, the controller forward vector, or the gaze direction). It tests this ray against the scene's collision geometry — virtual object meshes, UI panels defined as interactable, and sometimes the real-world mesh from scene reconstruction. The nearest intersection point becomes the hit point, and the object at that point is the target.

Visual feedback is typically provided by a cursor rendered at the hit point — a small dot, ring, or reticle — and a beam drawn along the ray from the hand to the cursor. This visual representation makes the pointing direction legible and gives the user feedback about what they are currently targeting. Many systems curve the beam slightly downward (parabolic arc) for natural ergonomics, matching the natural drooping of an outstretched arm.¹

Origins and Origins

Ray casting as a spatial interaction technique was established in early VR research in the 1980s and is a direct extension of the pointer metaphor from desktop interfaces. In desktop UI, a mouse cursor points at a 2D target; in XR, a ray from the hand points at a 3D target. The Microsoft HoloLens 2 hand ray, Meta Quest's Pointer Interactor, and Apple Vision Pro's gaze-directed attention all implement variants of the same fundamental mechanic.¹²

Far-Field vs. Near-Field

Ray casting is specifically the far-field interaction primitive. When an object or panel is within arm's reach (roughly 70 cm), direct manipulation — reaching out and touching — is more natural, faster, and provides better precision. The crossover point between far-field ray casting and near-field direct manipulation is typically detected by proximity: when the user's hand enters a threshold distance from an interactive object, the system switches from ray casting to near-field interaction mode.¹

This two-mode design — ray for far, direct touch for near — is the standard in HoloLens 2's MRTK, Meta's Interaction SDK, and visionOS's spatial input model. The transitions between modes must be smooth to avoid jarring switches in interaction behaviour.

In AR World Interaction

Beyond virtual UI, ray casting is also the primary technique for hitting real-world surfaces in AR. When a user taps on a phone screen in ARKit or ARCore, the framework casts a ray from the camera through the tapped screen point into the scene, testing against the detected planes and mesh geometry. The resulting hit point is where the virtual object is placed.⁴ This "tap-to-place" interaction pattern is the most common mechanism for placing AR content in mobile AR applications.

Limitations

Ray casting at long distances amplifies pointing jitter: a small rotation of the wrist produces a large cursor movement at the end of a long ray. This is the far-field equivalent of Fitts's Law — targets that are far away or small are harder to hit precisely, requiring systems to implement target snapping (snapping the cursor to the nearest valid target within a threshold) or dwell confirmation (requiring the cursor to remain on a target for a brief period before registering a selection).²

See also: Interaction & UI · Spatial UI Design · Hand Tracking · Gaze-Dwell Selection · Virtual Keyboards · ARKit