MRTK

MRTK (Mixed Reality Toolkit) is Microsoft's open-source framework for building mixed reality applications in Unity, providing spatial interaction patterns, UX components, and cross-platform support for HoloLens, Meta Quest, and PC VR headsets. It began as a developer convenience layer for HoloLens and evolved into the most widely referenced set of codified XR interaction design patterns in the industry.

History

MRTK started as a collection of Unity scripts and prefabs bundled with early HoloLens development documentation, intended to give developers a starting point for common interactions. By HoloLens 2's launch in 2019, MRTK 2 had grown into a comprehensive framework covering input, spatial awareness, UX components, and a flexible input abstraction system. It became the de facto standard for professional HoloLens application development and significantly influenced spatial UI design patterns across the industry.¹

MRTK3 (2022) was a full architectural rebuild, dropping MRTK's proprietary input abstraction in favour of OpenXR and Unity's XR Interaction Toolkit (XRI) as the underlying standards. MRTK3 is modular and package-based rather than monolithic, allowing developers to take only the components they need.⁴

Interaction Primitives

MRTK codified three fundamental interaction types for spatial computing, which remain the standard vocabulary across the industry:³

Hand Ray (far-field): A ray extending from the user's palm in the pointing direction, visualised as a beam with a cursor at the hit point. Used for selecting and manipulating UI panels and objects at any distance beyond arm's reach. Equivalent to a mouse cursor extended into 3D space. See Ray Casting.

Direct Touch (near-field): Fingertip contact with a virtual surface or button when the hand is within arm's reach. The fingertip triggers a button on physical penetration of its front face. Near-field touch enables natural keyboard and button interactions without requiring precise pointing.

Grab (object manipulation): Closing the hand around a virtual object to pick it up, move it, rotate it, and release it. Two-handed grabbing enables scaling and rotation. Grab is the primary manipulation primitive for volumetric objects as opposed to flat UI panels.

UX Components

MRTK provides a library of ready-to-use spatial UI components designed for XR ergonomics:⁵

• Pressable Button: A 3D button with a physical front face that responds to both near touch and far ray interaction, with animated press travel and haptic/audio feedback
• Slider: A spatial slider control with grabbable thumb
• Hand Menu: A UI panel that attaches to the user's hand and appears when the palm faces up
• Near Menu: A panel that follows the user's gaze at a comfortable reading distance
• Object Manipulator: A component that adds grab-and-move behaviour to any GameObject
• Bounds Control: Visible handles and edges around an object enabling resize and rotation via grab

Cross-Platform Support

MRTK3's OpenXR foundation means the same application code runs on HoloLens 2, Meta Quest (2, 3, Pro), PC VR headsets via SteamVR and Windows Mixed Reality, and theoretically any OpenXR-conformant runtime. Platform-specific capabilities are accessed through OpenXR extensions rather than platform APIs, with MRTK providing abstractions where behaviour differs across runtimes.²

Industry Influence

MRTK's interaction patterns — hand ray, direct touch, grab — have been adopted or cited by virtually every serious spatial UI framework. The design principles published alongside MRTK (minimum target sizes, comfortable rendering distances, feedback requirements for gaze-dwell, audio cues for spatial interaction) function as an applied design language for the field, referenced in the Spatial UI Design literature independently of the toolkit itself.

See also: Standards & SDKs · OpenXR · Spatial UI Design · Hand Tracking · Ray Casting · Microsoft HoloLens