Voice Input in XR

Voice input in XR uses speech recognition to accept spoken commands, queries, and text dictation from users wearing headsets or using mobile AR. It is the most natural complement to spatial hand interaction — filling the gap where gestures are impractical (text entry, complex commands, system navigation) and where hands are occupied or unavailable.

Why Voice Matters in XR

XR headsets create a uniquely favourable environment for voice input. The user is already wearing a device near their mouth; microphones built into the headset can achieve high signal quality with noise cancellation tailored to the physical configuration. Unlike phone voice assistants, the headset system has full context about what the user is looking at, what objects are in the scene, and what application is in focus — enabling contextual disambiguation that flat voice interfaces cannot match.¹

The gap voice fills is most visible in text entry: no virtual keyboard technique approaches the speed or endurance of physical typing, but voice dictation on modern speech recognition models achieves 150–180 words per minute with accuracy exceeding 95% for clean audio. For search queries, annotations, messages, and document dictation, voice is the fastest available input modality in XR.²

Command vs. Dictation

Voice input in XR divides into two modes:

Command recognition maps specific spoken phrases to discrete actions — "go home," "take screenshot," "scroll down," "select," "confirm." Command vocabularies are typically small (dozens to hundreds of phrases), can run entirely on-device, and achieve near-perfect accuracy within their vocabulary. HoloLens has supported spatial voice commands since its first generation: users could say "select" to activate the currently gazed-at element, or speak the label of any visible UI element to activate it directly.¹

Free-form dictation transcribes continuous speech to text using large vocabulary speech recognition. This requires significantly more compute and was historically dependent on cloud APIs. OpenAI Whisper (2022) demonstrated that a large-scale transformer model trained on 680,000 hours of audio could achieve robust multilingual transcription running entirely on-device on consumer hardware — changing the feasibility calculus for offline XR dictation.³ Apple Vision Pro uses on-device dictation (via Siri) throughout visionOS as a first-class text input method.²

Multimodal Interaction

Voice is most powerful when combined with other input modalities rather than used in isolation. The canonical multimodal XR pattern is "look and say": gaze or hand ray to target an object or element, voice to command the action on that target — "move this here," "what is this?", "show me more." This pattern mirrors natural human communication (we look at something before speaking about it) and allows spatial disambiguation without long command strings.⁴

Conversational AI integration — shipping in devices from 2023 onward — extends voice input from command execution to open-ended dialogue: asking the headset questions about the current environment, requesting explanations of visible objects, navigating unfamiliar software by natural language description. Apple Vision Pro's Siri integration and Meta's AI assistant on Quest both implement this model.

Challenges

Privacy and social acceptability are the primary adoption barriers for voice in XR. Speaking commands aloud in shared spaces — offices, public locations — exposes conversational content to bystanders and can feel socially awkward. Subvocalisation detection (interpreting lip movements or throat vibrations without audible speech) is an active research area that would address both concerns, but is not yet available in consumer devices.

Noise robustness in physically active XR scenarios — location-based VR arenas, construction sites, warehouses — degrades recognition accuracy. On-device noise cancellation models, directional microphone arrays, and bone conduction microphones (which pick up skull vibration rather than airborne sound) are partial mitigations.⁴

Latency for cloud-dependent recognition introduces perceptible delays between speech and response, breaking the natural cadence of interaction. On-device models eliminate this entirely for command vocabularies; on-device large-vocabulary dictation is now viable on devices with Apple silicon or Snapdragon XR2 and above.

See also: Interaction & UI · Gaze-Dwell Selection · Virtual Keyboards · Spatial UI Design · Apple Vision Pro · Microsoft HoloLens