Passthrough

Passthrough is the technique of capturing the physical environment with outward-facing cameras on a head-mounted display and rendering that camera feed — composited with virtual content — on the device's internal displays. The result is a mixed reality view: the user sees the real world, but it is mediated through video, allowing virtual objects to be placed, anchored, and occluded within it with full computational control.

Passthrough sits at the heart of the current generation of consumer mixed reality headsets. It is what allows a Meta Quest 3 or Apple Vision Pro user to see their room, their hands, and their furniture — and simultaneously interact with virtual objects sitting on real surfaces, persisting in real space. The mediating step — real world → camera → processor → display — is what distinguishes video passthrough from optical see-through systems, and it is what makes the full mixed reality compositing pipeline possible.

Apple Vision Pro passthrough showing mixed reality compositing of virtual and physical environments

Optical See-Through vs. Video Passthrough

There are two fundamental approaches to blending virtual content with the physical world in a headset.

Optical see-through places a partially transparent optical element — a waveguide, a half-mirror, or a diffractive combiner — in front of the eye. The real world is visible directly and continuously through the optic; virtual content is projected onto the combiner and overlaid additively. The user's view of reality has zero latency and full natural resolution, because they are simply looking through glass. The Microsoft HoloLens and Magic Leap both use optical see-through waveguides.⁹ The fundamental constraint of optical see-through is that virtual content can only add light — it cannot make any part of the real scene darker, cover real objects with opaque virtual surfaces, or fully occlude reality. Bright environments wash out holograms; the field of view for virtual content is constrained to the size of the combiner element.

Video passthrough closes the optical path entirely. The headset's outward cameras continuously capture the physical environment; that footage — along with rendered virtual content — is composited on the device's internal displays before it reaches the eye. The user sees a digital representation of the real world, not the world directly. This approach surrenders optical see-through's lossless view of reality, but gains something critical: the compositor has full control over the combined image. Virtual objects can be rendered opaque, can be correctly occluded behind real geometry, and can interact with real-world lighting in physically plausible ways. The tradeoffs are camera resolution, dynamic range, latency, and the power cost of running two high-resolution camera pipelines continuously.

The industry has largely converged on video passthrough for consumer mixed reality headsets, driven by its superior compositing flexibility — and by the rapid improvement in camera sensors and dedicated processing silicon.

Apple Vision Pro

Apple's Vision Pro (launched February 2024) represents the current peak of video passthrough fidelity.¹ The device pairs two main wide-angle cameras with ten additional cameras, five sensors, and six microphones to build a continuous model of the physical environment. The passthrough video pipeline runs through the R1 chip — a dedicated co-processor separate from the main M2/M5 application processor — which processes sensor data and renders the passthrough image in under 12 milliseconds, below the threshold at which most humans can perceive the mediation between camera and display.¹

The displays behind the lenses are micro-OLED panels at 3,660×3,200 pixels per eye — 23 million pixels total — with a 7.5-micron pixel pitch. The resulting pixel density is high enough that the individual pixels are not visible at normal viewing distances, which was previously impossible in headset optics.² The outcome is that, at close range and in well-lit environments, the passthrough view of the real world is nearly indistinguishable from unmediated vision — fine text, surface textures, and spatial depth all render faithfully.

Apple frames this not as VR with passthrough but as "spatial computing": an environment in which physical and digital objects coexist on equal terms. The EyeSight feature extends this logic outward — a lenticular display on the front face of the headset renders a representation of the wearer's eyes, attempting to preserve social legibility when others are present.

At $3,499 at launch, Vision Pro is not a consumer mass-market product in the conventional sense. Its passthrough pipeline is a reference point for what the technology can achieve, rather than what it reaches.

Meta Quest 3

The Meta Quest 3 (launched October 2023 at $499) is the first consumer headset to bring full-colour mixed reality passthrough to a mainstream price point.⁴ Its predecessor, the Quest 2, had black-and-white passthrough useful primarily for safety checks; the Quest 3 replaces this with two RGB colour cameras delivering 18 pixels per degree — sufficient to read text and recognise faces in the passthrough view, though perceptibly lower fidelity than Vision Pro.³

The Quest 3 pairs its passthrough cameras with an active depth projector — a structured infrared sensor that builds a real-time mesh of the physical environment. This mesh enables scene understanding: the headset knows where floors, walls, and furniture are, so virtual objects can sit on real surfaces, bounce off real walls, and be occluded behind real geometry. The Snapdragon XR2 Gen 2 processor handles the compositing pipeline. The result, at its best, is persistent mixed reality — a virtual plant sitting on a real table, visibly behind the table leg when viewed from the right angle — in a device that costs less than most smartphones.

The Scale Inversion: Mobile AR

Both Vision Pro and Quest 3 represent the sophisticated end of passthrough mixed reality. But in terms of the number of people actually experiencing AR-style passthrough every day, dedicated headsets are a rounding error.

In 2024 there were an estimated 1.03 billion mobile AR users worldwide — people using phones to overlay digital content onto camera views of the real world.⁵ The global installed base of AR-capable smartphones exceeds 1.7 billion devices. Against that, the total number of XR headsets sold across all manufacturers and all years sits in the low tens of millions.

The phone as passthrough device lacks the stereoscopic depth, the peripheral-vision coverage, the world-locking precision, and the hands-free nature of a headset. What it has is ubiquity. The camera is already in every pocket; the AR experience requires no special hardware, no setup, and — with WebAR — no app installation at all. The user points their phone at a product, a place, or a face; the world behind the screen becomes the canvas.

This is not a lesser form of mixed reality. It is a different one, optimised for different constraints — and for most commercial applications, it is the form that actually reaches people.

A phone camera scanning a wine label to trigger a Living Wine Labels augmented reality experience

Commercial AR and the Living Wine Label

The clearest demonstration of mobile passthrough's reach at consumer scale is the category it created: AR-activated product packaging. The mechanic is simple — a phone camera scans a physical label, the image is recognized, and virtual content (animation, video, characters, information) is composited over the camera view in real time. No headset. No depth sensor. No spatial anchor. Just image recognition, a camera feed, and a renderer.

Tactic, an XR and AR studio specialising in product-first immersive experiences, built some of the defining examples of this format.⁷ Their most significant work was a series of animated characters for 19 Crimes, a Treasury Wine Estates brand whose labels feature historical convicts. Pointing the Living Wine Labels app at a 19 Crimes bottle caused the figure on the label to animate — turning to face the camera, speaking directly to the viewer, narrating the story of their crime and conviction. The app covered multiple wines in the Treasury portfolio including Beringer, Matua, Squealing Pig, Wolf Blass, and Lindeman's, alongside the 19 Crimes range.

Treasury Wine Estates described Living Wine Labels as "the wine industry's most loved augmented reality app, and a true innovation within the industry globally."⁶ The campaign won an Effie Award — the advertising industry's effectiveness metric — for measurable impact on brand engagement and purchase intent.⁸

Tactic extended the format across multiple clients: Rabble Wines (native app, five AR experiences across the product line), Aviana Wine (web-based AR, activating on every bottle without an app install), Jack Daniels, and White Claw (featuring Charli XCX). Snoop Dogg's involvement with 19 Crimes produced a dedicated experience for Snoop Cali Red, accessible at calibysnoop.com, which has outlasted the main Living Wine Labels native app — the broader TWE app was decommissioned in March 2023 as AR experiences migrated to WebAR, while Snoop and Martha Stewart's continued celebrity sustained the 19 Crimes experiences on the web.⁶

The trajectory of Tactic's work encodes the broader evolution of consumer AR delivery: native app (2015–2023) → WebAR (2023–present). WebAR removes the final barrier — the app store download — by running AR experiences directly in the mobile browser via the WebXR Device API.¹⁰ A user scans a QR code on a bottle, a menu, or a print advertisement; their browser opens; the camera activates; the experience runs. No installation. No account. No friction between the physical trigger and the digital content.

The Convergence

Headset passthrough (Vision Pro, Quest 3) and mobile camera AR (Tactic's wine labels, ARKit experiences, WebAR) are technically distinct but drawing on the same stack: camera input, image recognition or plane detection, a scene model, and a real-time compositor that places virtual objects within a live camera view.

The divergence is in the depth of immersion and the breadth of reach. Headsets provide stereo depth, peripheral coverage, world-locked persistence, and hands-free operation — immersive enough that the user's physical context is fully mediated. Mobile provides none of those things, but reaches a billion people with no barrier to entry.

For commercial applications — brand experiences, packaging activation, retail AR, tourism — mobile passthrough wins on reach by orders of magnitude. For industrial, medical, or spatial computing applications where precision, persistence, and hands-free operation matter, headset passthrough wins on capability. The most interesting space is in the middle: where WebAR's frictionlessness approaches headset-quality immersion as phone cameras, on-device ML, and browser rendering engines continue to improve.