NeRF

NeRF (Neural Radiance Field) is a technique for representing and rendering 3D scenes by encoding them as a continuous volumetric function approximated by a neural network, trained from a set of 2D photographs. Introduced by Mildenhall et al. at ECCV 2020, NeRF produced photorealistic novel view synthesis of real-world scenes at a quality level that had not been achievable with prior explicit 3D reconstruction methods, catalysing a wave of research in neural scene representations with direct implications for XR content creation.¹

How NeRF Works

A NeRF encodes a scene as a 5D continuous function: given a 3D position (x, y, z) and a 2D viewing direction, the function returns colour (r, g, b) and volume density. This function is represented by a multi-layer perceptron (MLP) — a fully connected neural network with positional encoding to handle high-frequency detail.

Training the network requires a set of images of the scene from known camera positions (typically captured from many angles), plus the corresponding camera intrinsics and extrinsics. The network is trained to minimise the difference between pixels rendered by volume rendering the NeRF and the actual captured photographs. After training, the network can synthesise the scene from any viewpoint not present in the training set — novel view synthesis.¹

Volume rendering integrates the colour and density values sampled along each camera ray: dense regions contribute their colour; sparse regions transmit light through. This produces realistic depth, soft shadows, and view-dependent effects (specular highlights that change with viewing angle) without any explicit geometry representation.

Original Limitations

The original NeRF was computationally expensive in both training and rendering. Training required 1–2 days on a high-end GPU for a single scene. Rendering a single 800×800 image required 30 seconds of computation. These constraints made the original NeRF unsuitable for real-time XR applications despite its visual quality.

Additionally, NeRF assumed a static scene captured under consistent lighting — it could not represent moving objects and failed on unconstrained photo collections (variable exposure, transient obstructions).

Accelerations and Extensions

Instant-NGP (Müller et al., SIGGRAPH 2022) introduced a multiresolution hash encoding that reduced training time from days to seconds and rendering to real-time on a modern GPU.² Instant-NGP made NeRF practically interactive: users could capture a short video on a phone, train a NeRF in under a minute, and render novel views in real time.

NeRF in the Wild (2021) extended the formulation to unconstrained photo collections — tourism images from the internet, captured at different times of day with different crowds — by adding per-image appearance embeddings and transient object modelling.³

Block-NeRF (2022) scaled NeRF to city blocks by decomposing large scenes into independently trained sub-networks, enabling photorealistic reconstruction of street-level environments from Street View imagery.⁴

Relationship to 3D Gaussian Splatting

3D Gaussian Splatting (SIGGRAPH 2023) emerged as an alternative neural scene representation that matched NeRF's visual quality while achieving 100+ FPS rendering on consumer GPUs through an explicit Gaussian primitive representation and rasterisation-based rendering.⁵ Gaussian Splatting has largely supplanted NeRF for applications requiring real-time rendering, though NeRF variants remain competitive for reconstruction quality in constrained settings.

XR Applications

NeRF and its successors address one of the core challenges of XR content creation: photorealistic capture of real environments. Traditional 3D scanning produces geometric meshes that require manual texturing and lighting work to appear realistic; NeRF-based methods produce view-consistent photorealistic captures directly from photographs. Applications include:

• Telepresence: capturing real spaces or people for presence in VR
• Set extension: placing virtual cameras in captured real environments for mixed live-action and virtual production
• Spatial asset creation: generating 3D assets from reference photos for AR placement
• Heritage documentation: archiving physical sites at photographic fidelity

See also: Companies & Research · 3D Gaussian Splatting · Scene Reconstruction · Depth Anything · History of XR