3D Gaussian Splatting

3D Gaussian Splatting is a technique for representing and rendering real-world scenes as a collection of three-dimensional Gaussian primitives — ellipsoidal "splats" with position, scale, rotation, opacity, and view-dependent colour — that can be rasterised at real-time frame rates on consumer GPUs. Introduced by Kerbl et al. at SIGGRAPH 2023 and immediately released as open-source code, it produced photorealistic rendering of captured scenes at 100+ FPS while matching or exceeding NeRF in visual quality, establishing itself as the dominant approach for real-time neural scene rendering within months of publication.¹

Representation

Where NeRF encodes a scene implicitly in the weights of a neural network, Gaussian Splatting represents it explicitly as a set of 3D Gaussians. Each Gaussian has:

Position (x, y, z): the centre of the Gaussian in world space
Covariance (3×3 matrix, stored as scale + rotation): the shape and orientation of the ellipsoid
Opacity $\alpha$: how opaque the Gaussian is
Spherical harmonic coefficients: encoding view-dependent colour (how the colour changes as you look at the Gaussian from different angles)¹

A typical scene is represented by one to six million Gaussians initialised from a sparse point cloud (produced by Structure-from-Motion on the input photographs using COLMAP) and then optimised through differentiable rendering to minimise the difference between rendered images and the training photographs.³

Rendering

To render a view, the 3D Gaussians are projected onto the image plane as 2D Gaussians ("splatted"), sorted by depth, and alpha-composited front-to-back. This is standard rasterisation — no ray marching, no neural network inference at render time — which is why it is fast. Modern GPUs are highly optimised for tile-based rasterisation; the same hardware that renders triangle meshes at high frame rates renders Gaussian splats efficiently.²

The open-source implementation achieves 100–200 FPS at 1080p on an NVIDIA RTX 3090, and optimised mobile implementations run at interactive rates on high-end phones.

Training

Training a Gaussian Splatting scene from a video or photograph set takes 20–40 minutes on a consumer GPU, compared to the hours required by standard NeRF (though Instant-NGP reduced NeRF training to seconds on specialised hardware). The training alternates between:

Rendering and loss: render images from training viewpoints, compute pixel loss against captured images
Gradient update: propagate gradients back to adjust Gaussian positions, shapes, opacities, and colours
Adaptive density control: split Gaussians that are too large or insufficiently cover a region; prune Gaussians with low opacity

The result is a file containing the Gaussian parameters — typically 50–500 MB for a room-scale scene.

Comparison with NeRF

Gaussian Splatting and NeRF address the same problem — novel view synthesis from photographs — through fundamentally different representations. NeRF's implicit MLP is compact and differentiable but slow at inference. Gaussian Splatting's explicit primitives are larger in memory but render in real time without any neural network computation at render time.⁴ For applications requiring real-time rendering — interactive XR, games, real-time telepresence — Gaussian Splatting has become the preferred approach. For applications where rendering speed is less critical and storage efficiency matters, NeRF variants remain competitive.

XR and Web Deployment

Gaussian Splatting files can be rendered in WebGL, enabling captured real-world scenes to be delivered and viewed in a browser without any plugin.⁵ This opens photorealistic 3D capture of real spaces to web-based AR and VR deployment — a consumer scanning their apartment and sharing a link others can explore in 3D, or a retail brand placing a photorealistic product scan in an AR try-before-buy experience without manual 3D modelling.

The combination of fast phone-based capture (via apps that process video into Gaussian splats), cloud training, and WebGL delivery represents a plausible path to democratised photorealistic 3D content creation for XR.