Link Search Menu Expand Document (external link)

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

[Project Page] [Paper] [Code]

Authors

  • Ben Mildenhall
  • Pratul P. Srinivasan
  • Matthew Tancik
  • Jonathan T. Barron
  • Ravi Ramamoorthi
  • Ren Ng

Table of contents


What’s NeRF?

NeRF (Neural Radiance Fields)

Neural Radiance Fields (NeRF) is proposed in the ECCV 2020 paper, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” (an honorable mention for Best Paper). NeRF synthesizes the novel viewpoint of the scene, exploiting a collection of posed images (e.g. images with corresponding camera position and rotation). In the rendering step, NeRF synthesizes the RGB color of each novel viewpoint in terms of the appearance of the scene varied from the viewpoints.

nerf

NeRF model consists of small MLPs that learn the radiance distributions from each viewpoint. It uses the conventional volumetric rendering in training steps, minimizing the residual between synthesized and ground truth observed RGB value. Specifically, NeRF learns from the input with 3D location (x,y,z) and view direction(theta, phi) calculated from the posed images and ray sampling points, and MLP produces the emitted color and volume density (sigma).

From the effect of synthesizing the novel-view images, NeRF tremendously compresses the scene of the 3D geometry and appearance because the user can generate the novel-view scene, which is not possible in conventional video clips (collection of images). In short, around 30MB of NeRF weights generate the infinite image collection of 3D scenes, which has benefits for immersive experience VR and AR industry.

Scores

Blender

ScenePSNRSSIMLPIPSCheckpoint
chair34.930.97940.0293link
drums25.280.92920.0802link
ficus31.280.97180.0329link
hotdog37.160.98030.0356link
lego34.380.97310.0321link
materials30.450.95580.0555link
mic35.180.98870.0146link
ship29.950.87840.1613link

MS-Blender

ScenePSNRSSIMLPIPSCheckpoint
chair32.830.96850.0394link
drums25.240.92780.0760link
ficus30.230.97150.0311link
hotdog35.240.97910.0319link
lego31.450.96490.0381link
materials29.540.96610.0489link
mic32.200.98040.0346link
ship29.410.90160.1096link

LLFF

ScenePSNRSSIMLPIPSCheckpoint
fern25.190.80450.2597link
flower27.940.84670.1898link
fortress31.730.89500.1441link
horns28.030.85850.2207link
leaves21.170.71410.2892link
orchids20.290.64080.3210link
room32.960.95420.1598link
trex27.520.90090.2233link

Tanks and Temples

ScenePSNRSSIMLPIPSCheckpoint
M6018.270.64470.4851link
Playground21.680.67020.4945link
Train17.370.55810.5059link
Truck21.440.69540.4475link

LF

ScenePSNRSSIMLPIPSCheckpoint
Africa28.530.86850.3062link
Basket21.640.81460.3981link
Ship26.260.79100.3646link
Statue29.760.87450.3059link
Torch23.240.75420.3947link

NeRF-360-v2

ScenePSNRSSIMLPIPSCheckpoint
bicycle21.820.43060.5732link
bonsai29.030.83280.3242link
counter26.980.77080.3574link
garden23.640.56420.4292link
kitchen27.160.73670.3165link
room30.100.86030.3223link
stump22.930.48110.5531link

Shiny Blender

ScenePSNRSSIMLPIPSCheckpoint
ball27.180.93560.2088link
car26.420.91970.0710link
coffee30.640.96410.1330link
helmet27.610.93880.1436link
teapot45.370.99630.0133link
toaster22.510.88560.1407link