nerf-toy

A 2D toy illustration of Neural Radiance Fields

Description • Features • Examples • References • License

Original	No Mapping	Basic Mapping	Fourier Feature Mapping

Description

nerf-toy is a 2D toy illustration of the Neural Radiance Fields. Memorizing a 2D image using an MLP is a nice problem to understand before going into solving the problem NeRF tackles i.e. memorising a 3D scene given multiple view points.

The task is to produce the (r, g, b) value of an image as a function of (x,y) image coordinates. Our model is a coordinate-based multilayer perceptron.

This is implemented using 2D convolutions with a kernel size of (1, 1), which act as "pointwise" convolutions. This is equivalent to a densely connected multilayer perceptron for each coordinate. Also, we use batchnorm to speed up convergence.

The model is trained with the following input mappings $\gamma (\mathbf{v})$ for comparison :

• No mapping: $\gamma(\mathbf{v})= \mathbf{v}$.

• Basic mapping: $\gamma(\mathbf{v})=\left[ \cos(2 \pi \mathbf{v}),\sin(2 \pi \mathbf{v}) \right]^\mathrm{T}$.

• Gaussian Fourier feature mapping: $\gamma(\mathbf{v})=\left[ \cos(2 \pi \mathbf B \mathbf{v}), \sin(2 \pi \mathbf B \mathbf{v}) \right]^\mathrm{T}$, where each entry in $\mathbf B \in \mathbb R^{m \times d}$ is sampled from $\mathcal N(0,\sigma^2)$

Features

• Transforms: Basic, Positional Encoding and Gaussian Fourier Feature
• Data loader for any input image, where filepath, image url or bytes can be passed as input
• Keras based NeRF toy model, which can be customised based on number of layer and output channels
• Metrics: PSNR , SSIM
• Custom training callback: PredictionVideoSaverCallback, PlotLossesAndMetricsCallback
• Utility functions to read and manipulate image

Examples

The demo notebook demonstrates the core idea with full model training from scratch.

Original	No Mapping	Basic Mapping	Fourier Feature Mapping

It can be observed that directly regressing the (x, y) image coordinates results in blurry reconstructions.

Instead, lifting the input pixel coordinates (x, y) to higher dimensions via transformations (e.g. gaussian fourier feature) makes it easier for network to learn high frequency functions in low dimensional domains. Training with transformed (x, y) coordinates shows dramatic improvements in the results and can preserve the sharp edges in the image.

References

1. NeRF Paper
2. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
3. Neural Tangent Kernel: Convergence and Generalization in Neural Networks
4. NeRF Video

License

MIT

Image Credits

• Unsplash
• POV-Ray Hall of Fame