nerfbaselines 0.0.13

A collection of NeRF baselines for benchmarking

NerfBaselines

NerfBaselines is a framework for evaluating and comparing existing NeRF methods. Currently, most official implementations use different dataset loaders, evaluation protocols, and metrics, which renders the comparison of methods difficult. Therefore, this project aims to provide a unified interface for running and evaluating methods on different datasets in a consistent way using the same metrics. But instead of reimplementing the methods, we use the official implementations and wrap them so that they can be run easily using the same interface.

DISCLAIMER: This project is at a very early stage of its development. Stay tuned!

Please visit the project page to see the results of implemented methods on dataset benchmarks.

Project Page + Results

Getting started

Start by installing the nerfbaselines pip package on your host system.

pip install nerfbaselines

Now you can use the nerfbaselines cli to interact with NerfBaselines.

WARNING: the default installation only installs the core nerfbaselines package which does not depend on either PyTorch or JAX. However, the LPIPS metric requires PyTorch to be installed and will be disabled otherwise. Similarly, if you install JAX and have a GPU available, the dataloading and evaluation will be faster as some parts of the pipeline will be moved to GPU. Therefore, we recommend installing the extras package by following the Advanced installation section.

The next step is to choose the backend which will be used to install different methods. At the moment there are the following backends implemented:

• docker: Offers good isolation, requires docker to be installed and the user to have access to it (being in the docker user group).
• apptainer: Similar level of isolation as docker, but does not require the user to have privileged access.
• conda (not recommended): Does not require docker/apptainer to be installed, but does not offer the same level of isolation and some methods require additional dependencies to be installed. Also, some methods are not implemented for this backend because they rely on dependencies not found on conda.
• python (not recommended): Will run everything directly in the current environment. Everything needs to be installed in the environment for this backend to work.

The backend can be set as the --backend <backend> argument or using the NB_BACKEND environment variable.

Advanced installation

The LPIPS metric requires PyTorch to be installed and will be disabled otherwise. Similarly, if you install JAX and have a GPU available, the dataloading and evaluation will be faster as some parts of the pipeline will be moved to GPU. In this section we describe how to install the packages required for LPIPS and accelerated dataloading. We recommend this as the default installation (unless there is a reason for not installing PyTorch or JAX). Select one of the following configurations:

• CPU-only install

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install jax[cpu]
pip install 'nerfbaselines[extras]'

• CUDA 11.8 install

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install jax[cuda11_pip]
pip install 'nerfbaselines[extras]'

• CUDA 12.1 install

pip install torch torchvision torchaudio
pip install jax[cuda12_pip]
pip install 'nerfbaselines[extras]'

Downloading data

For some datasets, e.g. Mip-NeRF 360 or NerfStudio, the datasets can be downloaded automatically. You can specify the argument --data external://dataset/scene during training or download the dataset beforehand by running nerfbaselines download-dataset dataset/scene. Examples:

# Downloads the garden scene to the cache folder.
nerfbaselines download-dataset mipnerf360/garden

# Downloads all nerfstudio scenes to the cache folder.
nerfbaselines download-dataset nerfstudio

# Downloads kithen scene to folder kitchen
nerfbaselines download-dataset mipnerf360/kitchen -o kitchen

Training

To start the training use the nerfbaselines train --method <method> --data <data> command. Use --help argument to learn about all implemented methods and supported features.

Rendering

The nerfbaselines render --checkpoint <checkpoint> command can be used to render images from a trained checkpoint. Again, use --help to learn about the arguments.

Interactive viewer

Given a trained checkpoint, the interactive viewer can be launched as follows:

nerfbaselines viewer --checkpoint <checkpoin> --data <dataset>

Even though the argument --data <dataset> is optional, it is recommended, as for some methods some viewers use the camera poses to performa gravity alignement for better viewing experience. Again, you can use the --backend <backend> flag or NS_BACKEND=<backend> environment variable to change the backend.

Currently, there are two viewers implemented: viser and nerfstudio. You can switch between the two by using the --viewer <viewer> flag.

Results

In this section we present results of implemented methods on standard benchmark datasets. For detailed results, visit the project page: https://jkulhanek.com/nerfbaselines

Mip-NeRF 360

Mip-NeRF 360 is a collection of four indoor and five outdoor object-centric scenes. The camera trajectory is an orbit around the object with fixed elevation and radius. The test set takes each n-th frame of the trajectory as test views. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/mipnerf360

Method	PSNR	SSIM	LPIPS	Time	GPU mem.
Zip-NeRF	28.516	0.828	0.138	5h 30m 49s	26.2 GB
Mip-NeRF 360	27.670	0.792	0.196	7h 29m 42s	127.0 GB
Gaussian Splatting	27.439	0.814	0.180	22m 45s	11.1 GB
Tetra-NeRF	25.468	0.670	0.352	17h 32m 35s	13.4 GB
Instant NGP	24.899	0.673	0.355	4m 16s	5.6 GB
NerfStudio (Nerfacto-huge)	23.776	0.736	0.226	2h 14m 34s	14.9 GB
NerfStudio	23.611	0.670	0.328	9m 2s	4.0 GB

Blender

Blender (nerf-synthetic) is a synthetic dataset used to benchmark NeRF methods. It consists of 8 scenes of an object placed on a white background. Cameras are placed on a semi-sphere around the object. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/blender

Method	PSNR	SSIM	LPIPS	Time	GPU mem.
Gaussian Splatting	33.308	0.969	0.023	6m 6s	3.1 GB
Instant NGP	32.191	0.959	0.031	2m 23s	2.6 GB
Tetra-NeRF	31.951	0.957	0.031	6h 53m 20s	29.6 GB
Mip-NeRF 360	30.345	0.951	0.038	3h 29m 39s	114.8 GB
NerfStudio	28.277	0.939	0.051	10m 34s	3.9 GB
NerfStudio (Nerfacto-huge)	26.375	0.931	0.057	3h 8m 18s	14.8 GB

Nerfstudio

Nerfstudio Dataset includes 10 in-the-wild captures obtained using either a mobile phone or a mirror-less camera with a fisheye lens. We processed the data using either COLMAP or the Polycam app to obtain camera poses and intrinsic parameters. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/nerfstudio

Method	PSNR	SSIM	LPIPS	Time	GPU mem.
NerfStudio	20.711	0.644	0.375	13m 30s	4.4 GB
Instant NGP	20.653	0.601	0.452	4m 33s	4.2 GB
NerfStudio (Nerfacto-big)	20.459	0.644	0.329	1h 48m 42s	17.8 GB
NerfStudio (Nerfacto-huge)	20.324	0.668	0.247	2h 32m 19s	17.5 GB
Gaussian Splatting	-	-	-	-	-

Implementation status

Methods:

• Nerfacto
• Instant-NGP
• Gaussian Splatting
• Tetra-NeRF
• Mip-NeRF 360
• Zip-NeRF
• CamP
• NeRF
• Mip-NeRF

Datasets/features:

• Mip-NeRF 360 dataset
• Blender dataset
• any COLMAP dataset
• any NerfStudio dataset
• automatic dataset download
• interactive viewer
• undistorting images for methods that do not support complex camera models (Gaussian Splatting)
• logging to tensorboard, wandb
• Tanks and Temples
• LLFF dataset
• HDR images support
• RAW images support
• handling large datasets
• loading/creating camera trajectories in the interactive viewer

Contributing

Contributions are very much welcome. Please open a PR with a dataset/method/feature that you want to contribute. The goal of this project is to slowly expand by implementing more and more methods.

License

This project is licensed under the MIT license.

Thanks

A big thanks to the authors of all implemented methods for the great work they have done.