nerfstudio-pixel-nerf

Unofficial pixelNeRF integration for Nerfstudio. A framework for training and rendering few-shot Neural Radiance Fields using image-conditioned feature extraction.

Install

uv pip install -e .

Verify method registration

ns-train --help | grep -i pixel-nerf

Dataset Format

This integration uses a custom pipeline to inject source views dynamically into the RayBundle metadata. It is compatible with the standard NerfstudioDataParserConfig. Point the dataparser to a scene root containing a valid transforms.json.

Configure PixelNeRF

Basic Training (Full Network Training - Default)

By default, the entire network (both ResNet encoder and NeRF MLPs) is trainable (freeze_mode = "none"):

ns-train pixel-nerf --data /path/to/scene_root

Transfer Learning & Freezing Strategies

When using pretrained weights (or transferring priors across distinct image domains such as histology datasets), you can choose how to freeze the encoder via --pipeline.model.freeze_mode:

1. Full Training (none - Default):

   ns-train pixel-nerf \
     --data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode none
   

2. Freeze Late Layers (late): Keeps low-level stem and early layers (conv1, bn1, layer1, layer2) trainable to adapt to distinct low-level image distributions (e.g. histopathology, microscopy), while freezing high-level semantic layers (layer3, layer4):

   ns-train pixel-nerf \
     --data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode late
   

3. Freeze Early Layers (early): Freezes low-level feature extractors (conv1, bn1, layer1, layer2) and trains only high-level layers (layer3, layer4):

   ns-train pixel-nerf \
     --data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode early
   

4. Freeze All Encoder Weights (all): Freezes the entire feature extractor ResNet:

   ns-train pixel-nerf \
     --data /path/to/scene_root \
     --pipeline.model.transfer_learning True \
     --pipeline.model.pretrained_ckpt_path /path/to/pretrained.pth \
     --pipeline.model.freeze_mode all
   

Differential Learning Rates

You can configure different learning rates for the image encoder and NeRF MLPs directly via CLI optimizer flags:

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --optimizers.encoder.optimizer.lr 1e-5 \
  --optimizers.nerf.optimizer.lr 1e-4