nerfstudio-pixel-nerf

Unofficial pixelNeRF integration for Nerfstudio. A framework for training and rendering few-shot Neural Radiance Fields using image-conditioned feature extraction.

Install

uv pip install -e .

Verify method registration

ns-train --help | grep -i pixel-nerf

Dataset Format

This integration uses a custom pipeline to inject source views dynamically into the RayBundle metadata. It is compatible with the standard NerfstudioDataParserConfig. Point the dataparser to a scene root containing a valid transforms.json.

Test-Time Adaptation & Fine-Tuning Taxonomy

PixelNeRF supports formal test-time adaptation (TTA) strategies across Full Fine-Tuning (FFT), Partial Fine-Tuning (PFT), and Parameter-Efficient Fine-Tuning (PEFT/LoRA) via --pipeline.model.adaptation_strategy:

1. Zero-Shot Generalization (Default)

Runs test-time evaluation without updating model weights:

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy zero-shot

2. Full Fine-Tuning (FFT)

Updates both the ResNet encoder and NeRF MLPs:

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy full-ft \
  --optimizers.encoder.optimizer.lr 1e-5 \
  --optimizers.nerf.optimizer.lr 1e-4

3. Partial Fine-Tuning (PFT)

PFT-Decoder (Freeze 2D Feature Extractor)

Keeps the 2D ResNet encoder frozen and adapts only the neural synthesis MLPs:

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy pft-decoder

PFT-Backbone (Freeze Neural Renderer)

Optimizes the 2D feature extractor for distinct domain shifts (e.g. histopathology, microscopy) while freezing synthesis MLPs:

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy pft-backbone

BitFit (Bias-Only Fine-Tuning)

Freezes all weight matrices and updates only additive bias terms:

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy bitfit

4. Parameter-Efficient Fine-Tuning (PEFT with LoRA)

Injects low-rank trainable decomposition matrices $A$ and $B$ into the synthesis MLP layers while keeping base weights 100% frozen:

$$W = W_0 + \frac{\alpha}{r} (B \cdot A)$$

ns-train pixel-nerf \
  --data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/pixelnerf_pretrained.pth \
  --pipeline.model.adaptation_strategy peft-lora \
  --pipeline.model.lora_rank 4 \
  --pipeline.model.lora_alpha 8.0 \
  --pipeline.model.lora_target_modules '["q_proj", "v_proj", "fc"]'

Zero-Overhead Inference (Weight Merging)

Before evaluation or deployment, call model.merge_lora_weights() to fold the adapter update directly into the base weights ($W_0 \leftarrow W_0 + \Delta W$). This guarantees zero FPS overhead during rendering. To resume fine-tuning, call model.unmerge_lora_weights().

# In Python
model.merge_lora_weights()  # Merges delta W into base weights (zero runtime overhead)
# ... run high-FPS inference ...
model.unmerge_lora_weights()  # Restores separate adapters if continuing training

Console Diagnostic Logs

When initializing any adaptation strategy, PixelNeRF logs precise parameter accounting to the console:

==============================================================================
 [Few-Shot NeRF] Adaptation & Fine-Tuning Configuration
==============================================================================
 Model:               PixelNeRFModel
 Strategy:            PEFT-LORA (peft-lora)
 Total Parameters:    21,464,433
 Trainable Parameters:14,376
 Trainable Ratio:     0.0670% (peft-lora: 0.07% trainable)
 Configuration:       PEFT-LoRA (rank=4, alpha=8.0, targets=['q_proj', 'v_proj', 'fc']): 16 linear layers adapted.
==============================================================================

Smoke Check

python test.py