nerfstudio-gnt

Unofficial GNT (Generalizable NeRF Transformer) integration for Nerfstudio.

Install

uv pip install -e .

Verify method registration

ns-train --help | grep -i gnt

Dataset format

This integration uses Nerfstudio's native VanillaDataManager with NerfstudioDataParserConfig. Point the dataparser to a scene root containing transforms.json.

Test-Time Adaptation & Fine-Tuning Taxonomy

GNT supports formal test-time adaptation (TTA) strategies across Full Fine-Tuning (FFT), Partial Fine-Tuning (PFT), and Parameter-Efficient Fine-Tuning (PEFT/LoRA) via --pipeline.model.adaptation_strategy:

1. Zero-Shot Generalization (Default)

Runs test-time evaluation without updating model weights:

ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy zero-shot

2. Full Fine-Tuning (FFT)

Updates both the 2D feature network and GNT transformer rays:

ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy full-ft \
  --optimizers.feature_net.optimizer.lr 5e-5 \
  --optimizers.network.optimizer.lr 5e-4

3. Partial Fine-Tuning (PFT)

PFT-Decoder (Freeze 2D Feature Extractor)

Keeps the ResUNet backbone frozen and adapts only the ray transformer synthesis network:

ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy pft-decoder

PFT-Backbone (Freeze Neural Renderer)

Optimizes the 2D feature extractor for distinct domain shifts (e.g. histopathology, microscopy) while freezing rendering:

ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy pft-backbone

BitFit (Bias-Only Fine-Tuning)

Freezes all weight matrices and updates only additive bias terms:

ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy bitfit

4. Parameter-Efficient Fine-Tuning (PEFT with LoRA)

Injects low-rank trainable decomposition matrices $A$ and $B$ into attention projections and linear layers while keeping base weights 100% frozen:

$$W = W_0 + \frac{\alpha}{r} (B \cdot A)$$

ns-train gnt \
  --pipeline.datamanager.dataparser.data /path/to/scene_root \
  --pipeline.model.transfer_learning True \
  --pipeline.model.pretrained_ckpt_path /path/to/gnt_pretrained.ckpt \
  --pipeline.model.adaptation_strategy peft-lora \
  --pipeline.model.lora_rank 4 \
  --pipeline.model.lora_alpha 8.0 \
  --pipeline.model.lora_target_modules '["q_proj", "v_proj", "fc"]'

Zero-Overhead Inference (Weight Merging)

Before evaluation or deployment, call model.merge_lora_weights() to fold the adapter update directly into the base weights ($W_0 \leftarrow W_0 + \Delta W$). This guarantees zero FPS overhead during rendering. To resume fine-tuning, call model.unmerge_lora_weights().

# In Python
model.merge_lora_weights()  # Merges delta W into base weights (zero runtime overhead)
# ... run high-FPS inference ...
model.unmerge_lora_weights()  # Restores separate adapters if continuing training

Console Diagnostic Logs

When initializing any adaptation strategy, GNT logs precise parameter accounting to the console:

==============================================================================
 [Few-Shot NeRF] Adaptation & Fine-Tuning Configuration
==============================================================================
 Model:               GNTModel
 Strategy:            PEFT-LORA (peft-lora)
 Total Parameters:    8,948,772
 Trainable Parameters:11,824
 Trainable Ratio:     0.1321% (peft-lora: 0.13% trainable)
 Configuration:       PEFT-LoRA (rank=4, alpha=8.0, targets=['q_proj', 'v_proj', 'fc']): 37 linear layers adapted.
==============================================================================

Smoke Check

python test.py /path/to/scene_root/transforms.json