nps: distributed point cloud sampling

nps samples point clouds of every segment in a large segmentation volumes

• Compatible with CloudVolume (precomputed or graphene).
• It processes the volume blockwise on local, LSF or Slurm workers and writes the points in pocaduck format.

pip install nps-cli
nps --help

Usage

# Whole volume
nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation

# Subvolume (begin_x begin_y begin_z end_x end_y end_z, in voxels), 1% of each segment
nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation \
    --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01

# Also store the supervoxel ID of every point, looked up in a separate volume
# that shares the voxel grid of --cv-path
nps --cv-path precomputed://gs://<bucket>/agglomerated \
    --svid-cv-path precomputed://gs://<bucket>/supervoxels

# Graphene: read supervoxels from the same volume (agglomerate=False)
nps --cv-path graphene://https://<server>/segmentation/table/<table> --sample_svids

Key options (run nps --help for all of them):

Output

<output-dir>/
  block_mask.npy            # occupancy mask from the empty-block pre-scan
  points/
    unified_index.db        # pocaduck index over all workers
    worker_<id>/...         # parquet files + per-worker index

Each point is [x, y, z], or [x, y, z, svid] when SVIDs are sampled, stored under its segment label.

nps refuses to write into an --output-dir that already holds points, since every run re-samples all blocks and old points would show up as duplicates. Pass --overwrite to delete them first.

Skipping empty blocks

Before sampling, nps scans the ROI at a coarse mip (--mask-mip, default the coarsest one) and never dispatches blocks without labels. The mask is dilated by one block, so thin structures lost in downsampling are still sampled. Use --no-skip-empty to disable the scan, or --block-mask <path> to reuse a block_mask.npy from an earlier run with the same --bbox, --block-size and --mip.

Reading point clouds

from pocaduck import Query, StorageConfig

query = Query(storage_config=StorageConfig(base_path="<output-dir>/points"))
labels = query.get_labels()
points = query.get_points(label=12345)  # all points of one segment
query.close()

See the pocaduck docs for more, including optimizing the storage for fast reads.

Example scripts

Release

pip install -e ".[dev]"
python -m build
twine upload dist/*