als-finder

A high-performance, cloud-native CLI engine for discovering, streaming, and partitioning raw LiDAR point cloud data across the globe.

als-finder gathers complete acquisition footprints (project boundaries), true WGS84 point densities, and metadata from USGS 3DEP (EPT/COPC), NOAA Digital Coast, NASA G-LiHT, NEON, and OpenTopography into clean .json manifests, QGIS-ready .gpkg tables, and on-demand metric spatial tiles with automated Out-Of-Memory (OOM) protection.

🚀 Quick Install

Because als-finder uses advanced C++ geospatial libraries (GDAL, PDAL, GEOS), Conda or Docker is the recommended installation method:

# 1. Create environment with all C++ and Python dependencies
conda create -n als-finder -c conda-forge python=3.11 geopandas pdal python-pdal pystac stac-validator psutil shapely pyproj tqdm pyogrio requests click python-dotenv laspy -y

# 2. Activate environment & install als-finder
conda activate als-finder
pip install git+https://github.com/cms-2024-hudak/als-finder.git

🧭 The Big-Data Architecture: The 4-Stage ALS Lifecycle

Traditional tools force researchers to download massive, multi-gigabyte survey archives before doing any work. als-finder is built from the ground up for cloud-native lazy evaluation and HPC job arrays:

┌─────────────────┐     ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  1. DISCOVERY   │ ──> │   2. PLANNING    │ ──> │  3. ACQUISITION  │ ──> │   4. DERIVATIVE  │
│    (search)     │     │     (plan)       │     │     (fetch)      │     │    & MOSAIC      │
└─────────────────┘     └──────────────────┘     └──────────────────┘     └──────────────────┘
 • Federated APIs        • Spatial tiling index   • On-demand streaming    • Crop buffers
 • ROI & temporal        • Memory risk audit      • OOM auto-quadrants     • gdalbuildvrt
 • Provenance catalog    • Slurm task lists       • Sidecar crop bounds    • Zero edge-effects

⚡ Master Tutorial: End-to-End Walkthrough

This single, comprehensive tutorial walks through the complete lifecycle using a single workspace: scratch/test_workspace.

Step 0: Extract the Bundled Example Area of Interest (AOI)

als-finder comes with a bundled Lake Tahoe Basin Management Unit (LTBMU) boundary polygon so you can follow along immediately:

# Extract the example polygon (ltbmu_boundary.gpkg) to your current directory
als-finder get-example-roi

Step 1: Federated Discovery (als-finder search)

Search across all public LiDAR registries intersecting your AOI. Using --cloud-native filters exclusively for streaming-ready formats (USGS EPT / COPC), and --dedup eliminates duplicate survey records:

als-finder search \
  --roi ./ltbmu_boundary.gpkg \
  --workspace scratch/test_workspace \
  --cloud-native \
  --dedup

Console Output:

=================================================================================================================
 LiDAR Data Search Results 
=================================================================================================================
 | Provider        | Name                                   | Date         |   Est (GB) |   pts/m2 |   Area km2 |
-----------------------------------------------------------------------------------------------------------------
 | USGS_EPT        | CA_SierraNevada_5_2022                 | 2022-??-??   |    1380.20 |  29.1800 |    6347.47 |
 | USGS_EPT        | CA_SierraNevada_6_2022                 | 2022-??-??   |    1136.46 |  26.2100 |    5819.74 |
 | USGS_EPT        | CA_SierraNevada_8_2022                 | 2022-??-??   |    1171.62 |  25.1400 |    6255.39 |
 | NASA_GLIHT      | NASA G-LiHT Sierra Nevada / Lake Tahoe | 2022-07-18   |      30.73 |  32.5000 |     126.90 |
 | USGS_EPT        | NV_WestCentralEarthMRI_3_2020          | 2020-??-??   |     433.16 |   5.3400 |   10890.04 |
 | USGS_EPT        | CA_UpperSouthAmerican_Eldorado_2019    | 2019-??-??   |    2075.29 |  43.1600 |    6454.20 |
 | NOAA_STAC       | DigitalCoast_DAV:id_9452               | 2019-10-21   |    2075.29 |  46.7700 |    5954.91 |
 | USGS_EPT        | USGS_LPC_CA_NoCAL_Wildfires_B1_2018    | 2018-??-??   |     643.56 |  10.8900 |    7928.51 |
 | NOAA_STAC       | DigitalCoast_DAV:id_9036               | 2018-07-07   |     253.48 |   0.7800 |   43731.68 |
 | USGS_EPT        | USGS_LPC_NV_Reno_Carson_QL1_2017_LAS_2 | 2017-??-??   |     151.15 |   9.5400 |    2126.64 |
 | USGS_EPT        | CA_PlacerCo_2012                       | 2012-??-??   |      36.96 |   3.9500 |    1254.54 |
=================================================================================================================
 TOTAL DATASETS: 11 | ESTIMATED PAYLOAD: 9426.32 GB | QUERY TIME: 4.31s 
-----------------------------------------------------------------------------------------------------------------
 CATALOG TBL: scratch/test_workspace/catalog/catalog.gpkg
 JSON METADATA: scratch/test_workspace/catalog/manifest.json
=================================================================================================================

Step 2: Spatial Planning & Memory Risk Audit (als-finder plan)

Before launching jobs, use als-finder plan to inspect the study area's metric grid and audit memory safety against a target point budget (e.g. 4,000,000 points $\approx$ 4 GB RAM):

als-finder plan \
  --workspace scratch/test_workspace \
  --tile-size 500 \
  --buffer-size 50 \
  --max-points 4000000

Console Output:

==================================================
 ALS-FINDER SPATIAL PLANNING & GRID METRICS
==================================================
  Master Tiles:      5,657
  Total Leaf Tasks:  22,628
  Tile ID Range:     0 to 5656
  Tile Size:         500m (core)
  Buffer Size:       50m (overlap)
  Grid CRS:          EPSG:32610
  Target Tile ID:    0
  Basename:          CA_SierraNevada_5_2022_tile_E0737500_N4329500
  Hive Directory:    provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50
  Crop PDAL Bounds:  ([737500.0, 738000.0], [4329000.0, 4329500.0])
  Crop GDAL -te:     737500.0 4329000.0 738000.0 4329500.0
  Grid File:         scratch/test_workspace/catalog/grids/tilesize=500/buffer=50/grid.gpkg
--------------------------------------------------
 MEMORY RISK AUDIT (PRE-FLIGHT):
  Max Points Budget: 4,000,000
  Est Points/Tile:   10,504,800
  Subdivision:       5657 tiles exceed budget (split into 4 quadrants each)
  Slurm Array Size:  --array=1-22628
==================================================

Extracting Fields Directly for Shell Scripts (--field and --format env):

# Query exact scalar fields:
als-finder plan --workspace scratch/test_workspace --tile-size 500 --buffer-size 50 --field total_tiles
# Output: 5657

# Export full environment variables into your active shell:
eval $(als-finder plan --workspace scratch/test_workspace --tile-id 15 --tile-size 500 --buffer-size 50 --format env)
echo "Target Basename: $BASENAME"
echo "Crop Bounds: $CROP_GDAL_TE"

Step 3: Generating HPC Slurm Task Lists & Manifests (--tasks / --tasks-csv)

When scheduling Slurm job arrays, you can output either a plain leaf tile list (--tasks) or a complete CSV manifest (--tasks-csv) with spatial coordinates, Hive paths, point counts, and recommended Slurm memory allocations:

# 1. Plain leaf task list for simple Slurm array indexes:
als-finder plan \
  --workspace scratch/test_workspace \
  --tasks > slurm_tasks.txt

# 2. Comprehensive CSV manifest for Slurm arrays, HPC job schedulers, and tracking:
als-finder plan \
  --workspace scratch/test_workspace \
  --tasks-csv slurm_manifest.csv

Manifest Columns in slurm_manifest.csv:

• task_id: Unique leaf task identifier (0, 15_NW, etc.)
• tile_id: Parent master tile integer
• basename: Standardized spatial basename (<dataset>_tile_E<easting>_N<northing>[_quad])
• hive_dir & hive_path: Deterministic Hive partition directory and destination file path
• core_minx, core_miny, core_maxx, core_maxy: Core bounding coordinates
• point_density, est_points: Acquisition point density ($pts/m^2$) and pre-flight point count
• recommended_mem_gb: Pre-calculated Slurm job memory allocation ($\approx 1.5 \times$ point payload)

Step 4: On-Demand Point Cloud Streaming (als-finder fetch tile)

Acquire standard tile 15 directly over HTTP using clean positional syntax. Adding --sidecar writes a companion .json metadata file:

als-finder fetch tile 15 \
  --workspace scratch/test_workspace \
  --tile-size 500 \
  --buffer-size 50 \
  --sidecar \
  --overwrite

Console Output:

Successfully streamed tile 15 to scratch/test_workspace/data/tiles/provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50/CA_SierraNevada_5_2022_tile_E0738000_N4331000.laz
Wrote metadata sidecar to scratch/test_workspace/data/tiles/provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50/CA_SierraNevada_5_2022_tile_E0738000_N4331000.json

Step 5: Automatic OOM Protection & Quadrant Streaming

If a tile is hyper-dense and would exceed your RAM limit, passing --max-points causes als-finder to automatically bisect it into 4 safe quadrants:

als-finder fetch tile 15 \
  --workspace scratch/test_workspace \
  --tile-size 500 \
  --buffer-size 50 \
  --max-points 4000000 \
  --sidecar \
  --overwrite

Console Output:

Tile 15 exceeded budget. Subdivided into 4 quadrants:
  - Streamed 15_NW to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_NW.laz
  - Streamed 15_NE to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_NE.laz
  - Streamed 15_SW to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_SW.laz
  - Streamed 15_SE to .../CA_SierraNevada_5_2022_tile_E0738000_N4331000_SE.laz

Step 6: Multi-Tile Batch Streaming

Stream multiple tiles in a single concise command:

als-finder fetch tile 14 16 \
  --workspace scratch/test_workspace \
  --tile-size 500 \
  --buffer-size 50 \
  --sidecar \
  --overwrite

Console Output:

Successfully processed 2 target tile(s) (2 file(s) generated):
  - Tile 14 -> .../CA_SierraNevada_5_2022_tile_E0738000_N4330500.laz
  - Tile 16 -> .../CA_SierraNevada_5_2022_tile_E0738000_N4331500.laz

Step 7: Downstream Unbuffering & 1-Line Mosaicking

Because every .laz file includes a companion .json sidecar storing exact unbuffering crop parameters (crop_gdal_te / crop_pdal_bounds), downstream processing and mosaicking are completely automated:

TILES_DIR="scratch/test_workspace/data/tiles/provider=USGS_EPT/dataset=CA_SierraNevada_5_2022/tilesize=500/buffer=50"

# 1. Process point clouds and crop away the 50m buffer
for LAZ in "$TILES_DIR"/*.laz; do
    SIDECAR="${LAZ%.laz}.json"
    CROP_BOX=$(jq -r '.crop_gdal_te' "$SIDECAR")

    # Example: compute Canopy Height Model (CHM) with buffer, then crop to core:
    # my_chm_tool "$LAZ" temp.tif
    # gdal_translate -projwin $CROP_BOX temp.tif "${LAZ%.laz}_chm.tif"
done

# 2. Build a seamless mosaic across the entire study area (works for both 500m and 250m tiles):
# gdalbuildvrt full_study_area_chm.vrt "$TILES_DIR"/*_chm.tif

Native lidR (R) Integration & Buffer Point Tagging

Streamed .laz tiles embed a native LAS 1.4 buffer extra dimension (buffer=0 for points inside the core tile, buffer=1 for boundary buffer overlap points). In R using lidR, you can filter away buffer points after edge-artifact-free metric calculation with zero geometry math:

library(lidR)

# Read LAS tile with extra dimensions:
las <- readLAS("tile.laz", select = "xyzc", filter = "")

# Compute metrics (e.g., CHM, canopy cover) with buffer points preventing edge voids:
# ...

# Instant zero-copy filter of core points only:
core_pts <- filter_poi(las, buffer == 0)

Step 8: High-Throughput Slurm Job Array Recipe

Here is the production Slurm script for running on-demand streaming across hundreds of HPC cores with zero OOM risk and instant scratch cleanup:

#!/usr/bin/env bash
#SBATCH --job-name=als_tiles
#SBATCH --array=0-5656
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=00:30:00

set -e

WORKSPACE="/shared/project/workspace"
FINAL_DIR="/shared/project/products/chm"
LOCAL_SCRATCH="$TMPDIR/als_work_${SLURM_ARRAY_TASK_ID}"
mkdir -p "$LOCAL_SCRATCH" "$FINAL_DIR"

# 1. Fetch on demand to local scratch (auto-subdivides if dense)
als-finder fetch tile "$SLURM_ARRAY_TASK_ID" \
    --workspace "$WORKSPACE" \
    --tile-size 500 \
    --buffer-size 50 \
    --max-points 4000000 \
    --output "$LOCAL_SCRATCH" \
    --sidecar \
    --overwrite

# 2. Process all files in scratch (whether 1 tile or 4 quadrants)
for LAZ in "$LOCAL_SCRATCH"/*.laz; do
    SIDECAR="${LAZ%.laz}.json"
    BASENAME=$(basename "$LAZ" .laz)
    CROP_BOX=$(jq -r '.crop_gdal_te' "$SIDECAR")

    # Run point cloud metric tool
    # compute_metric "$LAZ" "$LOCAL_SCRATCH/${BASENAME}_raw.tif"

    # Crop off buffer to permanent storage
    # gdal_translate -projwin $CROP_BOX "$LOCAL_SCRATCH/${BASENAME}_raw.tif" "$FINAL_DIR/${BASENAME}_chm.tif"

    # Immediate cleanup of raw point cloud
    rm -f "$LAZ" "$SIDECAR"
done

# Clean scratch
rm -rf "$LOCAL_SCRATCH"

Step 9: Bulk Offline Survey Archiving (als-finder fetch survey)

For researchers who prefer to pre-download the entire raw dataset locally for air-gapped or offline processing:

als-finder fetch survey \
  --workspace scratch/test_workspace \
  --name "CA_SierraNevada_5_2022" \
  --execute

Step 10: Workspace Maintenance (als-finder workspace)

Update catalog filters or clean interim scratch files:

# Update catalog criteria (e.g. restrict to QL1):
als-finder workspace update --workspace scratch/test_workspace --density QL1

# Clean scratch workspace:
als-finder workspace clean --workspace scratch/test_workspace

🐍 Programmatic Python API

You can also import and use als_finder directly in Python scripts and Jupyter Notebooks:

import als_finder

# 1. Query tile specification and unbuffering bounds
spec = als_finder.get_tile_spec(
    workspace="scratch/test_workspace",
    tile_id=15,
    tile_size=500,
    buffer_size=50
)

print(f"Basename: {spec['basename']}")
print(f"Grid CRS: {spec['grid_crs']}")
print(f"Crop GDAL -te: {spec['crop_gdal_te']}")

# 2. Inspect Central Core Area vs Buffered Footprint
core = spec['core_poly']
buffered = spec['buffered_poly']

print(f"Core Tile Dimensions: {round(core.bounds[2] - core.bounds[0])}m x {round(core.bounds[3] - core.bounds[1])}m")
print(f"Buffer Extension: {round((buffered.bounds[2] - core.bounds[2]))}m on all sides")

🏛️ Authorship & Citation

This software is released under the open-source MIT License. Copyright Jonathan Greenberg.

Project Authors & Contributors:

• Jonathan Greenberg (University of Nevada, Reno): Lead Developer and Core Project Architect.
• Andrew Hudak (US Forest Service): Advisory feedback and scientific computing grant alignment.
• Antigravity (Google DeepMind): AI Software Engineering Partner.