ds-crawler

Build stable, queryable metadata for datasets whose structure is encoded in file paths.

ds-crawler scans directories or ZIP archives, extracts IDs, hierarchy, and properties with regular expressions, and writes a small set of versioned JSON artifacts. It also aligns modalities, creates reproducible splits, copies or extracts subsets, and indexes generated outputs.

• Directory and .zip inputs use the same API.
• Regex capture groups define IDs, hierarchy, and file properties.
• Include/exclude filters keep irrelevant files out of the index.
• Named splits and metadata scopes are first-class artifacts.
• Dataset semantics live in the shared euler-dataset-contract format.

Installation

pip install ds-crawler

Progress bars are optional:

pip install "ds-crawler[progress]"

Python 3.9 or newer is required.

Quick start

Given this dataset:

foggy_rgb/
├── .ds_crawler/
│   ├── dataset-head.json
│   └── ds-crawler.json
├── scene_01/
│   ├── 0001.png
│   └── 0002.png
└── scene_02/
    └── 0001.png

describe its semantic identity in .ds_crawler/dataset-head.json:

{
  "contract": {"kind": "dataset_head", "version": "1.0"},
  "dataset": {"id": "foggy_rgb", "name": "Foggy RGB"},
  "modality": {
    "key": "rgb",
    "meta": {"range": [0, 255]}
  },
  "addons": {
    "euler_train": {
      "version": "1.0",
      "used_as": "input",
      "slot": "dehaze.input.rgb"
    }
  }
}

Then describe how paths become index entries in .ds_crawler/ds-crawler.json:

{
  "contract": {"kind": "ds_crawler_config", "version": "2.0"},
  "head_file": "dataset-head.json",
  "source": {"path": "."},
  "indexing": {
    "id": {
      "regex": "^(?P<scene>[^/]+)/(?P<frame>\\d+)\\.png$",
      "join_char": "+"
    },
    "hierarchy": {
      "regex": "^(?P<scene>[^/]+)/",
      "separator": ":"
    },
    "properties": {
      "basename": {
        "regex": "^(?P<frame>\\d+)\\.(?P<ext>png)$"
      }
    },
    "files": {"extensions": [".png"]},
    "constraints": {"flat_ids_unique": true}
  }
}

Index and persist the dataset:

from ds_crawler import get_files, index_dataset_from_path

output = index_dataset_from_path("/data/foggy_rgb", save_index=True)

print(output["head"]["dataset"]["name"])
print(get_files(output))

The returned object is hydrated with the dataset head and indexing recipe for convenience. The saved index.json stays minimal.

For a self-contained example that creates a temporary dataset, see examples/basic_usage.py.

Artifact layout

For a physical root containing several logical modalities, prefer the scoped layout. Each modality gets a complete, isolated artifact set and scopes.json provides the discovery manifest:

foggy_rgb/
├── .ds_crawler/
│   ├── scopes.json
│   ├── rgb/
│   │   ├── dataset-head.json
│   │   ├── ds-crawler.json
│   │   ├── index.json
│   │   └── split_train.json
│   └── camera_extrinsics/
│       ├── dataset-head.json
│       ├── ds-crawler.json
│       └── index.json
└── ...

Pass metadata_scope="rgb" (or another safe modality name) to path-based APIs to read and write .ds_crawler/rgb/. Scoped writes maintain scopes.json automatically.

For a root with just one logical dataset, the compact unscoped layout remains supported:

foggy_rgb/
├── .ds_crawler/
│   ├── dataset-head.json
│   ├── ds-crawler.json
│   ├── index.json
│   └── split_train.json
└── ...

The files in either artifact set are:

See the configuration reference for every supported field, artifact shape, path filter, and hierarchy pattern.

Common workflows

Use the CLI

A CLI config can reference one or more datasets that already contain crawler metadata:

{
  "datasets": [
    {"path": "/data/foggy_rgb"},
    {"path": "/data/foggy_depth"}
  ]
}

ds-crawler index datasets.json

index is optional for backwards compatibility, so ds-crawler datasets.json is equivalent. Useful flags include:

Align modalities

from ds_crawler import align_datasets

aligned = align_datasets(
    {"modality": "rgb", "source": "/data/foggy_rgb"},
    {"modality": "depth", "source": "/data/foggy_depth"},
)

rgb_path = aligned["scene-scene_01+frame-0001"]["rgb"]["path"]

Alignment is by leaf ID. For modalities that apply at an ancestor hierarchy level, preserve the hierarchy and join in the consumer instead; the runnable augmented RGB example demonstrates that layout.

Create reproducible splits

from ds_crawler import create_dataset_splits, load_dataset_split

create_dataset_splits(
    "/data/foggy_rgb",
    split_names=["train", "validation"],
    ratios=[80, 20],
    seed=42,
)

train = load_dataset_split("/data/foggy_rgb", "train")

Use create_aligned_dataset_splits for several modalities, create_hierarchy_dataset_splits for hierarchy rules, or create_mapped_dataset_splits for an explicit qualified-ID mapping. copy_dataset_splits transfers an existing partition to a sibling dataset and fails before writing if any target ID is missing.

Write generated data

from ds_crawler import DatasetWriter

head = {
    "contract": {"kind": "dataset_head", "version": "1.0"},
    "dataset": {"id": "predicted_rgb", "name": "Predicted RGB"},
    "modality": {"key": "rgb", "meta": {"range": [0, 255]}},
}

writer = DatasetWriter("/data/predicted_rgb", head=head)
path = writer.get_path("/scene:scene_01/frame-0001", "0001.png")
path.write_bytes(encoded_image)
writer.save_index()

ZipDatasetWriter provides the same index-building behavior for a new ZIP archive and accepts bytes or writable file-like objects.

Migrate legacy metadata

Normal loading is intentionally strict about the current schema. Migrate older output.json-based datasets explicitly:

ds-crawler migrate-metadata /data/legacy_dataset
ds-crawler migrate-metadata /data/archive.zip
ds-crawler migrate-metadata /data/datasets --scan-zips

Run ds-crawler migrate-metadata --help for archive scanning, inline split, and metadata-scope options.

Python API

The main entry points are re-exported from ds_crawler:

All core path-based workflows support directories and ZIP archives. Scope-aware operations accept metadata_scope; multi-source operations also expose per-source scope arguments where needed.

Development

git clone https://github.com/d-rothen/ds-crawler.git
cd ds-crawler
uv sync --extra dev
uv run pytest
uv run ruff check .
uv build

Please read CONTRIBUTING.md before submitting a change. To report a vulnerability, follow SECURITY.md.

License

Licensed under the MIT License. Copyright © 2026 Daniel Rothenpieler rothenpielerdaniel@gmail.com.

See Phase 2 validated-publication for the opt-in captured spatial workflow (2.11.0, unreleased).