tree-distribution-shift 0.1.0

Tree Distribution Shift benchmark: export HF configs to COCO (train/id_test/ood_test + density buckets).

tree-distribution-shift

A pip-installable benchmark for Tree Distribution Shift — export HuggingFace dataset configs to COCO format on disk.

No git. No git-lfs. Works on clusters.

Data lives on the HF Hub: aadityabuilds/tree-distribution-shift

Getting started

pip install -U pip
pip install tree-distribution-shift

# See all available configs (queries HF Hub live)
tree-shift list

# Export a config to COCO format
tree-shift export --config intl_train_IN__ood_US --out ./data

That's it. No cloning, no LFS, no large downloads up front — data streams directly from the Hub.

Authentication (recommended)

Set an HF token for higher rate limits and faster downloads:

export HF_TOKEN=hf_...   # recommended

Or log in once with the CLI:

huggingface-cli login

You can also pass a token per-command:

tree-shift list --token hf_...
tree-shift export --config intl_train_IN__ood_US --out ./data --token hf_...

Export only the splits you need

tree-shift export \
  --config in_state_train_Karnataka__ood_Rajasthan \
  --out ./data \
  --splits train ood_test

Pin a revision for reproducibility

For papers and exact replication, pin a commit hash:

tree-shift export \
  --config in_state_train_Karnataka__ood_Rajasthan \
  --out ./data \
  --revision abc123def456

Python API

from tree_shift import list_configs, export_coco

print(list_configs())

export_coco(
    config="in_state_train_Karnataka__ood_Rajasthan",
    out_root="./data",
    splits=["train", "id_test", "ood_test"],
    streaming=True,   # default; avoids OOM on low-mem machines
    revision=None,     # or pin a commit hash
)

Output structure

./data/<config>/
  train/
    images/
      *.png
    annotations/
      instances_train.json
  id_test/
    images/
    annotations/
  ood_test/
    images/
    annotations/
  ood_same_density/
    images/
    annotations/
  ood_diff_density/
    images/
    annotations/

Annotation files follow the COCO format.

Dataset contract

Every example row in the HF dataset contains:

Column	Type	Description
image_id	int	Unique image identifier
filename	str	Image filename (e.g. 00001.png)
width	int	Image width in pixels
height	int	Image height in pixels
image_bytes	bytes	Raw image file contents
coco_annotations	str	JSON string — list of COCO annotation dicts
coco_categories	str	JSON string — list of COCO category dicts
country	str	Country code (e.g. IN, US)
state	str	State name
zone	str	Geographic zone
biome	str	Biome classification
density_category	str	Tree density bucket

Expected splits

• train — training set
• id_test — in-distribution test set
• ood_test — out-of-distribution test set
• ood_same_density — OOD test filtered to same density bucket as train
• ood_diff_density — OOD test filtered to different density bucket

Note: Some older configs may not include the ood_same_density / ood_diff_density splits.

Resume support

Exports are resume-safe. If a job is preempted or interrupted:

# Just re-run the same command — already-written images are skipped
tree-shift export --config intl_train_IN__ood_US --out ./data

Images that already exist on disk are not re-downloaded. The COCO annotation JSON is always regenerated to stay consistent with the full split.

Rate limits & performance

• Streaming is on by default — data is fetched shard-by-shard, avoiding OOM and memory-map errors.
• --no-streaming downloads the full split into an Arrow table first. This can OOM on large splits or cause mmap errors on low-memory machines. Use only if you have plenty of RAM.
• For faster downloads (when available), set the environment variable:

  export HF_HUB_ENABLE_HF_TRANSFER=1
  

• Exports call load_dataset once per split (not per example), keeping HF API calls minimal.
• If you hit 429 rate-limit errors, wait a few minutes and re-run — resume will skip already-written images.

Cache location (clusters/shared machines)

The exporter uses the standard HuggingFace cache. On clusters with limited home directory space, set a custom cache location on a large disk:

export HF_HOME=/path/to/big/disk/hf_cache

This prevents re-downloading and reduces load on the HF Hub.

Diagnostic command

If exports fail or behave unexpectedly, run the doctor command and share the output:

tree-shift doctor

Output includes package versions, environment variables, and Python version — helpful for debugging "works on my machine" issues.

Manifest file

Every export writes a manifest.json in the config output folder:

{
  "repo_id": "aadityabuilds/tree-distribution-shift",
  "config": "intl_train_IN__ood_US",
  "revision": "abc123def456",
  "splits": ["train", "ood_test"],
  "timestamp": "2026-02-17T04:20:00+00:00",
  "counts": {
    "train": {"images": 11783, "annotations": 181208},
    "ood_test": {"images": 2345, "annotations": 36000}
  }
}

This is essential for reproducibility: it records the exact revision, splits exported, and counts per split. Include manifest.json when sharing or publishing data exports.

API reference

list_configs(repo_id=..., revision=None) -> list[str]

Returns the list of available dataset configs from the Hub. No data is downloaded.

export_coco(config, out_root, repo_id=..., splits=None, revision=None, streaming=True) -> Path

Exports a config to COCO folders on disk.

Parameter	Default	Description
config	(required)	Name of the HF dataset config
out_root	(required)	Root output directory
splits	all 5 splits	Which splits to export
streaming	True	Stream data (recommended for low-mem machines)
revision	None	HF dataset revision / commit hash

Dependencies

This package is intentionally lightweight:

• datasets — HF dataset loading
• huggingface_hub — Hub API access
• orjson — fast JSON serialization
• tqdm — progress display

No torch, pycocotools, or opencv required.

Why this package?

• No git / git-lfs — works on clusters where git-lfs is painful.
• Streaming by default — avoids OOM and memory-map errors.
• Resume-safe — re-run after interruption without re-downloading.
• Uses HF cache — respects revisions, reproducible.
• Thin client — new configs on the Hub are visible immediately via list_configs() (no pip update needed).

Versioning policy

The pip package contains tools only. Data versions are pinned with --revision.

• 0.1.x — exporter stability, backward compatible CLI/API.
• Minor bump — when on-disk COCO layout or CLI flags change.
• Major bump — if split semantics change (unlikely).

New configs on the HF Hub are visible immediately via tree-shift list; no pip update required to access new data.

Maintenance

• Adding new configs/data → push to the HF dataset repo only.
• Package updates → only needed for new CLI flags or exporter behavior changes.
• Users see new configs automatically via tree-shift list.

Citation

If you use this dataset in research, please cite:

@misc{nalawade2025tree_distribution_shift,
  title={Tree Distribution Shift: A Benchmark for Out-of-Distribution Tree Detection},
  author={Nalawade, Aaditya},
  year={2025},
  howpublished={\url{https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift}}
}

License

MIT