Skip to main content

🚀 acmenra-ds

Production-ready, type-safe dataset management, YOLO parsing, caching, statistics, and in-memory augmentation for computer vision pipelines

PyPI Python License

# Install dataset toolkit (pulls acmenra-cv, no heavy ML stack)
pip install acmenra-ds

# OR install with development extras (pytest, ddt, black, mypy)
pip install "acmenra-ds[dev]"

📦 Overview

acmenra-ds is the data layer of the Acmenra stack. It turns a YOLO data.yaml plus a folder of images/labels into typed objects (Config, Dataset, Split, Sample, Instance) with strict validation, layout autodetection, RAM caching, visualization, a dataset snapshot, and a composable augmentation pipeline.

It sits on top of acmenra-cv geometry (Box, Polygon, Obb, Instance) and rendering (Drawer, Style). The model itself stays out of this package.

| Module             | Purpose                                         | Key Features                                                                  |
| ------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------- |
| **`data`**         | Config, Dataset, Split, Sample, LabelCache      | DIRECT / REVERSE layouts, `health()`, mutation logging, O(1) label cache      |
| **`image`**        | Atomic image ingestion                          | `ImageParser.parse()` → `ndarray` or `None`, strict vs lenient                |
| **`label`**        | YOLO `.txt` parse / write                       | DETECT / SEGMENT / OBB, inverse `LabelParser` ↔ `LabelWriter`                 |
| **`statistics`**   | Read-only snapshot of a dataset                 | Class counts, split sizes, center heatmap. Does not keep the dataset         |
| **`augmentation`** | In-memory `(image, List[Instance])` pipeline    | `Transform` + `Compose`, geometric + photometric, no source mutation          |

All geometries stay in normalized coordinate space [0.0, 1.0]. Disk I/O is Split / LabelWriter. Augmentation never walks the filesystem. Statistics reads a split once in the constructor and then only adds already computed numbers.


✨ Key Features

🔹 Typed dataset contract

  • Config.from_yaml() — Ultralytics-style YAML, autodetection of DIRECT (train/images) vs REVERSE (images/train)
  • Leaf names — train_leaf / images_leaf / labels_leaf for non-standard folder names (trn, imgs, lbs)
  • Fail-fast setters — relative paths only, no .., unique class names, bool rejected as int
  • Mutation warnings — changing a field after construction is logged; prefer a new object

🔹 Dataset as one object

  • Dataset owns train / val / test as Split instances — not three separate loaders
  • health() — filesystem integrity across splits (missing images, missing labels)
  • visualize() — Box / Polygon / OBB via acmenra-cv.Drawer
  • Unlabeled frames — Sample.label_path is Optional; empty List[Instance] is valid

🔹 Dataset snapshot before training

  • Statistics(dataset) — image counts, annotation counts, and a center heatmap
  • labels_per_class — "all" plus every name from config.names. A declared class with no boxes is 0, so a class missing from test is visible
  • labels_heat_map — int64 grid (grid, grid), default 20×20. Row 0 is the top of the frame, column 0 is the left. "all" is the sum of the class grids
  • Per split and summed — stat.train / stat.test / stat.val are the split snapshots. stat.labels_per_class and stat.labels_heat_map add them. No second walk of the label files
  • Fail-fast — a missing label, a class name that is not in config.names, or a center outside [0.0, 1.0] raises. The exact edge 1.0 belongs to the last cell
  • Does not store nc or the dataset. Class names are the keys. nc is len(keys) - 1

all alone hides a class that lives only in the center, or a class that is half the dataset. The per-class grid is what shows that before the model is trained.

🔹 Augmentation that keeps labels in sync

  • Single contract: (image, List[Instance]) → (image, List[Instance])
  • p is rolled in __call__, not inside apply()
  • Geometric steps rebuild Instance via map_instance (DETECT / SEGMENT / OBB independently)
  • Photometric steps copy the label list, coordinates unchanged
  • No source mutation — safe for DataLoader workers and offline export

🔹 Training-loop friendly

  • LabelCache — parsed List[Instance] keyed by stem, optional max_size
  • is_strict — raise on the first bad file, or log and skip
  • Shared ImageParser / LabelParser / LabelWriter across splits

🧩 Module Documentation

🔷 data — Config, Dataset, Split, Sample, Cache

Single source of truth for paths, classes, and splits

Classes

Config — YOLO data.yaml container

  • from_yaml(): Loads YAML, resolves root to the YAML parent, autodetects layout.
  • get_train_images() / get_train_labels() (and val/test): layout-aware absolute paths.
  • names, nc, task, layout: class map, TaskType, DatasetLayout.
  • *_leaf: rename split / images / labels folders without rewriting YAML.

Dataset — Train + optional Val/Test

  • __init__(config, category, is_cache, is_strict): builds all configured splits.
  • train / val / test: Split accessors.
  • health(): aggregated missing-file report.
  • visualize(sample, style): draws the sample with the dataset task flags.
  • from_yaml(): factory that parses YAML then constructs Dataset.

Split — One of TRAIN / VAL / TEST

  • Iterable and indexable (for sample in dataset.train, dataset.train[i]).
  • get_instances(idx): cache hit or parse-on-read.
  • visualize(idx, style): draw by index.
  • health(): per-split filesystem check.

Sample — Image + optional label

  • stem, image_path, label_path, is_labeled().
  • to_dict() / from_dict() for manifests.

LabelCache — RAM cache of List[Instance]

  • store(), get(), __getitem__ (raises KeyError on miss), remove(), clear().
  • Copy-on-read so callers cannot corrupt cache entries.

DataSplit & DatasetLayout

  • DataSplit: TRAIN, VAL, TEST.
  • DatasetLayout: DIRECT, REVERSE.

🔷 image — ImageParser

Atomic read: ndarray or None

Classes

ImageParser

  • parse(path, strict): cv2.imread + min dimension checks.
  • Strict mode raises; lenient mode logs and returns None.
  • Same contract as LabelParser — one return value, no status tuples.

🔷 label — LabelParser / LabelWriter

Inverse pair for YOLO .txt

Classes

LabelParser

  • Task-aware: DETECT (cx cy w h), SEGMENT (flat polygon), OBB (cx cy w h angle_rad).
  • Validates class id in [0, nc) and coordinates in [0.0, 1.0].
  • Builds acmenra_cv.Instance with Box / Polygon / Obb.

LabelWriter

  • Inverse of the parser. Writes a List[Instance] to a .txt.
  • Use this for offline augmentation export — not the transform classes.

🔷 statistics — Statistics / SplitStatistics

One read of the labels. After that, sums only.

Classes

Statistics — train + optional test / val

  • __init__(dataset, grid=20): builds a SplitStatistics per existing split. The dataset is not stored.
  • train / test / val: split snapshots. A missing split is None.
  • get_split_distribution(): image counts all, train, test, val. Does not open a label file. A missing split is 0.
  • labels_per_class: annotation counts summed across splits. "all" plus every class name.
  • labels_heat_map: center grids summed across splits. Same keys. int64 (grid, grid).

SplitStatistics — one split

  • name, count: split id and image count. No label I/O.
  • labels_per_class: copy of "all" and one count per config.names entry. A declared class with no annotations is 0.
  • labels_heat_map: copy of "all" and one grid per class. Row 0 is the top, column 0 is the left. "all" is the sum of the class grids.
  • Raises if an instance has no label, the class name is not declared, the task is not DETECT / SEGMENT / OBB, or a center is outside [0.0, 1.0].

🔷 augmentation — Transform, Compose, geometric, photometric

In-memory only. Disk is Split + LabelWriter.

Classes

Transform — ABC

  • __call__(image, labels): validate → roll p → apply().
  • apply(): subclasses implement the actual op; must not mutate inputs.
  • Helpers: _as_image, _as_labels, _as_prob, _as_float, _as_int.

Compose

  • Non-empty List[Transform]. Each step rolls its own p.
  • Fail-fast: first invalid step raises.
  • transforms getter returns a shallow copy.

Geometric (acmenra_ds.augmentation.geometric)

  • HorizontalFlip — cv2.flip(..., 1), x' = 1 - x on Box / Polygon / OBB.
  • VerticalFlip — cv2.flip(..., 0), y' = 1 - y.
  • Resize — pixels to (H, W); normalized labels pass through.

Photometric (acmenra_ds.augmentation.photometric)

  • Brightness, ChannelBrightness, Blur, Noise, Pixelate, Gray.
  • Normalize — uint8 [0, 255] → float32 [0.0, 1.0]. Last step in a training loop. Do not imshow / JPEG it.

Geometry math (acmenra_ds.augmentation.geometry)

  • Pure functions: hflip_box, vflip_box, hflip_polygon, vflip_polygon, hflip_obb, vflip_obb, map_instance, rebuild_instance.
  • OBB flip negates the angle. No OpenCV. Used by the geometric transforms.

💡 Quick Start

from enum import IntEnum
import cv2
import numpy as np

from acmenra_cv import Style, Stroke, Fill, Drawer, TaskType
from acmenra_ds import Config
from acmenra_ds.data import Dataset
from acmenra_ds.statistics import Statistics
from acmenra_ds.augmentation import (
    Compose,
    HorizontalFlip,
    VerticalFlip,
    Brightness,
    Blur,
    Noise,
)


class Defect(IntEnum):
    inclusion = 0
    scratch = 1
    crack = 2


# 1. YAML → Config (DIRECT / REVERSE is detected)
config = Config.from_yaml("/path/to/dataset/data.yaml")
print(config)  # DatasetConfig(root=..., nc=..., layout=direct, splits=[...])

# 2. Dataset owns train / val / test
dataset = Dataset(config=config, category=Defect, is_cache=True, is_strict=False)
print(dataset.health())

# 3. Snapshot before training. Labels are read here, once.
stat = Statistics(dataset=dataset, grid=20)
print(stat.get_split_distribution())
print(stat.labels_per_class)             # {"all": N, "inclusion": n, ...}
print(stat.train.labels_per_class)       # same keys, this split only
print(stat.labels_heat_map["crack"].shape)  # (20, 20)

# 4. Iterate a split
style = Style(
    palette=np.random.randint(0, 255, (80, 3)).tolist(),
    stroke=Stroke(thickness=2, alpha=0.8),
    fill=Fill(alpha=0.2),
    show=True,
)

for sample in dataset.train:
    frame = dataset.visualize(sample=sample, style=style)
    cv2.imshow("sample", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

# 5. Online augmentation (training loop)
pipeline = Compose([
    HorizontalFlip(p=0.5),
    VerticalFlip(p=0.5),
    Brightness(min_factor=0.6, max_factor=1.4, p=0.8),
    Blur(max_ksize=5, p=0.4),
    Noise(max_ratio=0.03, p=0.4),
])

drawer = Drawer(style=style)
sample = dataset.train[0]
image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
labels = dataset.train.get_instances(0)

aug_image, aug_labels = pipeline(image, labels)

vis = drawer.draw_instances(
    frame=aug_image.copy(),
    instances=aug_labels,
    is_box=dataset.task == TaskType.DETECT,
    is_obb=dataset.task == TaskType.OBB,
    is_polygon=dataset.task == TaskType.SEGMENT,
)
cv2.imshow("augmented", vis)
cv2.waitKey(0)

Offline export (new directory only — never overwrite the source dataset):

from pathlib import Path
from acmenra_ds.label import LabelWriter

out_img = Path("/data/aug/train/images")
out_lbl = Path("/data/aug/train/labels")
out_img.mkdir(parents=True, exist_ok=True)
out_lbl.mkdir(parents=True, exist_ok=True)

writer = dataset.train.label_writer
for idx, sample in enumerate(dataset.train):
    image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
    labels = dataset.train.get_instances(idx)
    if image is None:
        continue
    aug_image, aug_labels = pipeline(image, labels)
    cv2.imwrite(str(out_img / f"{sample.stem}_aug.jpg"), aug_image)
    writer.write(out_lbl / f"{sample.stem}_aug.txt", aug_labels)

📋 Requirements

Installed with pip install acmenra-ds:

acmenra-cv>=0.3.2.0
PyYAML>=6.0
numpy>=1.21.0
opencv-python>=4.5.0

Development extras (pip install "acmenra-ds[dev]"):

pytest>=7.0.0
ddt>=1.6.0
black>=23.0.0
mypy>=1.0.0

Python >=3.9. No PyTorch in the core package — Split is a plain iterable; wrap it in torch.utils.data.Dataset yourself if you need a DataLoader.


🧪 Testing

# All tests
pytest tests/

# By module
pytest tests/data/
pytest tests/augmentation/

Coverage includes:

  • Type / range validation (wrong type, invalid values, valid values)
  • DIRECT and REVERSE layouts, custom leaves
  • to_dict / from_dict round-trips
  • Mutation logging after construction
  • Compose construction and setter contract
  • Geometric / photometric property lifecycle

statistics does not have a test suite yet.


🔐 License

This project is licensed under the GNU Affero General Public License v3 or later (AGPL-3.0-or-later). See the LICENSE file for details.



acmenra.studio — Building reliable vision systems for the edge. Every sample, every label, every frame.

Metadata

Release files for acmenra-ds 0.2.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for acmenra-ds 0.2.1.0
File Size Uploaded Provenance Actions
acmenra_ds-0.2.1.0.tar.gz 92.6 kB Without provenance Details for acmenra_ds-0.2.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for acmenra-ds 0.2.1.0
File Interpreter ABI Platform Actions
acmenra_ds-0.2.1.0-py3-none-any.whl Python 3 none any Details for acmenra_ds-0.2.1.0-py3-none-any.whl

Total release size: 190.3 kB

Release files / acmenra_ds-0.2.1.0.tar.gz

Download URL acmenra_ds-0.2.1.0.tar.gz
Size 92.6 kB
Tags Source
SHA-256 checksum
How to use checksums
b48012468ec91d9abc121df729f76609202f4d092cf8424091fb3a2342318a41
BLAKE2b-256 checksum
How to use checksums
e71e09f8dc195d281ba2ab38cb8be2cf59c8a926bed8cc3345430380156c7fec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.6

Release files / acmenra_ds-0.2.1.0-py3-none-any.whl

Download URL acmenra_ds-0.2.1.0-py3-none-any.whl
Size 97.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
85586051b0f3fd7d160c73222253b64a769cfac8bd638771e0874b4112d126d9
BLAKE2b-256 checksum
How to use checksums
aa4e8165449b978de78b29ed78b6e31fc2e352a928457e0cde8d885d22c3a3b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.6

Security for acmenra-ds 0.2.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page