Skip to main content

🚀 acmenra-ds

Production-ready, type-safe dataset management, YOLO parsing, caching, and in-memory augmentation for computer vision pipelines

PyPI Python License

# Install dataset toolkit (pulls acmenra-cv, no heavy ML stack)
pip install acmenra-ds

# OR install with development extras (pytest, ddt, black, mypy)
pip install "acmenra-ds[dev]"

📦 Overview

acmenra-ds is the data layer of the Acmenra stack. It turns a YOLO data.yaml plus a folder of images/labels into typed objects (Config, Dataset, Split, Sample, Instance) with strict validation, layout autodetection, RAM caching, visualization, and a composable augmentation pipeline.

It sits on top of acmenra-cv geometry (Box, Polygon, Obb, Instance) and rendering (Drawer, Style). The model itself stays out of this package.

Module Purpose Key Features
data Config, Dataset, Split, Sample, LabelCache DIRECT / REVERSE layouts, health(), mutation logging, O(1) label cache
image Atomic image ingestion ImageParser.parse() → ndarray or None, strict vs lenient
label YOLO .txt parse / write DETECT / SEGMENT / OBB, inverse LabelParser ↔ LabelWriter
augmentation In-memory (image, List[Instance]) pipeline Transform + Compose, geometric + photometric, no source mutation

All geometries stay in normalized coordinate space [0.0, 1.0]. Disk I/O is Split / LabelWriter. Augmentation never walks the filesystem.


✨ Key Features

🔹 Typed dataset contract

  • Config.from_yaml() — Ultralytics-style YAML, autodetection of DIRECT (train/images) vs REVERSE (images/train)
  • Leaf names — train_leaf / images_leaf / labels_leaf for non-standard folder names (trn, imgs, lbs)
  • Fail-fast setters — relative paths only, no .., unique class names, bool rejected as int
  • Mutation warnings — changing a field after construction is logged; prefer a new object

🔹 Dataset as one object

  • Dataset owns train / val / test as Split instances — not three separate loaders
  • health() — filesystem integrity across splits (missing images, missing labels)
  • visualize() — Box / Polygon / OBB via acmenra-cv.Drawer
  • Unlabeled frames — Sample.label_path is Optional; empty List[Instance] is valid

🔹 Augmentation that keeps labels in sync

  • Single contract: (image, List[Instance]) → (image, List[Instance])
  • p is rolled in __call__, not inside apply()
  • Geometric steps rebuild Instance via map_instance (DETECT / SEGMENT / OBB independently)
  • Photometric steps copy the label list, coordinates unchanged
  • No source mutation — safe for DataLoader workers and offline export

🔹 Training-loop friendly

  • LabelCache — parsed List[Instance] keyed by stem, optional max_size
  • is_strict — raise on the first bad file, or log and skip
  • Shared ImageParser / LabelParser / LabelWriter across splits

🧩 Module Documentation

🔷 data — Config, Dataset, Split, Sample, Cache

Single source of truth for paths, classes, and splits

Classes

Config — YOLO data.yaml container

  • from_yaml(): Loads YAML, resolves root to the YAML parent, autodetects layout.
  • get_train_images() / get_train_labels() (and val/test): layout-aware absolute paths.
  • names, nc, task, layout: class map, TaskType, DatasetLayout.
  • *_leaf: rename split / images / labels folders without rewriting YAML.

Dataset — Train + optional Val/Test

  • __init__(config, category, is_cache, is_strict): builds all configured splits.
  • train / val / test: Split accessors.
  • health(): aggregated missing-file report.
  • visualize(sample, style): draws the sample with the dataset task flags.
  • from_yaml(): factory that parses YAML then constructs Dataset.

Split — One of TRAIN / VAL / TEST

  • Iterable and indexable (for sample in dataset.train, dataset.train[i]).
  • get_instances(idx): cache hit or parse-on-read.
  • visualize(idx, style): draw by index.
  • health(): per-split filesystem check.

Sample — Image + optional label

  • stem, image_path, label_path, is_labeled().
  • to_dict() / from_dict() for manifests.

LabelCache — RAM cache of List[Instance]

  • store(), get(), __getitem__ (raises KeyError on miss), remove(), clear().
  • Copy-on-read so callers cannot corrupt cache entries.

DataSplit & DatasetLayout

  • DataSplit: TRAIN, VAL, TEST.
  • DatasetLayout: DIRECT, REVERSE.

🔷 image — ImageParser

Atomic read: ndarray or None

Classes

ImageParser

  • parse(path, strict): cv2.imread + min dimension checks.
  • Strict mode raises; lenient mode logs and returns None.
  • Same contract as LabelParser — one return value, no status tuples.

🔷 label — LabelParser / LabelWriter

Inverse pair for YOLO .txt

Classes

LabelParser

  • Task-aware: DETECT (cx cy w h), SEGMENT (flat polygon), OBB (cx cy w h angle_rad).
  • Validates class id in [0, nc) and coordinates in [0.0, 1.0].
  • Builds acmenra_cv.Instance with Box / Polygon / Obb.

LabelWriter

  • Inverse of the parser. Writes a List[Instance] to a .txt.
  • Use this for offline augmentation export — not the transform classes.

🔷 augmentation — Transform, Compose, geometric, photometric

In-memory only. Disk is Split + LabelWriter.

Classes

Transform — ABC

  • __call__(image, labels): validate → roll p → apply().
  • apply(): subclasses implement the actual op; must not mutate inputs.
  • Helpers: _as_image, _as_labels, _as_prob, _as_float, _as_int.

Compose

  • Non-empty List[Transform]. Each step rolls its own p.
  • Fail-fast: first invalid step raises.
  • transforms getter returns a shallow copy.

Geometric (acmenra_ds.augmentation.geometric)

  • HorizontalFlip — cv2.flip(..., 1), x' = 1 - x on Box / Polygon / OBB.
  • VerticalFlip — cv2.flip(..., 0), y' = 1 - y.
  • Resize — pixels to (H, W); normalized labels pass through.

Photometric (acmenra_ds.augmentation.photometric)

  • Brightness, ChannelBrightness, Blur, Noise, Pixelate, Gray.
  • Normalize — uint8 [0, 255] → float32 [0.0, 1.0]. Last step in a training loop. Do not imshow / JPEG it.

Geometry math (acmenra_ds.augmentation.geometry)

  • Pure functions: hflip_box, vflip_obb, map_instance, rebuild_instance.
  • No OpenCV. Used by geometric transforms.

💡 Quick Start

from enum import IntEnum
import cv2
import numpy as np

from acmenra_cv import Style, Stroke, Fill, Drawer, TaskType
from acmenra_ds import Config
from acmenra_ds.data import Dataset
from acmenra_ds.augmentation import (
    Compose,
    HorizontalFlip,
    VerticalFlip,
    Brightness,
    Blur,
    Noise,
)


class Defect(IntEnum):
    inclusion = 0
    scratch = 1
    crack = 2


# 1. YAML → Config (DIRECT / REVERSE is detected)
config = Config.from_yaml("/path/to/dataset/data.yaml")
print(config)  # DatasetConfig(root=..., nc=..., layout=direct, splits=[...])

# 2. Dataset owns train / val / test
dataset = Dataset(config=config, category=Defect, is_cache=True, is_strict=False)
print(dataset.health())

# 3. Iterate a split
style = Style(
    palette=np.random.randint(0, 255, (80, 3)).tolist(),
    stroke=Stroke(thickness=2, alpha=0.8),
    fill=Fill(alpha=0.2),
    show=True,
)

for sample in dataset.train:
    frame = dataset.visualize(sample=sample, style=style)
    cv2.imshow("sample", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

# 4. Online augmentation (training loop)
pipeline = Compose([
    HorizontalFlip(p=0.5),
    VerticalFlip(p=0.5),
    Brightness(min_factor=0.6, max_factor=1.4, p=0.8),
    Blur(max_ksize=5, p=0.4),
    Noise(max_ratio=0.03, p=0.4),
])

drawer = Drawer(style=style)
sample = dataset.train[0]
image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
labels = dataset.train.get_instances(0)

aug_image, aug_labels = pipeline(image, labels)

vis = drawer.draw_instances(
    frame=aug_image.copy(),
    instances=aug_labels,
    is_box=dataset.task == TaskType.DETECT,
    is_obb=dataset.task == TaskType.OBB,
    is_polygon=dataset.task == TaskType.SEGMENT,
)
cv2.imshow("augmented", vis)
cv2.waitKey(0)

Offline export (new directory only — never overwrite the source dataset):

from pathlib import Path
from acmenra_ds.label import LabelWriter

out_img = Path("/data/aug/train/images")
out_lbl = Path("/data/aug/train/labels")
out_img.mkdir(parents=True, exist_ok=True)
out_lbl.mkdir(parents=True, exist_ok=True)

writer = dataset.train.label_writer
for idx, sample in enumerate(dataset.train):
    image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
    labels = dataset.train.get_instances(idx)
    if image is None:
        continue
    aug_image, aug_labels = pipeline(image, labels)
    cv2.imwrite(str(out_img / f"{sample.stem}_aug.jpg"), aug_image)
    writer.write(out_lbl / f"{sample.stem}_aug.txt", aug_labels)

📋 Requirements

Installed with pip install acmenra-ds:

acmenra-cv>=0.3.2.0
PyYAML>=6.0
numpy>=1.21.0
opencv-python>=4.5.0

Development extras (pip install "acmenra-ds[dev]"):

pytest>=7.0.0
ddt>=1.6.0
black>=23.0.0
mypy>=1.0.0

Python >=3.9. No PyTorch in the core package — Split is a plain iterable; wrap it in torch.utils.data.Dataset yourself if you need a DataLoader.


🧪 Testing

# All tests
pytest tests/

# By module
pytest tests/data/
pytest tests/augmentation/

Coverage includes:

  • Type / range validation (wrong type, invalid values, valid values)
  • DIRECT and REVERSE layouts, custom leaves
  • to_dict / from_dict round-trips
  • Mutation logging after construction
  • Compose construction and setter contract
  • Geometric / photometric property lifecycle

🔐 License

This project is licensed under the GNU Affero General Public License v3 or later (AGPL-3.0-or-later). See the LICENSE file for details.



acmenra.studio — Building reliable vision systems for the edge. Every sample, every label, every frame.

Metadata

Release files for acmenra-ds 0.2.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for acmenra-ds 0.2.0.1
File Size Uploaded Provenance Actions
acmenra_ds-0.2.0.1.tar.gz 87.1 kB Without provenance Details for acmenra_ds-0.2.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for acmenra-ds 0.2.0.1
File Interpreter ABI Platform Actions
acmenra_ds-0.2.0.1-py3-none-any.whl Python 3 none any Details for acmenra_ds-0.2.0.1-py3-none-any.whl

Total release size: 178.9 kB

Release files / acmenra_ds-0.2.0.1.tar.gz

Download URL acmenra_ds-0.2.0.1.tar.gz
Size 87.1 kB
Tags Source
SHA-256 checksum
How to use checksums
bf77edf8593944abbfebdb76c4b5e7452270224619cf8d2ebff8c1cb7de094a8
BLAKE2b-256 checksum
How to use checksums
8e3014b4f9601c7954cc27182d3301ff59cd51f43c72fd4ca10141714e6a8bc5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.6

Release files / acmenra_ds-0.2.0.1-py3-none-any.whl

Download URL acmenra_ds-0.2.0.1-py3-none-any.whl
Size 91.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
884abfe29a213df2757ca84596674a8ad9bda565fd6f6b51bfafbb2987d133ca
BLAKE2b-256 checksum
How to use checksums
438ecf72c4e6687ae363f1f2fe01bb33b9c2eed133f59a4c8b09ac1560b0305e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.6

Security for acmenra-ds 0.2.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page