🚀 acmenra-ds
Production-ready, type-safe dataset management, YOLO parsing, caching, and in-memory augmentation for computer vision pipelines
# Install dataset toolkit (pulls acmenra-cv, no heavy ML stack)
pip install acmenra-ds
# OR install with development extras (pytest, ddt, black, mypy)
pip install "acmenra-ds[dev]"
📦 Overview
acmenra-ds is the data layer of the Acmenra stack. It turns a YOLO data.yaml plus a folder of images/labels into typed objects (Config, Dataset, Split, Sample, Instance) with strict validation, layout autodetection, RAM caching, visualization, and a composable augmentation pipeline.
It sits on top of acmenra-cv geometry (Box, Polygon, Obb, Instance) and rendering (Drawer, Style). The model itself stays out of this package.
| Module | Purpose | Key Features |
|---|---|---|
data |
Config, Dataset, Split, Sample, LabelCache | DIRECT / REVERSE layouts, health(), mutation logging, O(1) label cache |
image |
Atomic image ingestion | ImageParser.parse() → ndarray or None, strict vs lenient |
label |
YOLO .txt parse / write |
DETECT / SEGMENT / OBB, inverse LabelParser ↔ LabelWriter |
augmentation |
In-memory (image, List[Instance]) pipeline |
Transform + Compose, geometric + photometric, no source mutation |
All geometries stay in normalized coordinate space [0.0, 1.0]. Disk I/O is Split / LabelWriter. Augmentation never walks the filesystem.
✨ Key Features
🔹 Typed dataset contract
Config.from_yaml()— Ultralytics-style YAML, autodetection ofDIRECT(train/images) vsREVERSE(images/train)- Leaf names —
train_leaf/images_leaf/labels_leaffor non-standard folder names (trn,imgs,lbs) - Fail-fast setters — relative paths only, no
.., unique class names,boolrejected asint - Mutation warnings — changing a field after construction is logged; prefer a new object
🔹 Dataset as one object
Datasetownstrain/val/testasSplitinstances — not three separate loadershealth()— filesystem integrity across splits (missing images, missing labels)visualize()— Box / Polygon / OBB viaacmenra-cv.Drawer- Unlabeled frames —
Sample.label_pathisOptional; emptyList[Instance]is valid
🔹 Augmentation that keeps labels in sync
- Single contract:
(image, List[Instance]) → (image, List[Instance]) pis rolled in__call__, not insideapply()- Geometric steps rebuild
Instanceviamap_instance(DETECT / SEGMENT / OBB independently) - Photometric steps copy the label list, coordinates unchanged
- No source mutation — safe for DataLoader workers and offline export
🔹 Training-loop friendly
LabelCache— parsedList[Instance]keyed by stem, optionalmax_sizeis_strict— raise on the first bad file, or log and skip- Shared
ImageParser/LabelParser/LabelWriteracross splits
🧩 Module Documentation
🔷 data — Config, Dataset, Split, Sample, Cache
Single source of truth for paths, classes, and splits
Classes
Config — YOLO data.yaml container
from_yaml(): Loads YAML, resolvesrootto the YAML parent, autodetectslayout.get_train_images()/get_train_labels()(and val/test): layout-aware absolute paths.names,nc,task,layout: class map,TaskType,DatasetLayout.*_leaf: rename split / images / labels folders without rewriting YAML.
Dataset — Train + optional Val/Test
__init__(config, category, is_cache, is_strict): builds all configured splits.train/val/test:Splitaccessors.health(): aggregated missing-file report.visualize(sample, style): draws the sample with the dataset task flags.from_yaml(): factory that parses YAML then constructsDataset.
Split — One of TRAIN / VAL / TEST
- Iterable and indexable (
for sample in dataset.train,dataset.train[i]). get_instances(idx): cache hit or parse-on-read.visualize(idx, style): draw by index.health(): per-split filesystem check.
Sample — Image + optional label
stem,image_path,label_path,is_labeled().to_dict()/from_dict()for manifests.
LabelCache — RAM cache of List[Instance]
store(),get(),__getitem__(raisesKeyErroron miss),remove(),clear().- Copy-on-read so callers cannot corrupt cache entries.
DataSplit & DatasetLayout
DataSplit:TRAIN,VAL,TEST.DatasetLayout:DIRECT,REVERSE.
🔷 image — ImageParser
Atomic read:
ndarrayorNone
Classes
ImageParser
parse(path, strict):cv2.imread+ min dimension checks.- Strict mode raises; lenient mode logs and returns
None. - Same contract as
LabelParser— one return value, no status tuples.
🔷 label — LabelParser / LabelWriter
Inverse pair for YOLO
.txt
Classes
LabelParser
- Task-aware: DETECT (
cx cy w h), SEGMENT (flat polygon), OBB (cx cy w h angle_rad). - Validates class id in
[0, nc)and coordinates in[0.0, 1.0]. - Builds
acmenra_cv.InstancewithBox/Polygon/Obb.
LabelWriter
- Inverse of the parser. Writes a
List[Instance]to a.txt. - Use this for offline augmentation export — not the transform classes.
🔷 augmentation — Transform, Compose, geometric, photometric
In-memory only. Disk is
Split+LabelWriter.
Classes
Transform — ABC
__call__(image, labels): validate → rollp→apply().apply(): subclasses implement the actual op; must not mutate inputs.- Helpers:
_as_image,_as_labels,_as_prob,_as_float,_as_int.
Compose
- Non-empty
List[Transform]. Each step rolls its ownp. - Fail-fast: first invalid step raises.
transformsgetter returns a shallow copy.
Geometric (acmenra_ds.augmentation.geometric)
HorizontalFlip—cv2.flip(..., 1),x' = 1 - xon Box / Polygon / OBB.VerticalFlip—cv2.flip(..., 0),y' = 1 - y.Resize— pixels to(H, W); normalized labels pass through.
Photometric (acmenra_ds.augmentation.photometric)
Brightness,ChannelBrightness,Blur,Noise,Pixelate,Gray.Normalize—uint8 [0, 255]→float32 [0.0, 1.0]. Last step in a training loop. Do notimshow/ JPEG it.
Geometry math (acmenra_ds.augmentation.geometry)
- Pure functions:
hflip_box,vflip_obb,map_instance,rebuild_instance. - No OpenCV. Used by geometric transforms.
💡 Quick Start
from enum import IntEnum
import cv2
import numpy as np
from acmenra_cv import Style, Stroke, Fill, Drawer, TaskType
from acmenra_ds import Config
from acmenra_ds.data import Dataset
from acmenra_ds.augmentation import (
Compose,
HorizontalFlip,
VerticalFlip,
Brightness,
Blur,
Noise,
)
class Defect(IntEnum):
inclusion = 0
scratch = 1
crack = 2
# 1. YAML → Config (DIRECT / REVERSE is detected)
config = Config.from_yaml("/path/to/dataset/data.yaml")
print(config) # DatasetConfig(root=..., nc=..., layout=direct, splits=[...])
# 2. Dataset owns train / val / test
dataset = Dataset(config=config, category=Defect, is_cache=True, is_strict=False)
print(dataset.health())
# 3. Iterate a split
style = Style(
palette=np.random.randint(0, 255, (80, 3)).tolist(),
stroke=Stroke(thickness=2, alpha=0.8),
fill=Fill(alpha=0.2),
show=True,
)
for sample in dataset.train:
frame = dataset.visualize(sample=sample, style=style)
cv2.imshow("sample", frame)
if cv2.waitKey(1) & 0xFF == ord("q"):
break
# 4. Online augmentation (training loop)
pipeline = Compose([
HorizontalFlip(p=0.5),
VerticalFlip(p=0.5),
Brightness(min_factor=0.6, max_factor=1.4, p=0.8),
Blur(max_ksize=5, p=0.4),
Noise(max_ratio=0.03, p=0.4),
])
drawer = Drawer(style=style)
sample = dataset.train[0]
image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
labels = dataset.train.get_instances(0)
aug_image, aug_labels = pipeline(image, labels)
vis = drawer.draw_instances(
frame=aug_image.copy(),
instances=aug_labels,
is_box=dataset.task == TaskType.DETECT,
is_obb=dataset.task == TaskType.OBB,
is_polygon=dataset.task == TaskType.SEGMENT,
)
cv2.imshow("augmented", vis)
cv2.waitKey(0)
Offline export (new directory only — never overwrite the source dataset):
from pathlib import Path
from acmenra_ds.label import LabelWriter
out_img = Path("/data/aug/train/images")
out_lbl = Path("/data/aug/train/labels")
out_img.mkdir(parents=True, exist_ok=True)
out_lbl.mkdir(parents=True, exist_ok=True)
writer = dataset.train.label_writer
for idx, sample in enumerate(dataset.train):
image = dataset.train.image_parser.parse(sample.image_path, is_strict=False)
labels = dataset.train.get_instances(idx)
if image is None:
continue
aug_image, aug_labels = pipeline(image, labels)
cv2.imwrite(str(out_img / f"{sample.stem}_aug.jpg"), aug_image)
writer.write(out_lbl / f"{sample.stem}_aug.txt", aug_labels)
📋 Requirements
Installed with pip install acmenra-ds:
acmenra-cv>=0.3.2.0
PyYAML>=6.0
numpy>=1.21.0
opencv-python>=4.5.0
Development extras (pip install "acmenra-ds[dev]"):
pytest>=7.0.0
ddt>=1.6.0
black>=23.0.0
mypy>=1.0.0
Python >=3.9. No PyTorch in the core package — Split is a plain iterable; wrap it in torch.utils.data.Dataset yourself if you need a DataLoader.
🧪 Testing
# All tests
pytest tests/
# By module
pytest tests/data/
pytest tests/augmentation/
Coverage includes:
- Type / range validation (wrong type, invalid values, valid values)
- DIRECT and REVERSE layouts, custom leaves
to_dict/from_dictround-trips- Mutation logging after construction
- Compose construction and setter contract
- Geometric / photometric property lifecycle
🔐 License
This project is licensed under the GNU Affero General Public License v3 or later (AGPL-3.0-or-later). See the LICENSE file for details.
🌐 Links
- PyPI: https://pypi.org/project/acmenra-ds/
- Core library: https://pypi.org/project/acmenra-cv/
- YOLO plugin: https://pypi.org/project/acmenra-yolo/
- Documentation: https://docs.acmenra.studio/
- Source: https://github.com/Acmenra/acmenra-ds
- Issues: https://github.com/Acmenra/acmenra-ds/issues
acmenra.studio — Building reliable vision systems for the edge. Every sample, every label, every frame.
Metadata
Release files for acmenra-ds 0.2.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| acmenra_ds-0.2.0.2.tar.gz | 87.5 kB | Without provenance | Details for acmenra_ds-0.2.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| acmenra_ds-0.2.0.2-py3-none-any.whl | Python 3 | none | any | Details for acmenra_ds-0.2.0.2-py3-none-any.whl |
Total release size: 179.7 kB
Release files / acmenra_ds-0.2.0.2.tar.gz
| Download URL | acmenra_ds-0.2.0.2.tar.gz |
|---|---|
| Size | 87.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bd9adf6df019324228798965e7f5f48f7511cb196c4110f136d632033da7e8fa
|
|
BLAKE2b-256 checksum How to use checksums |
7d1f2df4e28a94c2b01016684368cd1b219b98eca428e29bc722904fddf6a163
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.6
|
Release files / acmenra_ds-0.2.0.2-py3-none-any.whl
| Download URL | acmenra_ds-0.2.0.2-py3-none-any.whl |
|---|---|
| Size | 92.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a151bf339f87c0caa372fe04441726d24b570fd00c2eb6b8b9daa5125e913388
|
|
BLAKE2b-256 checksum How to use checksums |
68056133d092b68a1a18d8efe0c9c28b5b47e25259bfae84295934c8a2c40b34
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.6
|
Security for acmenra-ds 0.2.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.