CPDataKit

CPDataKit is a solver-independent Python toolkit for checking, normalizing, summarizing, and plotting crystal-plasticity simulation data.

Alpha software: A passing validation report confirms that the records match the selected schema. Scientific interpretation remains with the researcher. The bundled data are generated from a fixed seed, while public reference data stay at their upstream source.

When it helps

A hand-off can be as small as a column name. One exporter writes eps, another writes strain. One stores sigma_pa in Pa, another expects stress in MPa. CPDataKit puts those choices in a schema and an explicit mapping file, then keeps the validation result with the converted data.

Use it before an analysis script, when exchanging files with a colleague, or when you need to explain later why a column was renamed. The package stays at the data boundary and keeps data contracts, provenance, validation, and conversions explicit. Its documented readers and examples cover CPDataKit HDF5, selected DAMASK DADF5 data, and the Surfalex reference workflow.

Supported contracts and formats

The built-in CPDataKit schema v1.0 has three profiles:

• curve: ordered macroscopic steps such as time, strain, stress, and load curves.
• point: material-point, integration-point, element, or sample records.
• field2d: scalar samples with two-dimensional Cartesian coordinates.

Inputs are UTF-8 CSV, JSON arrays of records, and CPDataKit HDF5 (.h5/.hdf5). CSV and JSON take units and semantics from the selected schema. HDF5 stores the units, mapping, validation summary, source filename and SHA-256, UTC conversion time, Python and CPDataKit versions, and an operation log. The current HDF5 writer also puts the canonical schema and its SHA-256 digest in the file. If you provide a schema URI, CPDataKit records it and leaves it alone. The read-only DAMASK DADF5 adapter can inspect or report a selection when the file has one clear choice. CPDataKit HDF5 remains a separate format from DAMASK DADF5 and Abaqus ODB.

Schemas declare standard names, aliases, requiredness, dtype, per-record shape, role, unit, missing-value policy, index constraints, ranges, and scientific conventions. Custom fields must be declared or use user_. Stress/strain measures, tensor order, orientation representation, units, and identifier semantics come from the explicit schema or mapping. See the data format.

Install

Install the current release from PyPI:

python -m pip install cpdatakit

For a pinned GitHub release wheel, use:

python -m pip install "https://github.com/17636365690/cpdatakit/releases/download/v0.3.0/cpdatakit-0.3.0-py3-none-any.whl"

Then follow the five-minute quickstart to validate, summarize, convert, and plot a deterministic example.

Installing from the source checkout is intended for contributors:

git clone https://github.com/17636365690/cpdatakit.git
cd cpdatakit
python -m venv .venv

Activate on Windows PowerShell:

.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"

Activate on POSIX shells:

source .venv/bin/activate
python -m pip install -e ".[dev]"

Regenerate the fixed-seed examples at any time:

python examples/generate_sample_data.py --output sample_data

Workflows covered by the repository

The examples and tests cover these paths:

• validate exported curve, point, or two-dimensional field records against an explicit contract.
• normalize exporter-specific column names and units with a reviewable JSON mapping file. An explicit mapping converts each element of a declared shaped field and leaves its dimensions intact.
• preserve validated vectors and tensors in JSON/HDF5 with declared shapes and component order.
• convert records into auditable HDF5 with units, mapping, provenance, and validation metadata.
• inspect files and produce shareable aggregate reports.
• run deterministic synthetic fixtures in notebooks, CI, and documentation examples.
• run the Surfalex HF (AA6016A) Workflow 7A example with explicit tensor mappings, source hashes, and schema provenance. Raw third-party files are downloaded only when requested.

Useful links

• PyPI package
• v0.3.0 GitHub Release
• Quickstart
• Schema authoring and mapping guide
• Examples
• Public Reference Case #1: Surfalex HF
• Roadmap and Issue tracker

Command line

Validate and write a JSON report:

cpdatakit validate sample_data/synthetic_curve.csv --schema curve --json-output validation.json

Summarize, convert, and create both image formats:

cpdatakit summary sample_data/synthetic_curve.csv --schema curve --json-output summary.json
cpdatakit convert sample_data/synthetic_curve.csv --schema curve --output curve.h5 --source-description "Synthetic README example"
cpdatakit plot curve.h5 --schema curve --kind stress-strain --output stress-strain.png
cpdatakit plot curve.h5 --schema curve --kind stress-strain --output stress-strain.svg

For an exporter with different names or units, provide an explicit mapping file:

cpdatakit convert raw.csv --schema curve --mapping mapping.json --output curve.h5

See the schema authoring and mapping guide for the JSON format and explicit-convention rules.

When you need a quick look at a file, run:

cpdatakit inspect curve.h5 --format json --output inspect.json
cpdatakit report curve.h5 --schema curve --output report.html
cpdatakit report curve.h5 --schema curve --format markdown --output report.md

inspect accepts an optional schema. It prints the detected format, fields, dtype, shape, units, missing values, HDF5 chunks, provenance, adapter, and structural risks. report needs a schema and writes HTML by default. Markdown and JSON are available through --format. The HTML file contains its own styles, so it opens and prints offline. Reports carry summary statistics and validation findings. Raw records stay out of the file. Existing output files need --force before they can be replaced.

Expected errors are concise. Put the global --debug option before the subcommand when an unexpected failure needs more detail. validate, summary, inspect, and report return 0 when validation finds zero errors. They return 1 when the data contains findings. Usage, read, schema, and output failures return 2. A passing report confirms that the declared checks ran cleanly. Review physical and scientific correctness with the domain method alongside the report. Run cpdatakit --help or cpdatakit <command> --help for command details.

Python API

from cpdatakit import (
    FieldMapping,
    build_report,
    inspect_dataset,
    load_hdf5,
    load_dataset,
    normalize_dataset,
    summarize_dataset,
    validate_dataset,
)
from cpdatakit.adapters import DamaskDADF5Adapter

raw = load_dataset("raw.csv")
normalized = normalize_dataset(
    raw,
    "curve",
    [
        FieldMapping("increment", "step", "1", "dimensionless"),
        FieldMapping("eps", "strain", "1", "dimensionless"),
        FieldMapping("sigma_pa", "stress", "Pa", "MPa", "export specification"),
    ],
)
report = validate_dataset(normalized, "curve")
summary = summarize_dataset(normalized, "curve", validation=report)
print(report.valid, summary["record_count"])
inspection = inspect_dataset("curve.h5", schema="curve")
offline_report = build_report("curve.h5", "curve")
print(inspection["record_count"], offline_report["validation"]["valid"])

dadf5 = DamaskDADF5Adapter(
    kind="homogenization", label="Taylor", field="mechanical", datasets=["F", "P"]
).load("result.hdf5")
window = load_hdf5("curve.h5", fields=["step", "stress"], start=10, stop=20)

Mapping conflicts, unknown fields, and incompatible units raise documented subclasses of CPDataKitError. Normalization returns a copy and preserves unmapped columns unless drop_unmapped=True.

Example outputs

After running the commands above, stress-strain.png and stress-strain.svg contain a titled, unit-labeled synthetic curve with a legend. Plotting functions in cpdatakit.plotting return Matplotlib (Figure, Axes) for further editing and use the non-interactive Agg backend.

Development

pytest --cov=cpdatakit
ruff check .
ruff format --check .
python -m build

Architecture, extension boundaries, and maintainer checks are in the architecture documentation. Contributions follow CONTRIBUTING.md and the Code of Conduct.

When you need a new data contract or input format, open an issue with a small synthetic sample and the field rules it should follow. That gives the next change something concrete to test.

Known limitations and roadmap

Version 0.3.0 accepts in-memory tables, explicit vectors and tensors, and scalar field2d data. Native HDF5 inspection uses bounded reads. Report analysis uses the existing validation and statistics APIs. The bundled DAMASK DADF5 reader handles a narrow, read-only selection. Future adapter work follows the documented format evidence, license review, and reproducible-fixture process. See the roadmap for the next three versions.

Citation and license

Use CITATION.cff to cite the software. CPDataKit is licensed under Apache-2.0. See LICENSE. Direct runtime dependency licenses and review notes are in NOTICE. Bundled examples use fixed-seed synthetic data, and public reference files remain available from their upstream records.