Battery Data Steward

Inspect. Validate. Trace. Reuse.

Evidence-aware battery data preprocessing and quality assurance.

Battery Data Steward (BDS) is an open-source Python toolkit that helps researchers inspect raw battery data, identify quality risks, track preprocessing decisions, and prepare reviewable datasets for downstream analysis.

BDS detects supported formats, explains column and unit mappings, identifies time-axis and current-sign risks, reports unsupported fields, and converts supported time-series and EIS data to CSV or Parquet with review reports.

bds inspect raw_export.csv

Inspection is local and non-mutating. It does not modify the source file, insert missing samples, or silently change current values.

The package is vendor-neutral and independent. It is not certified by any cycler vendor or standards body. Adapter support describes behavior implemented and validated by this project; users should verify representative exports from their own cycler software before using the package in automated production workflows.

Installation

Python 3.10 or newer is required. The renamed distribution is battery-data-steward, starting with the planned 0.4.0 release. The PyPI command below becomes available after that release is published. Until then, install this checkout with python -m pip install ..

When migrating an environment that has the old distribution installed, first run python -m pip uninstall battery-data-standard. Both distributions provide the same import paths and CLI, so they should not be installed together.

pip install --upgrade battery-data-steward

Optional extras are available for additional input formats:

pip install "battery-data-steward[yaml]"
pip install "battery-data-steward[matlab]"
pip install "battery-data-steward[mpr]"

The distribution installs the bds command. Use the public short import; the historical module path remains compatible:

import bds

# Existing code remains supported.
import battery_data_standard as bds

Features

The package can:

• inspect unfamiliar battery cycler files before conversion;
• detect supported cycler and data formats;
• explain column, unit, and current-sign mappings;
• report missing fields, sampling gaps, and unmapped columns;
• provide suggested next steps when a file cannot be processed cleanly;
• convert supported time-series and EIS data to CSV or Parquet;
• generate JSON, HTML, Excel, and PDF reports;
• process directories, zip archives, and tar archives locally;
• export staging data for PyProBE, PyBaMM, cellpy, BEEP, DuckDB, and Polars workflows;
• use optional column-mapping profiles for lab-specific headers.

The package does not upload source data to an external service. It reads local files and writes local outputs.

Quick Start

Inspect the installed version:

bds --version

Inspect an unfamiliar file:

bds inspect raw_export.csv

Preview mappings for a downstream target:

bds inspect raw_export.csv --target pybamm --format json
bds inspect raw_export.csv --target pyprobe

Write review reports:

bds inspect raw_export.csv --output inspect.json --output inspect.html

Detect only the likely cycler format:

bds detect raw_export.csv

Convert a time-series file:

bds convert raw_export.csv normalized.bds.csv --cycler auto --report auto

--report auto writes normalized.bds.report.json and normalized.bds.report.pdf next to the converted CSV. Add --report-format html or --report-format xlsx when those review formats are needed.

Export directly to a downstream staging format:

bds convert raw_export.csv pybamm_drive_cycle.csv --target pybamm
bds convert raw_export.csv pyprobe_staging.parquet --target pyprobe --format parquet

Validate a converted file:

bds validate normalized.bds.csv

Convert a directory or archive and write a JSONL manifest:

bds batch raw_exports normalized_exports --recursive --manifest manifest.jsonl
bds batch raw_exports.zip normalized_exports --manifest manifest.jsonl

Audit a raw folder before committing to conversion:

bds audit raw_exports --recursive --json audit.json --html audit.html

The audit report scores each raw data file and highlights conversion failures, missing required fields, unit conversions, time-axis repairs, current-sign evidence, duplicate timestamps, non-monotonic time, suspicious flat voltage/current, and cycle/step anomalies. Optional-column coverage is reported separately so small but valid fixtures are not treated as low-quality data just because they omit fields such as temperature or energy.

Inspect runtime adapter metadata and the pinned schema:

bds formats
bds inspect-schema

bds formats reports each adapter's support tier and evidence tier so users can tell whether support is backed by public fixtures or unit tests.

Python API

Read a supported time-series export into a Polars dataframe:

import bds

df = bds.read("raw_export.csv", cycler="auto")

Show the user-facing export column names defined in the export template:

import bds

export_df = bds.to_export_frame(df)
print(export_df.columns)

Preserve Raw Current Sign And Repair Time Axis

For real experimental datasets, it is often useful to preserve the current sign exactly as recorded by the source file, allow documented time-axis repairs, and record any regular-sampling gaps:

df = bds.read(
    path,
    cycler="auto",
    current_sign="preserve",
    repair_policy="repair",
    time_sampling_policy="warn",
)

Use current_sign="preserve" when downstream analysis should keep the raw charge/discharge sign convention from the instrument. Use repair_policy="repair" when the pipeline accepts documented normalizations such as shifting elapsed test time to start at zero or sorting non-monotonic time values.

When a fixed sampling interval is detected, BDS reports missing time points by default (time_sampling_policy="warn") without inserting rows. Explicitly set time_sampling_policy="repair" to insert samples with time_sampling_interpolation="linear". Repairs skip gaps across known cycle or step boundaries, unknown segment identifiers, and step-time resets. If segment fields are absent, BDS cannot establish protocol boundaries: review the protocol before enabling interpolation. Set time_sampling_interval_s when the cadence is known.

Use an explicit cycler when the source format is known:

df = bds.read("arbin_export.csv", cycler="arbin")

Convert a file and keep the conversion report:

report = bds.convert(
    "raw_export.csv",
    "normalized.bds.csv",
    cycler="auto",
    report_path="auto",
)

This writes the converted CSV plus JSON and PDF reports. HTML and Excel reports can be added with report_formats=("html", "xlsx").

Write a downstream staging table by selecting an export target:

bds.convert("raw_export.csv", "pybamm_drive_cycle.csv", target="pybamm")
bds.convert("raw_export.csv", "cellpy_staging.csv", target="cellpy")

Read data and report information in memory:

df, report = bds.read_with_report("raw_export.csv", cycler="auto", strict=False)

Create step-level or cycle-level summaries from a normalized dataframe:

steps = bds.summarize_steps(df)
cycles = bds.summarize_cycles(df, capacity_reset="cycle")

Internal Normalized Model and Export Targets

The converter uses an internal BDS time-series table as an implementation contract for parsing and export. It is not presented as a universal scientific-data quality guarantee. Every successful default export contains three required fields:

Additional fields, such as cycle number, step number, capacity, energy, temperature, power, and internal resistance, are included when they are available in the source file.

The default BDS CSV and Parquet exports use user-facing labels with units in parentheses. Lower-level adapter data may use internal labels; prefer bds convert or to_export_frame(..., target="bds") for public handoff.

Supported Format Families

The package includes adapters for NEWARE, Arbin, Maccor, BioLogic, Repower, PEC, Novonix, BaSyTec, LANDT, and generic tabular exports. Generic readers support delimited text, Excel, MATLAB, and Parquet inputs where the file contains or can be mapped to time, voltage, and current columns.

BioLogic .mpt text exports are supported by default. Binary BioLogic .mpr files are supported through the optional mpr extra, which installs the galvani backend.

See docs/supported-formats.md for adapter scope and support-tier definitions.

EIS Data

EIS files use a separate standardized table from row-wise time-series data. Use EIS-specific commands or API functions for known impedance files:

bds detect-kind impedance.csv
bds convert-eis impedance.csv normalized.eis.csv

eis = bds.read_eis("impedance.csv")
report = bds.convert_eis("impedance.csv", "normalized.eis.csv")

EIS conversion skips entirely empty rows and exact repeated header rows, records cleanup notices, and rejects nonempty rows with invalid numeric data or non-positive frequency. Errors identify a parsed table row, not a physical file line. Explicit deg or rad phase labels determine the unit. Unitless Zphz, Zphase, and Phase headers assume degrees with a warning; use read_eis(path, phase_unit="rad") or bds convert-eis ... --phase-unit rad to override that assumption. Conversion reports retain these notices.

read() and convert() are time-series entry points. batch and batch_convert() can route mixed directories and archives that contain time-series files, EIS files, including Gamry .DTA ZCURVE files, and unsupported helper files.

Column-Mapping Profiles

Profiles map lab-specific column names to canonical column names. JSON profiles are supported by the base installation. YAML profiles require the yaml extra.

{
  "columns": {
    "test_time": "time_seconds",
    "voltage": "cell_voltage",
    "current": "cell_current"
  }
}

Use a profile from the CLI:

bds convert lab_export.csv normalized.bds.csv --cycler generic --profile profile.json

Intake Ambiguities and Accumulator Summaries

Dotted dates such as 01.02.2023 are parsed day-first. Slash-separated dates retain month-first parsing with a warning; set "date_order": "dayfirst" or "monthfirst" in a mapping profile. Profiles also accept an explicit encoding when automatic legacy encoding detection is uncertain. Ambiguous strings such as "1,000" are rejected rather than guessing whether a comma is a decimal or thousands separator. Set profile "decimal_separator": "," for decimal commas or "decimal_separator": "." for decimal points and comma thousands separators.

Headerless wide files require explicit column mapping. Multiple Arbin channel sheets require an explicit sheet or batch --excel-sheets all; independent channels are never silently concatenated. Sheet-level batch failures retain successful records and processing continues unless fail_fast=True.

Undirected capacity/energy values are preserved as source_capacity_ah or source_energy_wh. Directional fields are derived only from recognizable charge/discharge status. Reports carry accumulator_reset_scope, defaulting to unknown; a profile can declare the scope after protocol review.

summarize_cycles(df) warns when reset scope is unknown and labels maxima as recorded maxima, not transferred totals. Use capacity_reset="cycle" for cycle-reset accumulators or capacity_reset="step" to sum maxima from consecutive step segments. These modes require complete, zero-based accumulator coverage; truncated measurements and test-wide accumulators need a separate baseline-aware calculation. BDS does not invent cycle identifiers or integrate missing capacity.

Current Sign Convention

The default current convention for reading and BDS exports is charge-positive and discharge-negative. convert(..., target="pybamm") and batch PyBaMM exports instead default to discharge-positive. An explicit current_sign overrides the target default. Source power values are preserved; if current normalization changes their corresponding current values, reports warn that the source power sign convention requires review.

Use --current-sign preserve to retain the source sign convention, or --current-sign discharge-positive when a downstream workflow requires discharge-positive current. When a source file contains a recognizable charge/discharge status column, adapters use it to normalize current sign more explicitly.

BDS can record a warning-only adjacent-point current-sign sanity check in conversion metadata and audit records when current_sign is charge-positive or discharge-positive. Use --current-sign-check adjacent to enable this O(n) heuristic for files that need extra sign review.

Validation and Reports

Every conversion returns a machine-readable report with schema version, row count, columns, validation status, warnings, provenance, adapter metadata, repair operations, and time-sampling findings. With report_path="auto", BDS writes JSON and PDF reports by default; HTML and Excel reports are optional formats.

Strict validation is enabled by default. Repairable issues are reported with the default repair_policy="warn". Use repair_policy="repair" or --repair-policy repair only when the pipeline explicitly accepts the documented normalizations.

Research Scope and Roadmap

Today, BDS provides data inspection, table validation, diagnostics, conversion, and reports containing available provenance, parameters, warnings, and recorded repair operations. These outputs help researchers review data and processing; a successful conversion or validation does not establish that a dataset can support SOH, RUL, EIS fitting, or a specific scientific conclusion.

The planned Battery Data Quality & Readiness Package would add task-specific readiness criteria, explicit unresolved evidence, and review decisions. These readiness gates and a full transformation evidence workflow are future work, not current capabilities. See project direction.

BDS focuses on input quality and preprocessing review. Battery-Feature-Lab focuses on feature extraction and diagnostics; Battery Context focuses on evidence packages and scientific sufficiency. These are complementary project boundaries, not bundled integrations or guarantees of scientific validity.

Documentation

Public documentation is in the docs directory:

• Package migration and publishing
• Project direction and roadmap
• Python API reference
• Supported formats
• Export template
• Export targets
• Schema compatibility
• Diagnostics and audit reports
• Current sign convention
• Step and cycle semantics
• Warning and issue codes
• Fixture contribution guide
• Ecosystem integrations

License and Attribution

This project is distributed under the MIT License. See NOTICE.md for the project independence notice.