DKit — Data Toolkit

A Python data-processing library and CLI for data engineering and exploration. Data is handled in a canonical format (iterators or lists of dicts) and the library provides facilities to translate between that format and a wide variety of storage formats, manage schemas, manipulate data, and generate reports.

Version: 26.7.1 | Requires: Python ≥ 3.11 | License: MIT

Installation

pip install libdkit          # from PyPI
pip install .                # from source
pip install -e .             # editable install for development

Library overview

Supported data formats

Schema management

Schemas are stored in a compact shorthand encoding (e.g. String(str_len=20)) and validated using Pydantic. Export is supported for:

• Apache Arrow
• Apache Avro
• SQL DDL (via SQLAlchemy, any supported dialect)
• Apache Spark
• Pandas DataFrame dtypes
• Python dataclasses
• GraphViz entity-relation diagrams

ETL model

The ModelManager maintains connections, endpoints, entities, queries, transforms, relations, and encrypted secrets in a YAML model file.

from dkit.etl.model import ModelManager, Entity

m = ModelManager.from_file("model.yml")

# infer a schema from data
with m.source("data/input.jsonl") as src:
    entity = Entity.from_iterable(src)
m.entities["person"] = entity

# coerce CSV data to schema and write to parquet
with m.source("data/input.csv") as src:
    with m.sink("output/result.parquet") as snk:
        snk.process(entity(src))

Document generation

from dkit.doc2.document import Document
from dkit.doc2.html_renderer import HtmlRenderer

doc = Document()
doc.add_template("# Hello\n\nThis is a **report**.")

renderer = HtmlRenderer(css="email.css")
renderer.render(doc, "report.html")

# inline CSS for email clients
html = renderer.render_email_string(doc)

CLI — dk

The dk command-line tool exposes the library for interactive and scripted data processing. Run dk MODULE -h for full options on any module.

dk MODULE [subcommand] [options]

Maintenance modules

Action modules

Data exploration (dk xplore)

dk xplore head        data.jsonl           # first n rows
dk xplore summary     data.jsonl           # field statistics
dk xplore histogram   data.jsonl -f price  # histogram for a field
dk xplore count       data.jsonl -f status # count by field value
dk xplore distinct    data.jsonl -f country
dk xplore duplicates  data.jsonl -k id
dk xplore view        data.jsonl           # interactive grid
dk xplore table       data.jsonl           # tabular output
dk xplore fields      data.jsonl           # list field names
dk xplore struc       data.jsonl           # structure map
dk xplore plot        data.jsonl           # plot grammar
dk xplore sample      database.db          # sample from DB tables

Schema operations (dk schemas)

dk schemas infer      data.csv             # infer schema from data
dk schemas ls                              # list stored schemas
dk schemas print      person               # print a schema
dk schemas export     person --format arrow
dk schemas sql-reflect mydb --connection sqlite
dk schemas grep       "person.*"

Running pipelines (dk run)

dk run etl      input.csv  -o output.parquet
dk run agg      input.jsonl -o output.jsonl
dk run join     left.jsonl right.jsonl -o result.jsonl
dk run pivot    input.jsonl -o pivot.jsonl
dk run melt     pivot.jsonl -o output.jsonl
dk run query    --connection mydb "select * from person"
dk run exec     --connection mydb "create index ..."
dk run report   report.yml
dk run template template.j2 data.jsonl

Development

make test      # run test suite with coverage
make build     # build sdist and wheel
make install   # install into current environment
make clean     # remove build artefacts

Tests live in test/. Run from the repo root:

pytest               # all tests
pytest --cov         # with coverage report

Integration tests that require external services (HDFS, SFTP, ZMQ) are prefixed with _test_ and excluded from the default run.

Project links

• Source: https://github.com/cobusn/dkit
• Issues: https://github.com/cobusn/dkit/issues