datasetforge-kit 0.1.0

Turn messy public data into clean, ML-ready datasets.

DatasetForge

DatasetForge turns messy JSON sources into clean, reproducible datasets for analysis and ML workflows.

What it does

• Load local JSON or REST API responses
• Flatten nested JSON into tabular records
• Rename, select, and drop fields
• Fill missing values and deduplicate records
• Infer a simple schema summary
• Export CSV, JSON, and optional Parquet
• Write reproducible config, metadata, and dataset cards
• Run from a single CLI command

Install

pip install datasetforge-kit

For local development:

pip install -e .

Optional Parquet support:

pip install datasetforge-kit[parquet]

Quick start

Create a config file:

datasetforge init-config --path datasetforge.config.json

Run the pipeline:

datasetforge run --config datasetforge.config.json

Inspect the source:

datasetforge inspect --config datasetforge.config.json

Generate a schema summary:

datasetforge schema --config datasetforge.config.json

Config format

{
  "name": "sample-dataset",
  "source": {
    "kind": "json_file",
    "location": "data/input.json",
    "records_key": null,
    "line_delimited": false
  },
  "transforms": {
    "flatten_separator": ".",
    "rename": {"user.name": "name"},
    "select": ["id", "name", "score"],
    "drop": [],
    "missing_tokens": ["", "null", "N/A"],
    "fill_values": {"score": 0},
    "dedupe_keys": ["id"],
    "fuzzy_dedupe_keys": [],
    "fuzzy_threshold": 0.0
  },
  "validation": {
    "required_fields": ["id", "name"],
    "field_types": {"id": "integer"},
    "allowed_values": {}
  },
  "export": {
    "format": "csv",
    "path": "dist/sample-dataset.csv",
    "indent": 2,
    "include_metadata": false
  },
  "versioning": {
    "enabled": true,
    "directory": "dist/versions"
  }
}

Python API

from datasetforge import load_config, run_pipeline

config = load_config("datasetforge.config.json")
result = run_pipeline(config)
print(result.export_path)

Notes

• Parquet export requires pyarrow.
• REST sources use the standard library HTTP client and expect JSON responses.
• Versioned runs write a deterministic content hash to the output path and manifest.

Roadmap

The library is structured for scheduled jobs, deeper schema rules, version history, fuzzy dedupe, and richer notebook helpers without changing the CLI contract.