chalkml 1.0.3

ChalkML is a deterministic data processing framework with advanced data design patterns

ChalkML

Production-grade ML data processing framework with exceptional determinism

Overview

ChalkML is a data preprocessing framework for machine learning to assist researchers and engineers with advanced data processing which are missing from Pandas. In addition to its innovative positional notation, ChalkML comes with advanced data processing design patterns. Each pattern addresses critical gaps in production ML systems: deterministic reproducibility, feature selection, privacy preservation, synthetic data generation, and regulatory compliance.

Installation

pip install chalkml

Position Notation

ChalkML uses intuitive position notation for columns:

Notation	Meaning
01N	1st from left
02N	2nd from left
N01	Last (1st from right)
N02	2nd last

# Examples
chalkml -rm col 01N data.csv              # Remove 1st column
chalkml -mv col 02N N01 data.csv          # Move 2nd to last
chalkml -fillsmart col 03N mean data.csv  # Fill with mean

Core Operations

# Data manipulation
chalkml -rm col 01N data.csv                    # Remove
chalkml -mv col 01N 03N data.csv                # Move
chalkml -rn col 02N "Age" data.csv              # Rename

# Imputation
chalkml -fillsmart col 03N mean data.csv        # Mean
chalkml -fillsmart col 04N knn data.csv         # KNN

# Feature engineering
chalkml -derive "BMI" "col:weight/col:height**2" data.csv
chalkml -onehot col 05N data.csv
chalkml -scale col 03N data.csv --method standard

5 Design Patterns

MAP - Transform each element

chalkml -map col 05N "x*2" data.csv
chalkml -map col 06N "x**2" data.csv

REDUCE - Aggregate columns

chalkml -reduce col 01N,02N,03N sum Total data.csv

STENCIL - Sliding windows

chalkml -stencil col 03N 5 rolling_mean data.csv

SCAN - Cumulative operations

chalkml -scan col 03N cumsum data.csv

FARM - Parallel operations

chalkml -farm col range 01N:10N "x*2" data.csv

5 Advanced Patterns

QUANTUM - Schema-based compression (70-90% reduction)

chalkml -quantum compress --schema medical patients.csv
chalkml -quantum decompress --schema medical compressed.csv

RELEVANCE - Feature selection (95-98% reduction)

chalkml -relevance select --target 01N --threshold 0.1 data.csv
chalkml -relevance rank --target outcome --top 10 data.csv

REDACT - Privacy preservation (HIPAA, GDPR)

chalkml -redact hipaa --identifiers all patients.csv
chalkml -redact differential --epsilon 1.0 sensitive.csv

SCAFFOLD - Synthetic data generation

chalkml -scaffold sequence --type fibonacci --count 100
chalkml -scaffold distribution --dist normal --count 10000

BOW - Compliance standardization

chalkml -bow standard --standard GAAP financial.csv
chalkml -bow format --column date --type date --pattern "YYYY-MM-DD" data.csv

Python API

from chalkml import get_chalkml_engine
from chalkml.quantum_engine import get_quantum_engine

# Core operations
engine = get_chalkml_engine()
engine.remove_column("data.csv", "01N")
engine.fill_smart("data.csv", "03N", "mean")

# Advanced patterns
quantum = get_quantum_engine()
quantum.compress_file("data.csv", "schema", "out.csv")

License

MIT License

Authors

Hope Mogale & MY Pitsane
Mankind Research Labs ([e-mail])

Version 1.0.0 | Production-ready | 15+ validated use cases