Data Ingestors 📊

Move your data into the tracebloc training environment — validated, clean, and ready for model evaluation. Your raw data never leaves your infrastructure.

How it works

Your raw data
     │
     ▼
┌──────────────────┐     ┌──────────────────────────────────┐
│  Data ingestor   │────►│  Your Kubernetes cluster         │
│                  │     │                                  │
│  Validates       │     │  Validated dataset               │
│  Preprocesses    │     │  (ready for training)            │
│  Transfers       │     │                                  │
└──────────────────┘     └──────────────┬───────────────────┘
                                        │
                               Metadata only
                                        │
                                        ▼
                         ┌──────────────────────────┐
                         │  tracebloc web app       │
                         │  (dataset management UI) │
                         └──────────────────────────┘

Only metadata (schema, statistics, structure) syncs to the web app. Raw data stays put.

Supported data types

Type	Categories
Image	`image_classification`, `object_detection`, `keypoint_detection`, `semantic_segmentation`
Text / NLP	`text_classification`, `token_classification`, `sentence_pair_classification`, `masked_language_modeling`, `causal_language_modeling`, `seq2seq`, `embeddings`
Tabular	`tabular_classification`, `tabular_regression`
Time series	`time_series_forecasting`, `time_series_classification`, `time_to_event_prediction`

Each template ships a sample dataset and an example ingest.yaml you can copy as a starting point.

Quickstart — declarative YAML (recommended)

Describe your dataset in ~8 lines of YAML, then helm install. The official ingestor image (this package, signed + SBOM-attested, published as ghcr.io/tracebloc/ingestor) runs it. No Dockerfile, no Python script.

1. One-time: add the chart repo on your workstation.

helm repo add tracebloc https://tracebloc.github.io/client
helm repo update

The tracebloc/client parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The tracebloc/ingestor subchart submits per-dataset ingestion runs against it.

Already installed the client via the one-liner (bash <(curl -fsSL https://tracebloc.io/i.sh))? Use --reset-then-reuse-values so the helm upgrade doesn't drop the values the installer applied:
helm upgrade <workspace> tracebloc/client -n <namespace> --reset-then-reuse-values
Append --version <version-number> to pin a specific chart version.

2. Stage your data on the cluster's shared PVC.

The chart doesn't transport data into the cluster — it points at data already accessible to the cluster's shared PVC (client-pvc by default, mounted at /data/shared/ inside the ingestor Pod). Before installing, get your raw files there. The simplest pattern for a small dataset is a throwaway kubectl cp Pod that mounts the PVC; for production you'd typically use an init container with cloud-storage sync. Full staging recipe + manifests → tracebloc/client/ingestor/README.md#stage-your-data-on-the-shared-pvc.

3. Write your ingest.yaml.

The example below is for image_classification. Other categories require different fields — e.g. tabular_classification has no images: and instead needs a typed schema: block. time_series_classification additionally requires its schema: to declare the fixed sequence_id + timestamp columns (one label per sequence). Don't copy this one blindly; grab the matching file from examples/yaml/ (one per category) and edit from there. Per-category sample data and READMEs live under templates/.

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: image_classification
table: cats_dogs_train
intent: train
csv: /data/shared/cats-dogs/labels.csv
images: /data/shared/cats-dogs/images/
label: label

The top-level shape (apiVersion, kind, category, table, intent, label) is the same for every category; the category field picks the validator set, file-extension defaults, and column conventions, and the data-source fields (csv:, images:, schema:, …) vary per category. The paths are paths inside the ingestor Pod, which is the PVC mount you populated in step 2.

4. Install once per dataset.

helm install my-cats-dogs tracebloc/ingestor \
  --namespace tracebloc \
  --set-file ingestConfig=./ingest.yaml

The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. Repeat per dataset. Customers never build an image, never write a Dockerfile, never track digest versions — the cluster's auto-upgrade flow keeps the official image current.

Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → tracebloc/client/ingestor/README.md.

Advanced: custom processors (legacy Python pattern)

Use this when the declarative schema can't express what your data needs — typically when you have non-trivial preprocessing logic, a custom validator, or a BaseIngestor subclass.

1. Install the package.

pip install tracebloc-ingestor

2. Pick a template + adapt the script.

cp templates/image_classification/image_classification.py .

The package exports BaseIngestor, CSVIngestor, JSONIngestor, the run_ingestion runner, plus validators (FileTypeValidator, ImageResolutionValidator, TableNameValidator, etc.) and the Config / Database / APIClient helpers. See examples/ for working scripts.

3. Build + deploy as a Kubernetes Job.

The legacy Dockerfile and ingestor-job.yaml remain the canonical pattern for custom-processor flows:

docker build -t <your-registry>/<image-name>:latest .
docker push <your-registry>/<image-name>:latest
kubectl apply -f ingestor-job.yaml

The Job needs these environment variables (set in ingestor-job.yaml):

Variable	What it is
`CLIENT_ID`, `CLIENT_PASSWORD`	Tracebloc client credentials
`CLIENT_PVC`	PVC name shared with the client (must match `values.yaml`)
`MYSQL_HOST`	Hostname of the client's MySQL service
`DB_USER`, `DB_PASSWORD`	Required. Credentials for the dataset database. There is no built-in fallback account — the Job fails at startup without these. On installs with `serviceDbAccounts: true`, use the generated `tb_ingest` account (password in the `<release-name>-secrets` Secret, key `TB_INGEST_PASSWORD`).
`SRC_PATH`	Where your raw data is mounted in the ingestor pod
`LABEL_FILE`	Path to labels (e.g. `Xy_train.csv`)
`TABLE_NAME`	Destination table name in the client database
`TITLE`	(optional) Human-readable dataset name
`LOG_LEVEL`	(optional) `INFO`, `WARNING`, `ERROR`

Running custom-processor flows under Pod Security Standards (`restricted`)

If the namespace you're deploying into enforces the restricted Pod Security Standard (OpenShift, hardened clusters, many managed-Kubernetes namespaces), the stock Dockerfile and ingestor-job.yaml won't admit. (The declarative path's image is already PSA-restricted-compatible; this section only applies to custom Dockerfiles built from this repo.) Two changes are needed.

Check first:

kubectl get ns <namespace> -o jsonpath='{.metadata.labels}' | jq

Look for pod-security.kubernetes.io/enforce: restricted. If absent, the stock files admit fine and you can skip this section.

1. Dockerfile — drop root. Append before ENTRYPOINT:

# OpenShift-compatible: grant group write via GID 0
RUN chgrp -R 0 /app && chmod -R g=u /app
USER 1001

2. ingestor-job.yaml — add a hardened securityContext. Both pod-level and container-level:

spec:
  template:
    spec:
      securityContext:                    # pod-level
        runAsNonRoot: true
        runAsUser: 1001
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: api
        # ... existing container spec ...
        securityContext:                  # container-level
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]

Driving the ingestor classes directly

For data that doesn't fit a template, drive the ingestor classes yourself. The validator set and file handling are selected by category (via the modality registry); per-dataset tuning goes through csv_options / file_options:

from tracebloc_ingestor import Config, Database, APIClient, CSVIngestor, run_ingestion
from tracebloc_ingestor.utils.constants import TaskCategory, Intent, DataFormat

config = Config()
ingestor = CSVIngestor(
    database=Database(config),
    api_client=APIClient(config),
    table_name=config.TABLE_NAME,
    category=TaskCategory.IMAGE_CLASSIFICATION,
    data_format=DataFormat.IMAGE,
    label_column="label",
    intent=Intent.TRAIN,
)
run_ingestion(ingestor, config.LABEL_FILE, batch_size=config.BATCH_SIZE)

The per-category scripts in templates/ are the canonical starting point — copy the closest one and adapt its *_options.

Prerequisites

Python 3.11+
A tracebloc account
A running tracebloc client on your infrastructure

Links

Platform · Docs · Data preparation guide · Discord

Maintainers: see RELEASING.md for the release procedure.

License

Apache 2.0 — see LICENSE.

Questions? support@tracebloc.io or open an issue.

Pre-commit

Optional but recommended: pip install pre-commit && pre-commit install sets up the git hooks from .pre-commit-config.yaml. The hooks run automatically on each commit, only on the files you touch. They are a fast local guard — CI remains the guarantee.

License
- OSI Approved :: Apache Software License
Operating System
- OS Independent
Programming Language
- Python :: 3

Release files for tracebloc-ingestor 0.8.14

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tracebloc-ingestor 0.8.14
File	Size	Uploaded
tracebloc_ingestor-0.8.14.tar.gz	553.6 kB	Aug 27, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for tracebloc-ingestor 0.8.14
File	Interpreter	ABI	Platform	Reset
tracebloc_ingestor-0.8.14-py3-none-any.whl 656.0 kB Aug 27, 2026	Python 3	none	any	Details

Total release size: 1.2 MB

Release files / tracebloc_ingestor-0.8.14.tar.gz

Download URL	tracebloc_ingestor-0.8.14.tar.gz
Size	553.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`d81e98b43d474965330a707b11bf2eb52a58c1c6173fda1fe133647d4cfa4528`
BLAKE2b-256 checksum How to use checksums	`b17d28ede1a29a201d1b296fd7a6ea63447c9c609241b8ded1d4c11a1bd132fc`
Upload date	Aug 27, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.16`

Release files / tracebloc_ingestor-0.8.14-py3-none-any.whl

Download URL	tracebloc_ingestor-0.8.14-py3-none-any.whl
Size	656.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a6a78274ba8a250cede1f0df1c20b8fd0763ca18c08c9ab02f5c02f63b70dae0`
BLAKE2b-256 checksum How to use checksums	`5f0cea504100693d018d1cbed52ee6c05ef00332610d57600dd8a7ce015fcc0b`
Upload date	Aug 27, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.11.16`