Data Ingestors 📊

Move your data into the tracebloc training environment — validated, clean, and ready for model evaluation. Your raw data never leaves your infrastructure.

How it works

Your raw data
     │
     ▼
┌──────────────────┐     ┌──────────────────────────────────┐
│  Data ingestor   │────►│  Your Kubernetes cluster         │
│                  │     │                                  │
│  Validates       │     │  Validated dataset               │
│  Preprocesses    │     │  (ready for training)            │
│  Transfers       │     │                                  │
└──────────────────┘     └──────────────┬───────────────────┘
                                        │
                               Metadata only
                                        │
                                        ▼
                         ┌──────────────────────────┐
                         │  tracebloc web app       │
                         │  (dataset management UI) │
                         └──────────────────────────┘

Only metadata (schema, statistics, structure) syncs to the web app. Raw data stays put.

Supported data types

Each template ships a sample dataset and an example ingest.yaml you can copy as a starting point.

Quickstart — declarative YAML (recommended)

Describe your dataset in ~8 lines of YAML, then helm install. The official ingestor image (this package, signed + SBOM-attested, published as ghcr.io/tracebloc/ingestor) runs it. No Dockerfile, no Python script.

1. One-time: add the chart repo on your workstation.

helm repo add tracebloc https://tracebloc.github.io/client
helm repo update

The tracebloc/client parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The tracebloc/ingestor subchart submits per-dataset ingestion runs against it.

Already installed the client via the one-liner (bash <(curl -fsSL https://tracebloc.io/i.sh))? Use --reset-then-reuse-values so the helm upgrade doesn't drop the values the installer applied:

helm upgrade <workspace> tracebloc/client -n <namespace> --reset-then-reuse-values

Append --version <version-number> to pin a specific chart version.

2. Stage your data on the cluster's shared PVC.

The chart doesn't transport data into the cluster — it points at data already accessible to the cluster's shared PVC (client-pvc by default, mounted at /data/shared/ inside the ingestor Pod). Before installing, get your raw files there. The simplest pattern for a small dataset is a throwaway kubectl cp Pod that mounts the PVC; for production you'd typically use an init container with cloud-storage sync. Full staging recipe + manifests → tracebloc/client/ingestor/README.md#stage-your-data-on-the-shared-pvc.

3. Write your ingest.yaml.

The example below is for image_classification. Other categories require different fields — e.g. tabular_classification has no images: and instead needs a typed schema: block. time_series_classification additionally requires its schema: to declare the fixed sequence_id + timestamp columns (one label per sequence). Don't copy this one blindly; grab the matching file from examples/yaml/ (one per category) and edit from there. Per-category sample data and READMEs live under templates/.

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: image_classification
table: cats_dogs_train
intent: train
csv: /data/shared/cats-dogs/labels.csv
images: /data/shared/cats-dogs/images/
label: label

The top-level shape (apiVersion, kind, category, table, intent, label) is the same for every category; the category field picks the validator set, file-extension defaults, and column conventions, and the data-source fields (csv:, images:, schema:, …) vary per category. The paths are paths inside the ingestor Pod, which is the PVC mount you populated in step 2.

4. Install once per dataset.

helm install my-cats-dogs tracebloc/ingestor \
  --namespace tracebloc \
  --set-file ingestConfig=./ingest.yaml

The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. Repeat per dataset. Customers never build an image, never write a Dockerfile, never track digest versions — the cluster's auto-upgrade flow keeps the official image current.

Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → tracebloc/client/ingestor/README.md.

Advanced: custom processors (legacy Python pattern)

Use this when the declarative schema can't express what your data needs — typically when you have non-trivial preprocessing logic, a custom validator, or a BaseIngestor subclass.

1. Install the package.

pip install tracebloc-ingestor

2. Pick an example script + adapt it.

cp examples/image_ingestor.py .

The package exports BaseIngestor, CSVIngestor, JSONIngestor, the run_ingestion runner, plus validators (FileTypeValidator, ImageResolutionValidator, TableNameValidator, etc.) and the Config / Database / APIClient helpers. See examples/ for the full set of working scripts.

3. Build + deploy as a Kubernetes Job.

The legacy Dockerfile and ingestor-job.yaml remain the canonical pattern for custom-processor flows:

docker build -t <your-registry>/<image-name>:latest .
docker push <your-registry>/<image-name>:latest
kubectl apply -f ingestor-job.yaml

The Job needs these environment variables (set in ingestor-job.yaml):

The TRACEBLOC_-prefixed names above are the canonical spelling (RFC-0076 settings-naming). Their un-prefixed predecessors (DB_USER, DB_PASSWORD, SRC_PATH, LABEL_FILE) still work as a deprecated alias — new deployments should use the prefixed names; the alias is scheduled for removal 2026-12-31.

Running custom-processor flows under Pod Security Standards (restricted)

If the namespace you're deploying into enforces the restricted Pod Security Standard (OpenShift, hardened clusters, many managed-Kubernetes namespaces), the stock Dockerfile and ingestor-job.yaml won't admit. (The declarative path's image is already PSA-restricted-compatible; this section only applies to custom Dockerfiles built from this repo.) Two changes are needed.

Check first:

kubectl get ns <namespace> -o jsonpath='{.metadata.labels}' | jq

Look for pod-security.kubernetes.io/enforce: restricted. If absent, the stock files admit fine and you can skip this section.

1. Dockerfile — drop root. Append before ENTRYPOINT:

# OpenShift-compatible: grant group write via GID 0
RUN chgrp -R 0 /app && chmod -R g=u /app
USER 1001

2. ingestor-job.yaml — add a hardened securityContext. Both pod-level and container-level:

spec:
  template:
    spec:
      securityContext:                    # pod-level
        runAsNonRoot: true
        runAsUser: 1001
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: api
        # ... existing container spec ...
        securityContext:                  # container-level
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]

Driving the ingestor classes directly

For data that doesn't fit a template, drive the ingestor classes yourself. The validator set and file handling are selected by category (via the modality registry); per-dataset tuning goes through csv_options / file_options:

from tracebloc_ingestor import Config, Database, APIClient, CSVIngestor, run_ingestion
from tracebloc_ingestor.utils.constants import TaskCategory, Intent, DataFormat

config = Config()
ingestor = CSVIngestor(
    database=Database(config),
    api_client=APIClient(config),
    table_name=config.TABLE_NAME,
    category=TaskCategory.IMAGE_CLASSIFICATION,
    data_format=DataFormat.IMAGE,
    label_column="label",
    intent=Intent.TRAIN,
)
run_ingestion(ingestor, config.LABEL_FILE, batch_size=config.BATCH_SIZE)

The per-category scripts in templates/ are the canonical starting point — copy the closest one and adapt its *_options.

Prerequisites

• Python 3.11+
• A tracebloc account
• A running tracebloc client on your infrastructure

Links

Platform · Docs · Data preparation guide · Discord

Maintainers: see RELEASING.md for the release procedure.

License

Apache 2.0 — see LICENSE.

Questions? support@tracebloc.io or open an issue.

Pre-commit

Optional but recommended: pip install pre-commit && pre-commit install sets up the git hooks from .pre-commit-config.yaml. The hooks run automatically on each commit, only on the files you touch. They are a fast local guard — CI remains the guarantee.