seeknal 2.9.5

All-in-one platform for data and AI/ML engineering

Seeknal

Transform data with SQL and Python. Build ML features with point-in-time joins. Materialize to PostgreSQL and Iceberg — all from one CLI.

Seeknal is an all-in-one platform for data and AI/ML engineering. Define pipelines in YAML or Python, run them through a safe draft → dry-run → apply workflow, and materialize outputs to PostgreSQL and Apache Iceberg simultaneously. Python 3.11+ required.

Quick Start

pip install seeknal
# Optional, only for distributed Spark execution:
# pip install "seeknal[spark]"

seeknal init --name my_project
seeknal draft --name my_pipeline --type transform
seeknal dry-run
seeknal apply

Explore your data interactively or search docs from the terminal:

seeknal repl          # Interactive SQL on pipeline outputs
seeknal docs query    # Search documentation from the CLI

SELECT customer_id, COUNT(*) as order_count
FROM target.my_transform
GROUP BY customer_id;

Key Features

Dual Pipeline Authoring — Write pipelines in YAML, Python decorators, or both:

from seeknal.pipeline import source, transform

@source(name="orders", source="csv", table="data/orders.csv")
def orders():
    pass

@transform(name="order_metrics", inputs=["source.orders"])
def order_metrics(ctx):
    df = ctx.ref("source.orders")
    return ctx.duckdb.sql(
        "SELECT customer_id, SUM(amount) as total FROM df GROUP BY customer_id"
    ).df()

Multi-Target Materialization — Write to PostgreSQL and Iceberg from a single node:

materializations:
  - type: postgresql
    connection: local_pg
    table: analytics.my_table
    mode: upsert_by_key
    unique_keys: [id]
  - type: iceberg
    table: atlas.namespace.my_table

Environment Management — Isolated namespaces with per-environment profiles:

seeknal env plan dev --profile profiles-dev.yml
seeknal env apply dev
seeknal run --env dev

Feature Store — Define ML features in YAML or Python with entity keys, point-in-time joins, and automatic versioning. Supports offline (batch) and online (real-time) serving.

# seeknal/feature_groups/customer_features.yml
kind: feature_group
name: customer_features
entity:
  name: customer
  join_keys: ["customer_id"]
materialization:
  event_time_col: latest_order_date
  offline: { enabled: true, format: parquet }
  online: { enabled: false, ttl: 7d }
features:
  total_orders: { dtype: integer }
  total_spent: { dtype: float }
  avg_order_value: { dtype: float }
inputs:
  - ref: transform.customer_orders

# Or use Python decorators
@feature_group(name="customer_rfm", entity="customer")
def customer_rfm(ctx):
    df = ctx.ref("transform.clean_transactions")
    return ctx.duckdb.sql("""
        SELECT CustomerID, COUNT(DISTINCT InvoiceNo) as frequency,
               SUM(TotalAmount) as monetary_value
        FROM df GROUP BY CustomerID
    """).df()

seeknal entity list                           # Cross-feature-group consolidation
seeknal entity show customer                  # Inspect entity schema and feature groups

Interactive SQL REPL — Auto-registers parquets, PostgreSQL, and Iceberg sources at startup. Query pipeline outputs, explore data, iterate on SQL — all without leaving the terminal.

AI-Powered Thinking Partner — seeknal ask chat is your collaborative partner for data work. The agent uses thin tools for fast data access and fat skills for multi-step workflows like report generation, pipeline building, database analysis, and data profiling — all loaded on demand to keep responses fast:

seeknal ask chat                        # Start a brainstorm / build session (interactive TUI)
seeknal ask "What are the top 5 customers by revenue?"  # Quick one-shot question
seeknal ask report "customer analysis"  # Generate interactive HTML dashboard
seeknal ask test --project . --sql-only # Validate project prompt-to-SQL tests
seeknal ask chat --web                  # Enable web search for benchmarks

seeknal ask chat launches an interactive terminal UI (Bun + React + Ink) with streaming tokens, tool visualization, and arrow-key ask_user picker for approval gates. The TUI is bundled inside the wheel; end users do not need Bun or Node. One-shot (seeknal ask "...") and report (seeknal ask report) commands use Python-only rendering with no TUI.

Ask it to answer questions against existing read-only databases with seeknal source connect, reuse project SQL examples from seeknal/sql_pairs/, and validate important questions with executable seeknal/tests/ QA oracles. Ask it to build a pipeline from scratch, and it will draft a plan, walk you through the design, and wait for your go-ahead before generating code. Publish reports to a self-hosted Seeknal Report Server and share them with your team via a URL.

For editable installs (pip install -e .), set SEEKNAL_TUI_BINARY_PATH to your local TUI build. See src/seeknal/ask/tui/README.md for full TypeScript contributing guide and development workflow.

seeknal report-server start             # Host published reports
seeknal gateway start                   # Expose ask as an API (WebSocket/SSE/REST)
seeknal gateway worker --gateway-url http://gateway:8000 --api-token "$SEEKNAL_API_TOKEN"  # Token-routed Temporal worker

Supports Google Gemini (default), OpenAI-compatible providers, Anthropic-compatible providers, and Ollama (local). Use --provider ollama for fully local, private analysis.

Documentation

Getting Started	Installation, configuration, first pipeline
CLI Reference	All commands and flags
YAML Schema	Pipeline YAML reference
CLI Docs Search	Search documentation from the terminal (seeknal docs)
Tutorials	YAML Pipelines · Python Pipelines · Mixed · Seeknal Ask Agent · Report Exposures
Guides	Python Pipelines · Testing & Audits · Iceberg Materialization · Training to Serving
Servers	Gateway Server · Report Server
Concepts	Point-in-Time Joins · Virtual Environments · Exposures · Glossary

Changelog

v2.9.1 (April 2026)

HTTP-only Ask worker mode — Adds a gateway-routed worker topology where workers only need outbound HTTP(S) to Seeknal Gateway or a compatible kc-service gateway.

• HTTP worker transport: seeknal gateway worker --transport http long-polls gateway work-stream endpoints, runs Ask locally near the data, and posts streaming events plus completion back over HTTP.
• Gateway broker mode: seeknal gateway start --temporal --worker-transport http and seeknal gateway backend --worker-transport http keep Temporal routing inside the gateway while external workers avoid Temporal credentials/network access.
• Token-routed runtime config: token records can advertise worker_transport: http; workers still bootstrap from SEEKNAL_GATEWAY_URL + SEEKNAL_API_TOKEN.
• Worker reliability fixes: project .env is loaded in worker mode, gateway polling retries transient connection failures, and Temporal activities heartbeat while waiting for HTTP workers.
• pydantic-deep compatibility: skips unsupported stuck_loop_detection passthrough on current runtime versions while preserving config compatibility in tests/mocks.

v2.9.0 (April 2026)

Read-only Ask source harness + project SQL QA — Adds a TUI-first workflow for users who already have analytical tables in a database and want Seeknal Ask to answer business questions without building a pipeline.

• Connected-source registry: seeknal source connect/status/inspect/sync/test writes seeknal_agent.yml, generates .seeknal/context/sources/ metadata, and verifies read-only database attachments.
• SQL pairs for context: seeknal/sql_pairs/*.yml stores prompt-to-SQL examples the Ask agent can discover with list_sql_pairs / read_sql_pair.
• Ask SQL tests: seeknal ask test runs project-local prompt-to-SQL QA cases from seeknal/tests/, including SQL-only oracle checks and agent-answer checks.
• TUI QA cockpit: Ask chat can list, read, run, and inspect Ask tests via thin tools over the same test engine.
• Structured grading: Ask tests support assert.compare: dataframe for markdown/JSON table comparison against expected SQL rows.
• Init guidance: seeknal init now scaffolds AGENTS.md, CLAUDE.md, seeknal/sql_pairs/, and seeknal/tests/ for project-local agent conventions.

v2.8.0 (April 2026)

OpenAI/Anthropic providers + SQL safety + context files — Adds two new LLM provider families, execution guards on execute_sql, a pre-execution preview_query tool, persistent context files, and durable preferences.

• OpenAI + Anthropic support: gpt-4o, claude-*, Azure OpenAI, Together, Groq, vLLM, LM Studio, and any OpenAI-compatible proxy via SEEKNAL_ASK_OPENAI_BASE_URL / SEEKNAL_ASK_ANTHROPIC_BASE_URL
• execute_sql guards: rows capped at 500, columns at 50, per-cell length at 200 chars, 50 KB markdown budget — every truncation emits an actionable notice with accurate total row count
• preview_query tool: four pre-execution safety probes (row count, column count, JOIN fan-out, dry-run reachability) — blocks queries returning ≥100k rows; pure aggregations auto-skip
• Context files: list_context_files and write_project_file tools scan/write {project}/context/ with path-traversal guards
• Durable preferences: save_preference appends to preferences.yml; preferences are injected into the system prompt on every session

Install from Source

For development or contributing:

git clone https://github.com/mta-tech/seeknal.git
cd seeknal
uv venv --python 3.11 && source .venv/bin/activate
uv pip install -e ".[all]"

Contributing

Contributions are welcome! See CONTRIBUTING.md for setup, code style, testing, and PR guidelines.

License

Seeknal is Apache 2.0 licensed.