spark-connect-cli (scq)

An agent-friendly Spark Connect CLI — read-only querying plus async control for long-running jobs.

Built for LLM agents and humans who live in a shell. Unlike spark-sql / spark-submit, the client is a thin pure-Python gRPC client: no JVM, and no Kerberos on the client side — the Spark Connect server authenticates with its own keytab, so you just point at sc://host:15002 and go.

Why

• JSON-first, read-only by default. Safe for an agent to call for exploration; writes/DDL are blocked unless you opt in (--allow-ddl).
• Long jobs don't block you. A multi-minute Spark job shouldn't trap an agent in a 30-minute tool call. scq submits the job, hands back a durable job id, and returns immediately. Poll it whenever you like; the handle survives a client/container restart because it lives in an on-disk registry.
• Stable exit codes so a caller can branch without scraping text.

Install

pip install spark-connect-cli         # pulls pyspark-client (slim, no JVM jars)
# or, from source:
pip install -e .

The dependency is pyspark-client — Spark's connect-only client (~480 MB smaller than full pyspark[connect], same pyspark import). If your environment already has full pyspark installed, install this package with --no-deps to avoid the two distributions clobbering the shared module.

Quick start

export SPARK_REMOTE=sc://localhost:15002   # your Spark Connect endpoint

scq databases
scq tables mydb --like '%orders%'
scq describe mydb.orders
scq query "SELECT id, name FROM mydb.orders LIMIT 10"

Output is JSONEachRow (one JSON object per line) by default; pick another with --format json|csv|tsv|table.

Read-only guard

scq query allows only SELECT/SHOW/DESCRIBE/EXPLAIN/WITH. Anything else exits with code 3 unless you pass --allow-ddl.

Async jobs (Layer A)

Long work runs detached and is tracked by a file-based registry under $SCQ_JOBS_DIR (default ~/.spark-connect-cli/jobs).

# submit — returns a job id immediately, does NOT block
scq sync ods.orders --to clickhouse
# {"job_id": "j-20260625-...", "state": "running", "message": "... poll with ..."}

scq jobs list                       # all jobs + state
scq jobs status j-20260625-...      # full status (rows, timings, pid, exit code)
scq jobs logs   j-20260625-... --tail 40
scq jobs cancel j-20260625-...      # kills the whole process group
scq jobs prune --keep-days 7        # delete old finished jobs (running ones kept)

Design: each job is a directory with meta.json (state machine: submitted → running → succeeded|failed|cancelled) and out.log. The worker runs in its own process group, so cancel kills the entire tree (no orphans). A running job whose process has vanished is reconciled to failed on the next status read, so status never lies.

Hive → ClickHouse sync

scq sync is one job kind built on the async subsystem. It uses Spark direct write: a Spark Connect job reads the Hive table and writes to ClickHouse over JDBC. The write runs on the executors, so rows never pass through this process or the agent.

Modes control write parallelism — single (one connection, small tables), parallel (N partitions, large tables), auto (picks by row count).

Requires:

• clickhouse-jdbc on the Spark Connect server classpath (/opt/spark/jars/),
• cluster→ClickHouse network egress,
• a JDBC URL with credentials via --ch-jdbc / $SCQ_CH_JDBC,
• the target ClickHouse table created beforehand with a suitable engine. Auto-create (--order-by/--engine) is best-effort only: Spark's generic JDBC dialect has been observed failing to create ClickHouse tables, after which the insert dies with Code: 60 (UNKNOWN_TABLE). Create the table first, e.g. with the chsql skill.

Introspection

scq meta db.table            # one JSON: schema, created time, location,
                             # partitions, file count/size, mtime range
scq meta db.table --count    # also run an exact count(*)

scq exec stages?status=active            # read-only Spark REST passthrough
scq exec executors
scq exec stages/<id>/<attempt>/taskSummary?quantiles=0.5,0.95,1.0   # skew: max/median

scq exec auto-discovers the running Spark app via the YARN ResourceManager and proxies its monitoring REST API (GET-only). Set the RM base with $SCQ_YARN_RM.

Reading scq exec executors — the maxMemory field is Spark's storage/cache pool ((heap − 300 MB reserved) × 0.6), not the executor's total memory: a 512 MB executor reports ~93 MB, a 1536 MB driver ~741 MB. The real heap is spark.executor.memory (+ off-heap overhead). The driver row has 0 cores and runs no tasks. With dynamic allocation, idle executors are released — so the list may show only the driver when nothing is running.

Configuration

Use with an LLM agent

SKILL.md ships a ready-made skill (discover-before-query workflow, async-job etiquette, type-mapping table). Drop it into your agent's skills directory and the agent drives scq through a shell/Bash tool.

Roadmap

• scq cluster — optional read-only passthrough to the YARN ResourceManager REST (apps / queues / nodes), rounding out the introspection plane.
• Vendored/offline install path (bundle wheels) for air-gapped deployments.

License

MIT