manta-trading

Market-data acquisition, storage, and serving for US equities and Kalshi event contracts. CLI-first, TimescaleDB storage. PyPI distribution manta-trading-data, import package manta_trading, CLI entry point mt.

Overview

The system runs two independent acquisition pipelines into one TimescaleDB instance, and serves the equities side over a read-only HTTP API:

• Equities — daily and minute OHLCV from EODHD across an ~33k-symbol US registry (Finnhub enriches IPO dates), with corporate actions, adjusted-on-read pricing, per-symbol gap tracking and repair, named symbol lists, and point-in-time index universes (survivorship-bias-free S&P 500 membership). See Typical workflows.
• Kalshi event contracts — the full market catalog (series, events, markets, settlements), 1-minute candlesticks for a configurable slice of the market universe, the exchange-wide public trade tape, and a historical backfill that walks the archive back to a configured floor. See Kalshi event-contract data.
• Storage — TimescaleDB hypertables plus continuous aggregates for coarser equity grains (5m, 15m, 1h, 4h, …). Schema is owned by a migration chain in three tracks (minute, daily, kalshi); the chain is the single source of schema truth. See Setting up a new database.
• Serving — mt serve runs a read-only FastAPI server for equity bars, instrument metadata, gaps, and data health. See Data Serving API.
• Operations — production runs as bounded passes under systemd timers (not a long-lived daemon), with an hourly mt data health check, an operator wrapper mt-run, and offsite backups to S3-compatible storage. See Production deployment.

Top-level CLI map:

Installation

Requires Python 3.12+ and uv.

uv tool install manta-trading-data

mt --help should work immediately — no clone, no virtualenv activation.

The package is published on PyPI as manta-trading-data, but the Python import package is still manta_trading (import manta_trading) and the CLI command is still mt — only the install/upgrade name changed.

Updating

mt update          # check PyPI and install a newer release (prompts first)
mt update --yes    # non-interactive: install without prompting
mt update --json   # pure query: report versions, change nothing

mt update upgrades uv tool installs itself; on pipx or pip installs it prints the right command for your environment instead of running it. The equivalent manual command is always:

uv tool install --upgrade --refresh-package manta-trading-data manta-trading-data@latest

(--refresh-package matters right after a release: uv resolves against cached index metadata, so without it the upgrade can succeed while installing nothing. mt update runs this exact command and verifies the version moved.)

In a development (editable/source) checkout mt update refuses and points you at git pull && uv sync — it makes no network call there.

Development setup

To work on the code itself, use a source checkout instead:

git clone https://github.com/manta-digital/trading-data
cd trading-data
uv sync
source .venv/bin/activate   # macOS / Linux
# .venv\Scripts\activate    # Windows

mt --version reports dev in a source checkout (no installed distribution metadata to read); a uv tool install reports the real published version.

Environment

Copy .env_sample to .env and fill in the values. All variables use the MT_ prefix. .env_sample carries fuller commentary on each; this table is the summary.

Core

Acquisition tuning

Kalshi

See Kalshi event-contract data for what these control. All are optional — with none set, the client runs unauthenticated at the public rate tier and the default collection rule applies.

Renamed variables: the collection-rule variables were previously named MT_KALSHI_CANDLE_*. Every mt command now fails at startup while any old name is still set (env or .env), naming the replacement — there is no silent aliasing.

Tick / Databento

See Futures tick data. Both are optional until tick data is acquired; the preflight needs only the key.

Serving API

Backup / offsite

Backblaze B2 via its S3-compatible API (so rclone/aws tooling works unchanged). Use a bucket-scoped application key, never the account master key. Used by the backup scripts under scripts/, not by mt itself.

MT_BACKUP_S3_ENDPOINT, MT_BACKUP_S3_KEY_ID, MT_BACKUP_S3_APPLICATION_KEY, MT_BACKUP_S3_BUCKET.

MT_BACKUP_RESTIC_PASSWORD — the restic repository password for the nightly system backup (/etc, /root, crontabs, /home/manta to the same bucket under system/; slice 920). Lives in the dev checkout's .env beside the S3 keys, not in /etc/manta-trading.env. Losing it loses every system backup: keep a copy in the password manager (runbook 210 lists it as a bootstrap input).

Test / CI

Setting up a new database

The migration chain is the single source of schema truth. Bringing a fresh, empty Postgres database to the current schema is one command:

# 1. Create the database (TimescaleDB extension must be available on the instance).
PGPASSWORD=… createdb -h <host> -U postgres trading

# 2. Provision the DML/DDL role split (idempotent; run as superuser).
psql -h <host> -U postgres -d trading -f scripts/provision_roles.sql

# 3. Point at it and initialize (init needs the maintenance credential).
export MT_TIMESCALE_DB_URL=postgresql://trading_app:…@<host>:5432/trading
export MT_TIMESCALE_MAINTENANCE_URL=postgresql://trading_migrate:…@<host>:5432/trading
mt data init

mt data init is idempotent — re-running it on a healthy database applies zero migrations. Use --validate-only to inspect without changing anything (works with the application credential alone).

Migrations are organized in three tracks — minute (equity minute + shared infrastructure, the default), daily, and kalshi:

mt data migrate status --track kalshi   # check one track
mt data migrate apply  --track kalshi   # apply pending migrations on it

Verify after init:

mt data migrate status   # all rows should report "applied"
mt data caggs status     # every cagg present, each with a refresh policy

Typical workflows

First-time universe build

# Rebuild the instrument registry from EODHD (~33k symbols after OTC filter).
# Finnhub enrichment populates first_listing_date and promotes venue from
# transient 'US' to authoritative exchange. Takes ~9 hours at 60 req/min.
mt data instruments rebuild

# Skip Finnhub if you want registry populated quickly without IPO dates.
mt data instruments rebuild --skip-finnhub

# Populate delisted_date for delisted symbols.
mt data instruments populate-delisted-dates

Ongoing data acquisition

Production runs bounded passes on systemd timers (see Production deployment); the same command serves ad-hoc and catch-up use interactively:

# Run daemon indefinitely: daily + minute cycles + once-per-day CA update.
# Defaults to full active universe. Ctrl-C or SIGTERM exits cleanly.
mt data daemon run

# Bounded pass: minute data only, exit when the universe is caught up.
# (This is exactly what the mt-minute-pass systemd unit runs.)
mt data daemon run --minute --stop-when-done

# Limit to a named list; stop when done.
mt data daemon run --list priority1 --stop-when-done

# Limit to specific symbols; stop when done (--stop-when-done implied).
mt data daemon run --symbols AAPL,MSFT,SPY

# Cap credit spend.
mt data daemon run --max-credits 5000

Targeted gap fill

# Fetch all UNKNOWN daily gaps for the full universe.
mt data pull 1d --universe

# Fetch minute gaps for a specific symbol.
mt data pull 1m --symbol AAPL

# Fetch minute gaps for a named list, verbose progress.
mt data pull 1m --list priority1 -v

# Preview what would be fetched without making changes.
mt data pull 1m --universe --dry-run

# Reset terminal gaps (PROVIDER_HOLE / RETRY_EXHAUSTED) then refetch.
mt data pull 1m --symbol AAPL --reset

# Include delisted symbols (requires --universe).
mt data pull 1d --universe --include-delisted

Reading data

# Read adjusted daily bars for AAPL (default: adjusted=True).
mt data get AAPL 1d

# Read raw minute bars for a date range.
mt data get AAPL 1m --start 2024-01-01 --end 2024-03-31 --raw

# Output as JSON or CSV.
mt data get AAPL 1d --json
mt data get AAPL 1d --csv

System health

Start with mt data overview. One screen: what is running, what ran and how it ended, what is fresh, what the day's credits look like, and how much of the minute universe is covered. It reads the database plus one call to EODHD; it never shells out to systemctl or journalctl, which are how you investigate once the screen tells you where to look.

mt data overview
mt data overview --json

manta-trading overview                                                          2026-09-12 16:10 UTC

PASSES     cadence        now                            last run
minute     13:05 on Sat   idle                           13:05–15:41  complete (quota)
daily      00:35, 12:35   idle                           12:35–12:52  complete
kalshi     hourly :20     RUNNING candles (since Sat 09-12 16:20, progress 40 s ago)
                                                         15:20–15:31  complete
health     hourly :50     idle                           15:50–15:50  complete
accounting 16:30          idle                           Fri 09-11 16:30–16:31  complete
next       minute Sat 09-19 13:05 · daily Sun 09-13 00:35 · kalshi 16:20 · health 16:50
           accounting 16:30

SOURCES          newest                      health (15:50 UTC): healthy
minute bars      2026-09-11 20:00 UTC (20 h ago)
daily bars       2026-09-11 00:00 UTC (1 d ago)
kalshi candles   2026-09-12 16:05 UTC (5 min ago)
kalshi trades    none

EODHD credits    65,210 / 100,000 used today
minute universe  (accounting 09-11 16:31 UTC) 11,595,172/13,637,498 symbol-sessions covered (85.0%);
                 438,299 untraded; 1,604,027 fillable (hole 500,716, unknown 892,063, untracked
                 211,213, exhausted 35)

Outcomes read as complete, complete (quota), incomplete, provider unavailable or failed. Quota is a result, not a fault — a pass that collected the session and then spent the rest of the day's allowance on backfill is the designed steady state. A running pass shows its phase and progress; one whose process is gone shows ABANDONED with the dead pid.

# One read-only pass/fail check across the whole system: raw minute/daily
# freshness, every cagg's materialization lag, and Kalshi phase recency.
# One line per check; exit 0 pass, 1 breach, 2 could-not-run. This is what
# the hourly mt-health systemd unit runs.
mt data health
mt data health --json

# Source freshness plus the health footer — no per-symbol rows.
mt data status

# The per-symbol table (non-OK rows: GAPS, STALE, FAILED).
mt data status --detail

# Show all symbols including OK.
mt data status --all

# Drill into one symbol: detail panel + full gap listing.
mt data status --symbol AAPL

# Filter to daily or minute only.
mt data status --daily
mt data status --minute

# Machine-readable output.
mt data status --json

# Recompute the universe line the overview prints (several minutes).
# The daily mt-accounting-pass unit runs this at 16:30 UTC.
mt data accounting

mt data status summarises by default; any filter (--symbol, --health, --daily, --minute, --all, --json, --detail) prints the table, because asking for a filter is asking for rows.

Two numbers in that output changed meaning in slice 922:

• GAPS / gap_count counts open gaps only — those still being asked about. A gap the provider has answered (PROVIDER_HOLE) or given up on (RETRY_EXHAUSTED) is not counted, so the number can now reach zero. The footer's second line breaks the rest out per granularity: still asking, holes, exhausted.
• STALE means "not attempted in the last recorded universe walk", rather than "not attempted within a fixed interval". Under the weekly minute cadence the old rule marked every minute symbol STALE six days out of seven.

Corporate actions

# Bulk-fetch yesterday's splits + dividends for the full exchange (200 credits).
mt data ca update

# Full history for a single symbol.
mt data ca update --symbol AAPL

# Full history for a named list.
mt data ca update --list priority1

# Inspect stored CA data.
mt data ca show --symbol AAPL
mt data ca list --from 2024-01-01 --to 2024-12-31

Symbol lists and index universes

# List defined named lists with member counts.
mt data lists ls

# Print members of a list.
mt data lists show priority1

# Refresh the S&P 500 snapshot.
mt data lists refresh-sp500

# Show tracked index universes.
mt data universes ls

# Members of SP500 as of a date (point-in-time, survivorship-bias-free).
mt data universes as-of --name sp500 --date 2020-01-01

# Refresh index constituent tracking from source.
mt data universes refresh

Continuous aggregates

# Status of all caggs (last refresh, policy, row counts).
mt data caggs status

# Manually refresh all caggs (useful after a large backfill; needs the
# maintenance credential).
mt data caggs refresh

# Refresh a specific granularity or window.
mt data caggs refresh --granularity 1h

# Compare cagg contents against source data; repair divergence.
mt data caggs verify
mt data caggs repair

# Rebuild the coverage aggregates that back `available` ranges.
mt data caggs rebuild-coverage

Backup and restore

Nightly/weekly backup and offsite-sync scripts live under scripts/ (backup_prod.sh, offsite_sync.sh, check_archive_health.sh, …), targeting S3-compatible storage via the MT_BACKUP_S3_* variables. Restore is a CLI concern:

mt data restore assess   # read-only: what would a restore involve?
mt data restore run      # perform it (needs the maintenance credential)

The full procedure is documented in the backup-and-restore runbook (see Production deployment).

Trading session horizon

# Extend trading_sessions for all calendars (usually automatic via daemon/status).
mt data extend

# Extend a specific calendar.
mt data extend --calendar NYSE

# Alert if horizon is < 90 days out (useful in CI).
mt data extend --strict

Kalshi event-contract data

An independent acquisition pipeline for Kalshi event contracts, stored in its own kalshi schema alongside the equity data:

• Catalog — series, events, and markets (kalshi.series / kalshi.events / kalshi.markets), including settlement results. Write-on-change upserts; a persisted watermark drains the settled stream in 6-hour windows, and an awaiting-settlement set guarantees markets that closed but have not yet settled are re-checked.
• Candlesticks — 1-minute candles (kalshi.candlesticks, a hypertable) for markets selected by the collection rule (the MT_KALSHI_COLLECTION_* variables: traded-only, category allow/exclude lists, series/title exclusion regexes). Per-market watermarks.
• Trades — the exchange-wide public trade tape (kalshi.trades, a hypertable with 7-day chunks, compressed after 14 days), walked oldest-first in one-hour windows under a single watermark. Trades for unknown markets (the multi-leg tape) are counted and dropped, never an error.
• Historical backfill — an archive walk plus behind-cutoff candles and a backward trade-tape drain, filling history from before the pipeline was installed back to a configured floor.

Commands

# One bounded collection pass: every phase in order
# (catalog → candles → trades → historical). This is what the hourly
# mt-kalshi-pass systemd unit runs. Deliberately takes no phase selection.
mt data kalshi pass

# Full walk of the live catalog, the settled stream, and the awaiting set —
# the replay/repair tool. --settled-since must carry a UTC offset.
mt data kalshi sync
mt data kalshi sync --settled-since 2026-08-01T00:00:00+00:00

# Catalog counts, settlement watermark, awaiting-settlement set, and the
# candle / trades / historical blocks. Reads the database only — no API
# call, and reports sensibly before any sync has ever run.
mt data kalshi status

All three take --json. Shared exit codes: 0 OK, 1 preflight failure, 2 provider abort, 3 partial (item-level errors), 4 storage abort. A Kalshi command run before the kalshi migration track is applied exits 1 naming the missing migration.

Pass semantics

A pass is bounded: it runs each phase once, in order, and exits. An abort (provider or storage) stops the pass and reports the remaining phases as skipped; a partial (individual item errors) does not stop it. The pass outcome is the worst phase outcome.

The historical phase runs last and self-limits to thirty minutes of the client's rate budget per firing. Within a firing it: finishes the archive walk if incomplete (resumable cursor; the first firing after install runs hours, not minutes), fetches behind-cutoff candles for up to 1,000 markets, then drains the trade tape backward toward the historical floor (2026-01-01 UTC). Once the archive walk is done and the tape reaches the floor, the phase's steady-state work is just the candle top-up, and firings shorten accordingly. mt data kalshi status shows the descent progress and the effective coverage floor.

Authentication and rate budget

The client runs in one of two modes: public (unauthenticated, conservative rate tier) or authenticated (MT_KALSHI_API_KEY_ID + MT_KALSHI_PRIVATE_KEY_PATH, higher tier). Set both auth variables or neither. MT_KALSHI_REQUESTS_PER_MINUTE overrides either budget.

Futures tick data

CME futures tick data from Databento (initiative 220). estimate, adopt and reset buy nothing: estimate is a cost-and-size preflight over Databento's free metadata endpoints, and adopt and reset maintain the manifest. pass is the only command that can spend money, and only inside both spend ceilings. ingest, status and coverage load and report what is held; none of them needs the API key.

Storage. The tick database lives apart from trading. mt data init --database tick builds it: seven tables on the tick migration track. They are the manifest (tick_request, tick_archive_unit), contract definitions (tick_definition), the trades hypertable (tick_trade, both trades and tbbo tiers), the ingest ledger (tick_ingest_ledger), and Databento's dataset range and per-day condition (tick_dataset_edge, tick_day_condition). Event times are integer nanoseconds since the epoch, exactly as Databento delivers them.

In production it is trading_tick on its own PostgreSQL cluster, 17/tick (port 5433, data on /data, listening on 127.0.1.1 only), created by sudo scripts/provision_tick_cluster.sh. The script is check-then-act and safe to re-run (--check changes nothing and needs no root); it writes MT_TICK_DB_URL and MT_TICK_MAINTENANCE_URL into the checkout's .env and never touches the production cluster. tick_trade is a columnstore segmented by instrument and ordered by event time (tick_007); chunks older than 14 days are compressed by a policy, about 25 bytes per row (5.7 GiB of the proof set became 0.65 GiB). ES is configured at the tbbo tier, which carries the quote at every trade (slice 226's go/no-go).

# Records, billable size, and cost for each schema tier (trades, tbbo, mbp-1)
# plus the definition schema bought with each tier, for one instrument set
# and date range. --symbols and --stype are required. --end is EXCLUSIVE:
# this asks for 2025-01-06 through 2025-01-10.
mt data tick estimate --symbols ES.c.0 --stype continuous \
    --start 2025-01-06 --end 2025-01-11
mt data tick estimate --symbols ES.c.0 --stype continuous \
    --start 2025-01-06 --end 2025-01-11 --json

Each stored tier (trades, tbbo) shows its bundle cost (tier + definition) and a verdict against MT_TICK_SPEND_CEILING_USD: within, over, or no ceiling configured. mbp-1 is priced for comparison only and is not purchasable in this initiative.

Exit codes: 0 OK; 1 preflight (missing key, --end not after --start, --end past the dataset's available end, which the message names); 2 provider error. Click's own usage errors (a missing required option, an --stype outside the choices) also exit 2.

The archive. Purchased files live under MT_TICK_ARCHIVE_DIR (/data/tick-archive), one directory per Databento batch job, exactly as the provider lays it out. The directory is in the nightly restic backup (runbook 200), and scripts/verify_tick_archive_backup.sh proves a one-file restore.

Adopting a job already on disk. adopt takes a job directory or the provider's zip, checks every file against the job's manifest.json (size and SHA-256), copies it into the archive, and records the job in the manifest with the cost and times from Databento's own job record (a free call). Each UTC day a trading session touches gets one unit, verified against its file's header and Databento's free per-day record count. A session day with no file is recorded as a provider hole; a file on a day no session touches is named. Any mismatch refuses the whole job and writes nothing. A unit whose file fails verification is recorded as failed and the command exits 3. Adopting a job again changes nothing, except that it verifies any units a run cut short by a provider error left unverified. Adopting from the archive into an empty tick database rebuilds its manifest.

mt data tick adopt --job-id GLBX-20240930-USM7UXXJBA \
    --source /data/market-data/databento/GLBX-20240930-USM7UXXJBA
mt data tick adopt --job-id GLBX-20250123-XT4GD5UM6C \
    --source /data/market-data/databento/GLBX-20250123-XT4GD5UM6C.zip --json

Resetting failed units. reset puts exhausted units back to UNKNOWN (retried where they stand) and reopens provider holes so the day is wanted again. It asks you to type reset unless --yes or --json is given.

mt data tick reset --unit-id 17 --unit-id 18
mt data tick reset --all --yes

The acquisition pass. pass runs five phases and buys only what both spend ceilings allow. Every purchase is a batch job.

1. reconcile — matches any submit whose outcome was unknown to the job Databento lists (it was bought once, so it is never bought again), sweeps jobs whose 30-day retention has passed, and downloads, verifies and records finished jobs, earliest deadline first.
2. availability — records the dataset's edge and each day's condition; a changed condition reopens that day's holes.
3. purchase — plans monthly requests, costs each (free), applies both guards and submits. Today the plan is the definitions for the tier days the manifest already holds (about $0.004 for the two adopted jobs); the tier itself is chosen in slice 226.
4. await — polls until nothing is in flight or TICK_WAIT_BUDGET_SECONDS (1,800) is spent, then lists what is still running with its deadline.
5. definitions — projects verified definition units into tick_definition, refusing an undefined validity window, a changed definition or an overlapping window rather than guessing.

mt data tick pass --estimate-only          # plan and cost; buy nothing
mt data tick pass                          # the real run
mt data tick pass --start 2024-09-01 --end 2024-10-01   # narrow, never widen
mt data tick pass --json

--end is exclusive. Both spend ceilings must be set or the pass refuses (exit 5, naming both) and buys nothing:

A refused plan buys nothing at all, and the report says how far over each ceiling it is and from when it would fit. A submit whose outcome is unknown is never bought again blind: the next run searches the job list, and a request still unmatched after an hour is exhausted and re-submitted only after reset.

Exit codes for pass: 0 OK; 1 preflight (or --end not after --start); 2 provider abort; 3 some units failed; 4 storage abort (archive write, calendar, tick database); 5 a guard refused the purchase; 6 jobs still processing when the wait budget ended (run it again to continue).

Every manifest writer (adopt, reset, pass) runs behind one preflight and a run lock: they exit 1 naming the setting or command to fix (an unknown MT_TICK_* name, with the closest real one; the key; MT_TICK_DB_URL; MT_TICK_ARCHIVE_DIR; a pending tick migration; another run holding the lock). Exit codes for adopt and reset: 0 OK; 1 preflight or a refusal that wrote nothing; 2 provider error; 3 adopt finished but some units failed verification; 4 storage (archive write, the calendar, or the tick database).

Ingest. ingest loads every verified trades/tbbo unit into tick_trade, two units at a time, each in one transaction: the rows, their ledger (one row per contract and session, zero-record rows included) and the unit's move to ingested commit together or not at all. Three checks guard each unit: the provider's record count, the records decoded and the rows stored must agree; every record must belong to a known contract of the product; and every record must fall in a trading session. A unit that fails is marked exhausted with the check named first in its reason (counts:, resolution:, session_boundary:, overlap:, shape: or decode:); fix the cause, then mt data tick reset --unit-id N. A unit waits (it is skipped, not failed) while its day's definitions are not loaded yet, or while a higher-tier unit (tbbo over trades) holds its day. A tbbo unit loaded over an ingested trades day replaces that day's trades rows in the same transaction. Running it again loads nothing new.

mt data tick ingest                        # every unit ready to load
mt data tick ingest --unit-id 17 --json    # one unit; a unit that cannot load says why

Exit codes for ingest: 0 OK (skips included); 1 preflight (as above, but no API key is needed; this includes a tick database unreachable at start); 3 some units failed a check; 4 storage (the tick database lost or hung during the run, or the calendar unreachable; units already committed stay loaded).

Status and coverage. status shows, per product, the sessions held and how many are complete, awaiting ingest, in flight, pending, missing, failed, exhausted, holed or of unknown condition, with the dataset's edge and the contracts held (spreads hidden unless --all-instruments). Its complete means every unit covering the session is ingested; coverage proves the raw rows for a date range: it counts tick_trade rows per contract and session and compares them with the ledger.

mt data tick status
mt data tick status --all-instruments --json
mt data tick coverage --start 2024-09-03 --end 2024-09-07   # --end EXCLUSIVE

Neither command needs the API key or MT_TICK_ARCHIVE_DIR, and neither takes the run lock. Exit codes: 0 OK; 1 preflight (a setting, a pending tick migration, an unknown --product, --end not after --start, or dates past the calendar's populated sessions: run mt data extend); 3 coverage found a mismatch (it names the contract and both counts); 4 the tick or production (calendar) database is unreachable.

Data Serving API

# Start the API server (default: 0.0.0.0:8100).
mt serve

# Custom host/port, multiple workers.
mt serve --host 127.0.0.1 --port 8200 --workers 4

# Dev mode with auto-reload.
mt serve --reload

The API serves equity bars and the Kalshi prediction-market catalog and time series.

Full API reference: docs/api/reference.md. Every route, with per-field tables giving type, nullability and unit, executed curl examples, the error shapes, and the conventions (inclusive windows, UTC, decimal strings, the row ceiling). For LLM clients there is a capability-indexed companion at docs/api/agents.md, which states a retry verdict for every failure mode.

The machine-readable schema is committed at docs/api/openapi.json and regenerated with uv run python scripts/dump_openapi.py (no database required); a test fails the build on drift, and scripts/check_api_docs.py fails it when either document stops matching the schema.

Interactive docs are served at /docs while the server runs.

Server settings

Both are read once at startup; changing either requires a server restart. Note they interact — raising the bar ceiling without also raising the timeout trades a fast 422 for a slow 504.

The API is unauthenticated and CORS-open by design: it is read-only and bound to a LAN host. Exposing it beyond the LAN, or adding any route that writes, makes authentication a prerequisite.

Production deployment

Production does not run a long-lived daemon. It runs bounded passes under systemd timers from a pinned checkout at /opt/manta-trading owned by a nologin service account:

Unit files live in deploy/systemd/, alongside a resource slice (manta-acquisition.slice) and a journald namespace config.

• Install/update: deploy/install-production.sh --ref <tag> — idempotent, deploys a readable tag (e.g. prod-20260823). It enables nothing by itself; turning a timer on is an explicit operator action (sudo systemctl enable --now mt-kalshi-pass.timer).
• Operator front door: deploy/mt-run — mt-run daily|minute|kalshi fires a pass now, mt-run status shows every unit, mt-run follow [unit] tails logs, and mt-run <any mt command> runs it as the service account.
• Environment: the production env file lives at /etc/manta-trading.env (deploy/manta-trading.env.example documents it, including the Kalshi variables and PEM placement).
• Runbooks: operational procedures (production operations, backup and restore, cagg maintenance, test cluster) are indexed at project-documents/user/runbooks/__readme.md.

Integration tests

Most integration tests under test/integration/ require MT_TIMESCALE_DB_URL set to a database that already has the schema applied. They run against that DB and use a per-test fixture to reset state.

Tests that create throwaway databases (e.g. test/integration/test_cold_start.py) additionally require MT_TIMESCALE_TEST_URL — an admin connection using the trading_test_admin role (LOGIN CREATEDB only, never a superuser; see Test / CI above):

export MT_TIMESCALE_TEST_URL=postgresql://trading_test_admin:…@<host>:5432/postgres
uv run --extra dev pytest test/integration/test_cold_start.py

Load tests under test/load/ are gated behind MT_RUN_LOAD_TESTS=1.

Project structure

src/manta_trading/
  api/                   # Outbound provider HTTP clients (EODHD, Finnhub)
  api_server/            # FastAPI app (mt serve)
  cli/                   # Typer CLI (mt); commands/ per top-level group
  config/                # Settings (pydantic-settings, MT_* env vars)
  data/
    acquisition/         # Daemon, orchestrators, gap tracking
    adjustment/          # Adjusted-on-read: compute_k_factor, adjusted()
    base/                # InstrumentRegistry, TradingCalendar
    kalshi/              # Kalshi client, collection pass, sync + repositories
    maintenance/         # auto_extend, status_queries
    universe/            # EODHD symbol-list client, Finnhub IPO client
  market/
    schema/              # Migration tracks (minute / daily / kalshi) + runner
  providers/             # Provider registry, auth strategies, error taxonomy
config/
  symbol-lists.yaml      # Named symbol lists (priority1, priority2 / sp500)
deploy/
  systemd/               # Production units + timers
  install-production.sh  # Idempotent install/update by tag
  mt-run                 # Operator wrapper
scripts/                 # Backup/offsite, role provisioning, OpenAPI dump,
                         # operator cutover scripts (see scripts/README.md)
test/
  unit/                  # Unit tests (no DB required)
  integration/           # Integration tests (require MT_TIMESCALE_DB_URL)
  load/                  # Load tests (gated by MT_RUN_LOAD_TESTS=1)