manta-trading

Data acquisition, storage, and serving for equities. CLI-first, EODHD-backed, TimescaleDB storage. PyPI distribution manta-trading-data, import package manta_trading, CLI entry point mt.

Installation

Requires Python 3.12+ and uv.

uv tool install manta-trading-data

mt --help should work immediately — no clone, no virtualenv activation.

The package is published on PyPI as manta-trading-data, but the Python import package is still manta_trading (import manta_trading) and the CLI command is still mt — only the install/upgrade name changed.

Updating

mt update          # check PyPI and install a newer release (prompts first)
mt update --yes    # non-interactive: install without prompting
mt update --json   # pure query: report versions, change nothing

mt update upgrades uv tool installs itself; on pipx or pip installs it prints the right command for your environment instead of running it. The equivalent manual command is always:

uv tool install --upgrade --refresh-package manta-trading-data manta-trading-data@latest

(--refresh-package matters right after a release: uv resolves against cached index metadata, so without it the upgrade can succeed while installing nothing. mt update runs this exact command and verifies the version moved.)

In a development (editable/source) checkout mt update refuses and points you at git pull && uv sync — it makes no network call there.

Development setup

To work on the code itself, use a source checkout instead:

git clone https://github.com/manta-digital/trading-data
cd trading-data
uv sync
source .venv/bin/activate   # macOS / Linux
# .venv\Scripts\activate    # Windows

mt --version reports dev in a source checkout (no installed distribution metadata to read); a uv tool install reports the real published version.

Environment

Copy .env_sample to .env and fill in the values.

Setting up a new database

The migration chain is the single source of schema truth. Bringing a fresh, empty Postgres database to the current schema is one command:

# 1. Create the database (TimescaleDB extension must be available on the instance).
PGPASSWORD=… createdb -h <host> -U postgres trading

# 2. Point at it and initialize.
export MT_TIMESCALE_DB_URL=postgresql://postgres:…@<host>:5432/trading
mt data init

mt data init is idempotent — re-running it on a healthy database applies zero migrations. Use --validate-only to inspect without changing anything.

Verify after init:

mt data migrate status   # all rows should report "applied"
mt data caggs status     # 7 caggs, all with a refresh policy

Typical workflows

First-time universe build

# Rebuild the instrument registry from EODHD (~33k symbols after OTC filter).
# Finnhub enrichment populates first_listing_date and promotes venue from
# transient 'US' to authoritative exchange. Takes ~9 hours at 60 req/min.
mt data instruments rebuild

# Skip Finnhub if you want registry populated quickly without IPO dates.
mt data instruments rebuild --skip-finnhub

# Populate delisted_date for delisted symbols (slice 159).
mt data instruments populate-delisted-dates

Ongoing data acquisition

# Run daemon indefinitely: daily + minute cycles + once-per-day CA update.
# Defaults to full active universe. Ctrl-C or SIGTERM exits cleanly.
mt data daemon run

# Limit to minute data only; exit when universe is fully caught up.
mt data daemon run --minute --stop-when-done

# Limit to a named list; stop when done.
mt data daemon run --list priority1 --stop-when-done

# Limit to specific symbols; stop when done (--stop-when-done implied).
mt data daemon run --symbols AAPL,MSFT,SPY

# Cap credit spend.
mt data daemon run --max-credits 5000

Targeted gap fill

# Fetch all UNKNOWN daily gaps for the full universe.
mt data pull 1d --universe

# Fetch minute gaps for a specific symbol.
mt data pull 1m --symbol AAPL

# Fetch minute gaps for a named list, verbose progress.
mt data pull 1m --list priority1 -v

# Preview what would be fetched without making changes.
mt data pull 1m --universe --dry-run

# Reset terminal gaps (PROVIDER_HOLE / RETRY_EXHAUSTED) then refetch.
mt data pull 1m --symbol AAPL --reset

# Include delisted symbols (requires --universe).
mt data pull 1d --universe --include-delisted

Reading data

# Read adjusted daily bars for AAPL (default: adjusted=True).
mt data get AAPL 1d

# Read raw minute bars for a date range.
mt data get AAPL 1m --start 2024-01-01 --end 2024-03-31 --raw

# Output as JSON or CSV.
mt data get AAPL 1d --json
mt data get AAPL 1d --csv

System health

# Show non-OK symbols (GAPS, STALE, FAILED) — default view.
mt data status

# Show all symbols including OK.
mt data status --all

# Drill into one symbol: detail panel + full gap listing.
mt data status --symbol AAPL

# Filter to daily or minute only.
mt data status --daily
mt data status --minute

# Machine-readable output.
mt data status --json

Corporate actions

# Bulk-fetch yesterday's splits + dividends for the full exchange (200 credits).
mt data ca update

# Full history for a single symbol.
mt data ca update --symbol AAPL

# Full history for a named list.
mt data ca update --list priority1

# Inspect stored CA data.
mt data ca show --symbol AAPL
mt data ca list --from 2024-01-01 --to 2024-12-31

Symbol lists and index universes

# List defined named lists with member counts.
mt data lists ls

# Print members of a list.
mt data lists show priority1

# Refresh the S&P 500 snapshot.
mt data lists refresh-sp500

# Show tracked index universes.
mt data universes ls

# Members of SP500 as of a date (point-in-time, survivorship-bias-free).
mt data universes as-of --name sp500 --date 2020-01-01

# Refresh index constituent tracking from source.
mt data universes refresh

Continuous aggregates

# Status of all 7 caggs (last refresh, policy, row counts).
mt data caggs status

# Manually refresh all caggs (useful after a large backfill).
mt data caggs refresh

# Refresh a specific granularity.
mt data caggs refresh --granularity 1h

Trading session horizon

# Extend trading_sessions for all calendars (usually automatic via daemon/status).
mt data extend

# Extend a specific calendar.
mt data extend --calendar NYSE

# Alert if horizon is < 90 days out (useful in CI).
mt data extend --strict

Schema migrations

# Check migration state.
mt data migrate status

# Apply pending migrations.
mt data migrate apply

Data Serving API

# Start the API server (default: 0.0.0.0:8100).
mt serve

# Custom host/port, multiple workers.
mt serve --host 127.0.0.1 --port 8200 --workers 4

# Dev mode with auto-reload.
mt serve --reload

API endpoints:

• GET /api/v1/health — liveness check, plus a coarse coverage freshness signal
• GET /api/v1/bars/{symbol}?granularity=1d&start=…&end=…&adjusted=true — OHLCV bars. Responses carry is_stale: true means the continuous aggregate serving this granularity is behind its source, so the bars may be incomplete. Raw grains (1m, 1d) are never stale by construction.
• GET /api/v1/symbols?search=<prefix> — list instruments
• GET /api/v1/symbols/{symbol} — instrument detail + available data ranges. See available semantics below for what the reported range does and does not guarantee.
• GET /api/v1/status?symbol=…&health=…&granularity=…&all=true — per-symbol data-health rows, a whole-registry health summary, and coverage freshness. rows defaults to unhealthy entries only (GAPS, STALE, FAILED), matching mt data status; pass all=true for everything or health=OK for healthy rows. A healthy symbol therefore returns count: 0 by default — that means "nothing wrong", not "no such symbol". summary is always the full unfiltered whole-registry breakdown, whatever rows was filtered to.
• GET /api/v1/gaps/{symbol}?granularity=1m — data gap listing
• GET /docs — Swagger UI

The full schema is committed at docs/api/openapi.json and regenerated with uv run python scripts/dump_openapi.py (no database required).

available semantics

GET /api/v1/symbols/{symbol} reports one {start, end} per granularity. The two ends are computed differently and carry different guarantees, which matters if you use them to decide what to request:

• end is exact. It comes from a direct probe of the bar tables, bounded so it stays fast, and it reflects data written right up to the moment of the request. If a bar exists, end includes it.
• start is as of the last coverage materialization. It comes from the coverage continuous aggregates, which a background policy refreshes. Deep history backfilled after the relevant coverage bucket was last materialized will not move start until that bucket is rebuilt — so start can be later than the true first bar, never earlier. There is no cheap exact answer here: probing below the coverage floor costs 0.4–1.4 s per symbol on production (measured), because the bound excludes chunks after the start, which for a symbol with deep history is almost none of them.

Both ends are UTC dates. A granularity with no data is omitted entirely — an empty available means "no bars for this symbol", not "unknown symbol" (an unknown symbol is a 404).

One documented gap. The leading-edge probe is bounded by a universe-wide coverage edge rather than each symbol's own. A bar could in principle be missed if it falls between an individual symbol's coverage end and that universe edge and was written after coverage last materialized. Measured across a 28-symbol sample on production 2026-08-04 — dense, delisted, daily-only, and no-data instruments — the merged answer was identical to a direct MIN/MAX scan for every symbol, and no symbol had a single raw bar inside that window. The gap closes on its own when the coverage refresh repair lands.

Error shapes

Every error this server raises has the same body:

{ "error": "<message>" }

The one deliberate exception is FastAPI's own request-validation failure — an unparseable date, an unknown granularity — which keeps its native body so clients retain the per-field detail:

{ "detail": [ { "loc": ["query", "granularity"], "msg": "…", "type": "…" } ] }

Date windows are inclusive at both ends

start and end are both inclusive, at every granularity: start=2024-06-10&end=2024-06-14 returns Monday through Friday, and start=2024-06-10&end=2024-06-10 returns that whole day. Timestamps are UTC, and the store covers 08:00–23:59 UTC.

Empty windows are 200, not 404

A known symbol with no bars in the requested window returns 200 with count: 0 and bars: [] — a weekend, a holiday, or a pre-listing date is not an error. is_stale is still populated, so "no bars and the aggregate is stale" is distinguishable from "no bars because the market was closed". A 404 now means exactly one thing: the symbol is unknown.

Range cap

A bars request is admitted or rejected before any database work, from an estimate computed from the window alone: span_days × bars_per_trading_day × (252/365). Exceeding MT_API_MAX_BARS_PER_REQUEST (default 75,000) is a 422 whose message names the estimate, the ceiling, and the maximum span for that granularity. There is no pagination and no silent truncation.

Because the store covers extended hours (08:00–23:59 UTC, ~960 one-minute bars on a dense day), the cap binds only at intraday grains:

For bulk history beyond these spans, query TimescaleDB directly rather than paging over HTTP.

Server settings

Both are read once at startup; changing either requires a server restart. Note they interact — raising the bar ceiling without also raising the timeout trades a fast 422 for a slow 504.

The API is unauthenticated and CORS-open by design: it is read-only and bound to a LAN host. Exposing it beyond the LAN, or adding any route that writes, makes authentication a prerequisite.

Integration tests

Most integration tests under test/integration/ require MT_TIMESCALE_DB_URL set to a database that already has the schema applied. They run against that DB and use a per-test fixture to reset state.

test/integration/test_cold_start.py is the exception: it creates and drops throwaway UUID-named databases for each test, so it requires an admin connection:

export MT_TIMESCALE_TEST_URL=postgresql://postgres:…@<host>:5432/postgres
uv run --extra dev pytest test/integration/test_cold_start.py

CI wiring for integration tests is tracked in issue #17.

Project structure

src/manta_trading/
  api/                   # Outbound provider HTTP clients (EODHD, Finnhub)
  api_server/            # FastAPI app (mt serve)
  cli/                   # Typer CLI commands
  config/                # Settings (pydantic-settings, MT_* env vars)
  data/
    acquisition/         # Daemon, orchestrators, gap tracking
    adjustment/          # Adjusted-on-read: compute_k_factor, adjusted()
    base/                # InstrumentRegistry, TradingCalendar
    maintenance/         # auto_extend, status_queries
    universe/            # EODHD symbol-list client, Finnhub IPO client
  market/
    schema/              # Migration definitions and runner
config/
  symbol-lists.yaml      # Named symbol lists (priority1, priority2 / sp500)
test/
  unit/                  # Unit tests (no DB required)
  integration/           # Integration tests (require MT_TIMESCALE_DB_URL)