marketgoblin

Download, store, and load financial market data — fast and without fuss.

marketgoblin is a lightweight market data platform built on Polars and yfinance. It fetches multiple datasets (OHLCV, shares-outstanding, dividends), slices them into monthly Parquet files, writes JSON sidecars with metadata, and lets you load them back with a single call.

Features

• Multi-dataset — OHLCV, shares-outstanding, and dividends selected via a Dataset enum; per-source dispatch makes it easy to add more
• Tidy stacked OHLCV — adjusted and raw prices live in one frame, distinguished by an is_adjusted bool column; one network call per symbol covers both
• Single-symbol and batch fetch — fetch() and fetch_many() with thread-pool concurrency
• Disk persistence — monthly .pq slices with atomic writes; JSON sidecar per slice
• Lazy evaluation — all data paths return pl.LazyFrame (Polars)
• Date flexibility — dates stored as int32 YYYYMMDD on disk; use parse_dates=True to get pl.Date
• Retry logic — YahooSource retries transient failures with exponential backoff (3 attempts)
• Rate limiting — fetch_many() respects a configurable requests-per-second cap (default: 2 req/s)
• Input validation — dates are validated before any I/O; unsupported (provider, dataset) pairs raise at the dispatch boundary
• Pluggable providers — subclass BaseSource, implement _build_dispatch(), register in one line; YahooSource and TiingoSource included

Installation

pip install marketgoblin

Or with uv:

uv add marketgoblin

For development:

git clone https://github.com/aexsalomao/marketgoblin
cd marketgoblin
uv sync --extra dev

Quick Start

import polars as pl
from marketgoblin import Dataset, MarketGoblin

goblin = MarketGoblin(provider="yahoo", save_path="./data")

# Fetch and persist OHLCV — tidy stacked frame: each trading day appears
# twice (is_adjusted=True / False). Filter to pick a variant.
lf = goblin.fetch("AAPL", "2024-01-01", "2024-03-31", parse_dates=True)
adjusted = lf.filter(pl.col("is_adjusted")).collect()
print(adjusted)

# Load back from disk (no network call)
lf = goblin.load("AAPL", "2024-01-01", "2024-03-31", parse_dates=True)
print(lf.collect())

# Shares outstanding — sparse, corporate-action-driven series
shares = goblin.fetch("AAPL", "2024-01-01", "2024-03-31", dataset=Dataset.SHARES, parse_dates=True)
print(shares.collect())

# Dividends — event-driven (typically quarterly)
dividends = goblin.fetch("AAPL", "2024-01-01", "2024-03-31", dataset=Dataset.DIVIDENDS, parse_dates=True)
print(dividends.collect())

# Batch fetch — failed symbols are logged, never crash the batch
results = goblin.fetch_many(["AAPL", "MSFT", "GOOGL"], "2024-01-01", "2024-03-31")
for symbol, lf in results.items():
    print(f"{symbol}: {lf.collect().height} rows")

Run the full walkthrough:

python example.py

API

MarketGoblin

MarketGoblin(provider: str, api_key: str | None = None, save_path: str | Path | None = None, **source_kwargs)

Datasets

OHLCV is returned as a tidy stacked frame: each trading day appears twice (is_adjusted=True and is_adjusted=False). Filter downstream (.filter(pl.col("is_adjusted"))) to pick a variant. Adjusted Open/High/Low are derived locally from the Adj Close / Close ratio — verified to match yfinance's auto_adjust=True output exactly while halving network calls.

Data on disk

Adding a Provider

import polars as pl

from marketgoblin import Dataset
from marketgoblin.sources.base import BaseSource, Fetcher

class MySource(BaseSource):
    name = "mysource"

    def _build_dispatch(self) -> dict[Dataset, Fetcher]:
        return {Dataset.OHLCV: self._fetch_ohlcv}

    def _fetch_ohlcv(self, symbol: str, start: str, end: str) -> pl.LazyFrame:
        ...  # return a normalized LazyFrame with an is_adjusted column

Per-dataset fetchers all share the (symbol, start, end) signature — there is no adjusted toggle, since OHLCV variants are stacked into a single frame distinguished by the is_adjusted column.

Then register it in goblin.py:

_SOURCES = {"yahoo": YahooSource, "tiingo": TiingoSource, "mysource": MySource}

Running Tests

pytest
pytest --cov=marketgoblin   # with coverage

License

MIT © Antônio Salomão