datasets

A helper library to manage all kinds of datasets. Part of the QFIN workspace alongside equity, derivatives, fixed-income, and risk.

Install

python3 -m pip install -e .

Optional extras:

python3 -m pip install -e '.[data]'   # pandas, pyarrow, requests, python-dotenv, yfinance
python3 -m pip install -e '.[dev]'    # pytest, ruff, build

Massive providers require the [data] extra and MASSIVE_API_KEY in .env (see .env.example). FRED providers also need FRED_API_KEY.

DataSource and Dataset

• DataSource — a provider/backend (FRED, Massive, Treasury.gov, local CSV).
• Dataset — one specific dataset obtainable from that source, identified by a key.

Sources expose one or more dataset keys via source.datasets(). Use Dataset(source, key) as the handle for fetching a single dataset:

from datetime import date

from datasets.data import Dataset, MassiveDailyMarketSummaryRangeSource

source = MassiveDailyMarketSummaryRangeSource(date(2024, 1, 1), date(2024, 1, 5))
summary = Dataset(source, source.datasets()[0]).fetch()

Dataset also supports export-ready DataFrames and parquet export:

frame = Dataset(source, source.datasets()[0]).to_dataframe()
path = Dataset(source, source.datasets()[0]).export()

Core types

Type	Description
OhlcvBar / OhlcvHistory	Time-series OHLCV bars for any symbol (equity, index, option contract)
MarketBar / MarketSummary	Cross-sectional daily market snapshot
OptionContract / OptionContracts	Options reference data
TreasuryYieldPoint / TreasuryYieldCurve	U.S. Treasury constant-maturity yields (Massive)
CurvePoint / ParCurve	Treasury.gov par yield curve
RateObservation / RateSeries	Rate fixings (SOFR, FRED series)
FuturesSettle / FuturesSettleCurve / SofrFuturesSettleBundle	CME SOFR futures settles

Legacy names (StockBar, StockHistory, DailyMarketBar, DailyMarketSummary) remain available as deprecated aliases.

Rates market data (FRED / NY Fed / Treasury.gov / CME)

Source	Backend
TreasuryParCurveSource	Treasury.gov daily par yield CSV
NyFedSofrSource	NY Fed Markets API SOFR
FredSeriesSource / FredSofrSource / …	FRED API (FRED_API_KEY)
CmeSofrSettleCsvSource / CmeSofrSettleBundleSource	Local CME settle CSVs under data/raw/

from datetime import date
from datasets.data import NyFedSofrSource, TreasuryParCurveSource

par = TreasuryParCurveSource().fetch()
sofr = NyFedSofrSource().fetch(as_of=date(2026, 5, 22))

See docs/data_access.md. Ingest:

python3 scripts/ingest_market_data.py --as-of 2026-05-22

Massive.com providers

Source	Endpoint	Default window
MassiveDailyMarketSummarySource	Daily Market Summary	One trading date
MassiveStockHistorySource	/v2/aggs/ticker/{ticker}/range/1/day/...	5 calendar years
MassiveOptionContractsSource	All Contracts	Filtered contract index
MassiveOptionBarsSource	Custom Bars	1 calendar year
MassiveTreasuryYieldsSource	/fed/v1/treasury-yields	30 calendar days

Rate limits: default client enforces 5 calls/min (free tier). Downloads print an ETA from pending call count.

Download	API calls	~Time at 5/min
2-year daily market summary	~504 weekdays	~100 min
N tickers (5-year history)	N calls	N / 5 min
Option contracts (per underlying)	1+ paginated calls	varies
Option bars (per contract)	1 call	12 sec

Options example

from datetime import date

from datasets.data import Dataset, MassiveOptionBarsSource, MassiveOptionContractsSource

contracts_src = MassiveOptionContractsSource(underlying_ticker="AAPL", expired=False)
contracts = Dataset(contracts_src, contracts_src.datasets()[0]).fetch(as_of=date.today())

contract = contracts.contracts[0]
bars_src = MassiveOptionBarsSource(contract, lookback_days=90)
bars = bars_src.fetch()
frame = Dataset(bars_src, bars_src.datasets()[0]).to_dataframe()

Treasury yields example

from datetime import date

from datasets.data import Dataset, MassiveTreasuryYieldsSource

treasury_src = MassiveTreasuryYieldsSource(lookback_days=30)
curve = Dataset(treasury_src, treasury_src.datasets()[0]).fetch(as_of=date.today())
frame = Dataset(treasury_src, treasury_src.datasets()[0]).to_dataframe(as_of=date.today())

Yahoo Finance providers

Source	Endpoint	Default window
YahooIndexHistorySource	yfinance.Ticker(symbol).history(interval="1d")	5 calendar years

Rate limits: default client enforces 30 calls/min for bulk downloads to reduce throttling risk.

Download scripts

Edit the config block at the top of each script, then run:

cd datasets
python3 -m pip install -e '.[data]'

# Dry run (no network)
python3 scripts/download_daily_market_summary.py --dry-run
python3 scripts/download_stock_history.py --dry-run
python3 scripts/download_sector_index_history.py --dry-run
python3 scripts/download_option_contracts.py --dry-run
python3 scripts/download_treasury_yields.py --dry-run

# Full download (requires MASSIVE_API_KEY)
python3 scripts/download_daily_market_summary.py --lookback-years 2
python3 scripts/download_stock_history.py
python3 scripts/download_option_contracts.py --underlying SPY
python3 scripts/download_option_contracts.py --underlyings AAPL,MSFT
python3 scripts/download_stock_history.py --tickers AAPL,MSFT
python3 scripts/download_treasury_yields.py

# Full Yahoo download (no API key required)
python3 scripts/download_sector_index_history.py --lookback-years 10

Equity options (AAPL, MSFT): listed contracts are American-style. Use EQUITY_OPTION_UNDERLYINGS / equity_option_tickers() from datasets.data. Downstream BSM/IV in derivatives treats them as a European approximation for empirical study.

Outputs:

• data/processed/massive/daily_market_summary/{YYYY-MM-DD}.parquet
• data/processed/massive/stock_history/{SYMBOL}.parquet
• data/processed/massive/option_contracts/{UNDERLYING}_{YYYY-MM-DD}.parquet
• data/processed/massive/option_bars/{CONTRACT_TICKER}.parquet
• data/processed/massive/treasury_yields/{YYYY-MM-DD}.parquet
• data/processed/yahoo/index_history/{slug}.parquet

download_stock_history.py uses a TICKERS list at the top of the file (override with --tickers SPY,AAPL). download_option_contracts.py accepts --underlying SPY or --underlyings AAPL,MSFT. download_sector_index_history.py defaults to S&P 500 (^GSPC) plus 11 S&P 500 GICS sector indices, and supports overriding via --slugs sp500,energy,financials.

Quickstart

cd datasets
python3 -m pip install -e .
python3 examples/quickstart.py

Notebook example for S&P 500 and sector returns:

python3 -m pip install -e '.[data]'
python3 scripts/download_sector_index_history.py --lookback-years 10
jupyter notebook examples/sp500_10y_sector_returns.ipynb

import datasets
from datasets.data import project_data_dir

print(datasets.__version__, project_data_dir())

Project layout

datasets/
├── datasets/               # Python package
│   └── data/
│       ├── massive/        # Massive REST client and sources
│       ├── export.py       # Parquet writers
│       └── ...
├── data/
│   ├── raw/              # Manual CSV fallbacks (gitignored)
│   └── processed/        # Parquet snapshots (gitignored)
├── examples/             # Runnable example scripts and notebooks
├── scripts/              # Build and download helpers
└── tests/

Development

bash scripts/build_test.sh