tse_tick

A Python library for parsing, filtering, and querying Nikkei NEEDS tick data from the Tokyo Stock Exchange.

Who it's for: Researchers working with NEEDS tick data who need to convert thousands of zipped CSVs into queryable Parquet stores, filter by ticker or event windows, and handle format changes across historical eras.

What it solves: NEEDS data is delivered as daily ZIP files (1–27 parts per day) with era-dependent schemas — 2016 used fixed-width records for indices, 2017+ switched to CSV, and individual stocks have 95 columns with complex quote-book nesting. This library detects the format automatically, validates for security, parses everything into clean DataFrames, and writes Hive-partitioned Parquet.

Data access required: This tool does NOT provide NEEDS data itself. You must have an institutional subscription (Nikkei NEEDS) and access to the raw TICST120/TICSS110/TICIT110/TICIS110 ZIP files. If your data is shared via Google Drive, see the rclone download guide for mirroring it to local disk.

Features

• 4 data types — TICST120 (individual stock ticks, 95 cols), TICSS110 (daily stock summary, 82 cols), TICIT110 (index ticks, 10 cols), TICIS110 (daily index summary, 17 cols)
• Multi-era format support — 2016 fixed-width (TICIT010/TICIS010) and 2017-2025 CSV, auto-detected from the ZIP filename
• Polars backend — fast CSV parsing, vectorized cleaning, memory-efficient
• CLI batch ingestion — tse-tick ingest converts entire years/months/date ranges to partitioned Parquet; --parallel N spreads independent trading days across a RAM-aware process pool
• Ticker filtering (--tickers) — keep only specific stock codes at read time
• Event-window extraction (--filter-csv) — extract ±N minute windows around corporate events with automatic after-hours reaction-anchor shifting
• Bilingual columns — English and Japanese column names via --language en|jp
• One-shot reader (read_ticks) — raw ZIPs → a ticker/time-filtered DataFrame with no Parquet store to build first; part-pruned for individual_stock (opens only the ticker's parts, not the whole day)
• One-call two-stage (extract_to_store) — ingest one or more tickers for a period into a reusable store and get the DataFrame back in one call (the recommended path for repeated reads / large multi-ticker extractions; no 10M-row cap)
• Name translation (translate) — look up the tse_tick equivalent of a yfinance / Polygon / ccxt call (tables in tse_tick/data/translations.json; override with TSE_TICK_TRANSLATIONS)
• Typed enums (DataType, Language) — autocomplete-friendly and accepted anywhere the magic strings are
• Security guards — ZIP bomb detection (5 GB max decompressed, 100:1 compression ratio cap, max 5 entries), path traversal prevention, query row limits (10M)

Installation

pip install tse-tick               # from PyPI: core (polars, pyarrow)
pip install "tse-tick[query]"      # + DuckDB-powered Parquet queries

To work from the latest (unreleased) source instead, install in editable mode:

git clone https://github.com/tse-tick/tse_tick.git
cd tse_tick

pip install -e .             # core: polars, pyarrow
pip install -e ".[query]"    # + DuckDB-powered Parquet queries
pip install -e ".[dev]"      # + everything for development (tests, linters, jupyter)

Requires Python ≥3.9. Core dependencies are polars and pyarrow; the query extra adds DuckDB (see pyproject.toml).

Quick Start

Python API — load a single ZIP

import tse_tick

# Load individual stock tick data (auto-detects data type and year)
df = tse_tick.create_df("path/to/HTICST120.20230104.1.zip", language="en")

# Load with Japanese column names
df_jp = tse_tick.create_df("path/to/HTICST120.20230104.1.zip", language="jp")

# Sample first 1000 rows only
df_sample = tse_tick.create_df("path/to/HTICST120.20230104.1.zip", rows=1000)

# Explicit data type and year (skip auto-detection)
df = tse_tick.create_df(
    "path/to/file.zip",
    auto_detect=False,
    data_type="individual_stock",
    year=2023,
)

CLI — batch ingest to Parquet

# Ingest a date range
tse-tick ingest \
    --data-type individual_stock \
    --period 20240201-20240205 \
    --input-root /path/to/TSE_DATA \
    --output-root /path/to/PARQUET_STORE

# Ingest a full year
tse-tick ingest \
    --data-type individual_stock \
    --period 2024 \
    --input-root /path/to/TSE_DATA \
    --output-root /path/to/PARQUET_STORE

# Ticker-filtered ingest (keep only specified stocks)
tse-tick ingest \
    --data-type individual_stock \
    --period 2024 \
    --input-root /path/to/TSE_DATA \
    --output-root /path/to/PARQUET_STORE \
    --tickers 7203,6758,9984

# Ticker filter from file (one ticker per line)
tse-tick ingest \
    --data-type individual_stock \
    --period 2024 \
    --input-root /path/to/TSE_DATA \
    --output-root /path/to/PARQUET_STORE \
    --tickers @ticker_list.txt

# Event-window filtered ingest (±120 min around each event)
tse-tick ingest \
    --data-type individual_stock \
    --period 20250106-20250131 \
    --input-root /path/to/TSE_DATA \
    --output-root /path/to/PARQUET_STORE \
    --filter-csv event_filter_list.csv \
    --window 120

CLI — export one ticker to CSV or Parquet

For a quick slice straight from the raw ZIPs — ideal if you don't write Python. For individual_stock + a ticker it is part-pruned (opens only the ticker's parts), so it's fast and the result is complete:

tse-tick export \
    --data-type individual_stock \
    --tickers 7203 \
    --period 20240201-20240205 \
    --input-root /path/to/TSE_DATA \
    --output toyota.csv            # .csv or .parquet, chosen by extension

Add --store /path/to/store to build a reusable Parquet store as it exports (the two-stage path — best when you'll read the data again; requires the [query] extra):

tse-tick export --data-type individual_stock --tickers 7203 \
    --period 20240201-20240205 --input-root /path/to/TSE_DATA \
    --output toyota.csv --store /path/to/toyota_store

Query the Parquet store

Note: the query functions (query_ticks, query_sql, get_available_*) require the [query] extra — pip install "tse-tick[query]" (DuckDB). On the core install, use the DuckDB-free read_parquet_partition(store, "individual_stock", date=..., ticker=...) instead.

import tse_tick

# Query specific ticker and date
df = tse_tick.query_ticks(
    "/path/to/PARQUET_STORE",
    data_type="individual_stock",
    ticker=7203,
    date="20240201",
    start_time="09:00:00",
    end_time="11:30:00",
)

# Get available dates and tickers
dates = tse_tick.get_available_dates("/path/to/PARQUET_STORE")
tickers = tse_tick.get_available_tickers("/path/to/PARQUET_STORE", date="20240201")

Feature extraction

import tse_tick

df = tse_tick.query_ticks("/store", ticker=7203, date="20220201")

# Bid-ask spread
spread = tse_tick.compute_spread(df)

# Order-book depth (levels per side; default 10)
depth = tse_tick.compute_depth(df, levels=5, side="both")

# Order flow imbalance over rolling window
ofi = tse_tick.compute_flow_imbalance(df, window="5min")

# All features in one pass
features = tse_tick.compute_all_features(df)

Two access patterns

tse_tick gives you a filtered DataFrame two ways. For anything you'll read more than once, prefer the two-stage store — you pay the raw scan once and every later query is sub-second.

1. Two-stage (recommended — scale / repeated work). ingest the raw ZIPs into a Hive-partitioned Parquet store once, then query_ticks it repeatedly (~694× faster than a pandas CSV scan; see Performance). extract_to_store(...) does both in one call:

import tse_tick

# The if __name__ == "__main__": guard is REQUIRED for max_workers > 1 when this
# runs as a .py script: parallel ingest starts 'spawn' worker processes, which
# re-import the script — an unguarded top-level call would re-run itself in every
# worker (tse_tick raises an explanatory error if you forget).
if __name__ == "__main__":
    # Build a reusable store for one or several tickers and get the DataFrame back —
    # in one call. No 10M-row cap, so a whole month of active tickers comes back
    # complete (past ~10M rows a capturable LargeResultWarning fires — for multi-year
    # periods, ignore the returned frame and read the store in query_ticks slices).
    df = tse_tick.extract_to_store(
        "/path/to/TSE_DATA",          # a .zip, flat folder, or ANY folder above the data
        "/path/to/PARQUET_STORE",     # reusable store (built once; resume-safe, part-pruned)
        "202402",                     # a day, month, year, or range (e.g. "2021-2023")
        ["7203", "9984"],             # one code, or a list — Toyota + SoftBank
        max_workers=8,                # parallel per-date ingest (capped by cores + RAM)
    )
    # ...every later read of the store is sub-second:
    df = tse_tick.query_ticks("/path/to/PARQUET_STORE", data_type="individual_stock",
                              ticker=7203, date="20240201")

2. One-shot (quick, targeted exploration). read_ticks(...) reads straight from raw ZIPs to a ticker/time-filtered DataFrame with no store to build first. For individual_stock + a ticker filter it is part-pruned — it opens only the small run of numbered parts that hold the ticker (not every part of the day), so a single-ticker read is several times faster while returning identical rows (pass prune_parts=False to force a full scan). Accepts a date range (date="20240201-20240205"); best for one or a few tickers over a bounded window. The tse-tick export CLI wraps it to CSV/Parquet for non-coders.

import tse_tick

# Toyota (7203) over a date range — straight from the raw ZIPs, no store.
# Part-pruned: opens only 7203's parts; the result is complete and exact.
df = tse_tick.read_ticks(
    "/path/to/TSE_DATA",          # a .zip, a flat folder, or ANY folder above the data (located by type+date)
    ticker_filter={"7203"},
    date="20240201-20240205",     # single day "20240201", a month "202402", a year "2024", or a range
    start_time="09:00:00",
    end_time="11:30:00",
)

⚠️ Reading a lot at once? Mind the 10M-row cap. read_ticks and query_ticks both return at most 10,000,000 rows per call by default — on hitting it, the result is truncated and a capturable tse_tick.TruncationWarning is emitted. A whole month of a couple of active tickers can exceed this: e.g. Toyota (7203) + SoftBank (9984) for one January is ~25M rows, so a single read_ticks(..., date="202201") stops at ~10M (about 10 of 19 days). To get everything, pick one:

• Two-stage (recommended for scale) — ingest_period(...) → query_ticks(...) in bounded slices (per day / per month; pass limit=None per slice for the full rows). The store is reusable and every slice read is sub-second.
• extract_to_store(...) — the only call with no row cap: it builds the store and returns the whole period as ONE in-memory DataFrame. Past ~10M rows a capturable tse_tick.LargeResultWarning fires first — for multi-year periods of an active ticker, ignore the returned frame and read the store in query_ticks slices instead (the frame can be tens of GB).
• Loop per day — call read_ticks(..., date=day) for each trading day and pl.concat(...); each day stays well under the cap.
• Lift the cap — read_ticks(..., rows=None) / query_ticks(..., limit=None) reads it all in one shot, bounded only by memory.

If a one-shot read might be large, check for a TruncationWarning (it's the signal to switch to the two-stage store):

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    df = tse_tick.read_ticks(root, ticker_filter={"7203", "9984"}, date="202201")
if any(issubclass(x.category, tse_tick.TruncationWarning) for x in w):
    ...  # truncated — build a store instead

Data Types

Multi-Era Format Support

The format changed only once, after 2016, and only for the index types (fixed-width to CSV); individual stock and stock summary files were CSV throughout. The library detects the era automatically from the ZIP filename (the year) and applies the correct parser.

No user action needed — if your ZIP filename contains 2016, the fixed-width parser is used automatically for index data.

Performance

tse_tick is built on Polars (CSV parsing, vectorized cleaning) and DuckDB over Hive-partitioned Parquet (queries). Measured on one day of HTICST120 (4.78 M rows, 95 columns, 2.16 GB raw CSV) on an Intel Core i5-14400F (10-core / 16-thread) with 32 GB RAM, Python 3.11, Polars 1.40, pandas 2.2.

The three Polars speedup numbers are deliberately reported together: against the original pandas Python-engine prototype, against a fair C-engine baseline (all-string dtypes, forced column count), and at single-thread parity to isolate the contribution of threading from the engine itself. Polars wins on all three.

tse_tick defaults to Polars because the ingest workload (multi-GB daily CSVs, mostly columnar transformations) hits exactly the case where lazy expression planning and parallel CSV parsing dominate; pandas-on-DataFrame's row-oriented model leaves throughput on the table even with the C engine. For querying, the Parquet store + DuckDB combination converts repeated single-ticker / single-date filters from full file scans into partition pruning, which is the source of the ~700× query speedup.

To reproduce: python benchmarks/run_all.py (see benchmarks/ENVIRONMENT.md).

Expected Input Layout

The CLI expects NEEDS data organized as delivered by Nikkei:

{input_root}/
  2016/
    201601/
      HTICST120.20160104.1.zip
      HTICST120.20160104.2.zip
      ...
    201602/
    ...
  2017/
    201701/
    ...

Real NEEDS deliveries are often nested — e.g. 個別株式{year}/TICST120/{yyyymm}/HTICST120.*.zip (a Japanese-named year folder, then the data-type code, then the month). You don't have to match the strict layout above: point --input-root (or read_ticks(...) / tse-tick export) at any folder that contains the data — files are located by type + date, regardless of folder names or depth. Tip: aim at the common parent (e.g. G:\NEEDS) to cover several years at once.

Parquet Output Layout

Standard ingest produces Hive-partitioned Parquet per ticker per date:

{output_root}/
  individual_stock/
    date=20230104/
      ticker=7203.parquet
      ticker=6758.parquet
      ...

Event-window filtered ingest writes per-date files:

{output_root}/
  year=2025/
    month=01/
      20250106.parquet
      20250107.parquet
      ...

CLI Reference

Event Filter CSV Format

When using --filter-csv, the ingest path builds each window from zip_date, ticker, and reaction_anchor_dt. The CSV may also carry these descriptive columns:

For after-hours events, reaction_anchor_dt shifts to the next trading day's 09:00 open, and zip_date points to that next day's ZIP file. This is critical: centering on the event time (e.g., 15:30) would produce empty windows because the market is closed.

Python API Reference

create_df(folder_path, language="en", rows=None, auto_detect=True, data_type=None, year=None, ticker_filter=None, max_oneshot_bytes=<5 GB>)

Load and clean tick data from a ZIP file or directory of ZIP files.

• folder_path — path to a .zip file or directory of .zip files
• language — "en" or "jp" for column names
• rows — max rows to return
• auto_detect — if True, detect data type and year from path. If False, must provide data_type and year
• data_type — "individual_stock", "stock_summary", "indices", or "indices_summary"
• year — data year (e.g., 2023)
• ticker_filter — optional set of 4-digit stock codes to pre-filter at line level
• max_oneshot_bytes — cumulative decompressed-size ceiling for the one-shot read (default 5 GB; None disables). Crossing it raises a catchable OneShotMemoryError; use the two-stage ingest_* → query_ticks path instead.

Returns a Polars DataFrame with English or Japanese column names.

export_to_csv(folder_path, output_path=None, language="en", rows=None)

Load and export to CSV. If output_path is None, generates a filename.

read_ticks(source, *, data_type="individual_stock", ticker_filter=None, date=None, start_time=None, end_time=None, columns=None, rows=10_000_000, language="en", prune_parts=True, max_oneshot_bytes=...)

One-shot read: raw NEEDS ZIPs → a ticker/time-filtered DataFrame, no store.

• source — a .zip, a flat folder, or any folder above the data (located by type + date)
• ticker_filter — a set of codes (e.g. {"7203"}); a bare "7203"/7203 also works
• date — a day "YYYYMMDD", month "YYYYMM", year "YYYY", or a "start-end" range
• prune_parts — for individual_stock + a ticker_filter, open only the contiguous run of numbered parts that hold the ticker (plus the day's trailing appendix part) instead of every part. Falls back to a full scan if the ascending-code layout can't be confirmed, so results are identical — only faster. Default True; set False to force a full scan.

extract_to_store(input_root, output_dir, period, ticker, *, data_type="individual_stock", start_time=None, end_time=None, language="en", resume=True, max_workers=1)

Two-stage in one call: ingest ticker for period into a reusable, part-pruned Parquet store (output_dir), then return the queried DataFrame. ticker accepts one code or an iterable ("7203" or ["7203", "9984"]); several tickers are ingested in one pass and returned concatenated. Prefer it over read_ticks when the data will be read more than once — the raw scan is paid once, later query_ticks reads are sub-second, and there's no 10M-row cap. The result is scoped to period even on a reused store holding other days. max_workers>1 parallelizes the per-date ingest (from a script, the call must sit inside if __name__ == "__main__": — spawn workers re-import your script). Everything is returned as ONE in-memory DataFrame — past ~10M rows a capturable LargeResultWarning fires; for big periods read the built store in query_ticks slices instead. Requires the [query] extra (DuckDB).

Security

Built-in protections for local data processing:

Release History

See CHANGELOG.md for what changed in each release.

Notes for library users

• Quiet by default. create_df, read_ticks, and the ingest_* functions emit diagnostics via logging, not print, so they never write to stdout (or crash on non-ASCII paths) unless you opt in with logging.basicConfig(level=logging.INFO). The tse-tick CLI still prints progress.
• Windows-friendly print. On Windows, importing tse_tick switches Polars to ASCII table borders and reconfigures stdout/stderr to UTF-8, so a bare print(df) no longer raises UnicodeEncodeError on a cp1252 console — neither the box-drawing borders nor the content glyphs (datetime[μs], ≤ in column names, — in exchange values). Opt out of both with TSE_TICK_ASCII_TABLES=0; tse_tick.display(df) prints any DataFrame as UTF-8 on any platform regardless.
• Discovery round-trips. get_available_tickers(...) returns string codes (e.g. ["6758", "7203"]) you can pass straight to read_ticks(ticker_filter=...); alphanumeric codes (e.g. "130A") are preserved rather than dropped. (read_ticks / query_ticks also accept int codes.)
• Flexible discovery. Structured-root read_ticks / discover_zips find ZIPs under the documented {year}/{yearmonth}/ layout, a {yearmonth}/ folder directly under the root (e.g. a …/TICST120 type folder), and — as a fallback — recursively under nested delivery trees such as 個別株式{year}/TICST120/{yyyymm}/.
• One numbered ZIP is part of a day. NEEDS splits each day across parts by ascending code, so filtering a lone HTICST120.<date>.N.zip by ticker can return 0 rows (Toyota 7203 is in a later part) — pass the day's directory or a structured root for complete coverage.
• Numeric dtypes. Price/quote columns (Execution Price, Sell Quote 1 Best, …) are Float64, and all stock_summary measures (OHLC, VWAP, volumes, amounts, counts) are Float64 too — so .mean() and arithmetic work without manual casting. (Stores ingested before the relevant change held these as String — re-ingest to refresh.)
• Time filters keep the whole order book. For individual_stock, quote-only book updates have a blank Execution Time but a real Update Time; read_ticks / query_ticks time windows fall back to Update Time for those rows, so a session filter retains in-window quote updates (not just trade-coincident snapshots) — what compute_depth / compute_spread / compute_flow_imbalance need.
• Index codes are raw codes. indices and indices_summary both return Index Code as the raw numeric code (e.g. "101"), matching what you pass to ticker_filter and the ticker= partition; ticker_filter also accepts the display name ("Nikkei 225"). (Stores written before 0.7.0 held decoded names for indices — re-ingest to refresh.)
• Empty results keep their schema and warn. A read that matches nothing — a date with no ZIPs (e.g. a market holiday), an unknown ticker/index code, or an over-tight filter — returns an empty but fully-typed DataFrame (all columns present), so chained access like df["Exchange Code"] won't raise. read_ticks also emits a capturable tse_tick.NoDataWarning (a UserWarning) for every zero-row result across all four types, so "no data" is never silent — trap it with warnings.catch_warnings() or silence it with warnings.filterwarnings("ignore", category=tse_tick.NoDataWarning). The rows= cap likewise emits a capturable tse_tick.TruncationWarning when it truncates a result (the signal to build a store and use query_ticks).
• Big queries spill to system temp. DuckDB-backed queries (query_ticks, query_sql, extract_to_store's Stage-2) spill large sorts to <system temp>/tse_tick_duckdb_spill rather than a .tmp/ folder in your working directory. An interrupted query can leave files there; they are safe to delete.
• Summary stores are compact (date-partitioned). The two daily-aggregate types (stock_summary, indices_summary) write one Parquet file per date with the code kept as a column — not one tiny file per (date × ticker), which previously blew a 15 MB month up to a 2.4 GB store of ~87k files. query_ticks / get_available_tickers prune/read that column; the tick types keep per-ticker files. Re-ingest summary stores built before this change.
• Ingestion entry points are the functions ingest_period, ingest_single_zip, ingest_year_from_root, … — tse_tick.ingest itself is the submodule.
• Event windows. extract_event_window(store, ticker, event_date, event_time, before, after, data_type=...) returns the ticks around one event from a store, adding a seconds_from_event column (quote-only rows are timed via Update Time, like query_ticks). It works for the tick types (individual_stock, indices); before/after apply only when event_time is given (omit it for the whole day). extract_batch_event_windows runs many events and returns None for any that fail.

Contributing

Contributions are welcome. Please open an issue or submit a pull request.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/your-feature)
3. Commit your changes (git commit -m 'Add your feature')
4. Push to the branch (git push origin feature/your-feature)
5. Open a Pull Request

Development setup:

pip install -e ".[dev]"
pytest tests/ -v

Testing

pytest tests/ -v

The suite collects 430 tests. Without a local NEEDS store, 382 pass and 48 skip; with a complete NEEDS store, all 430 pass. Stage-1 (ingestion) and Stage-2 (query, order-book features, and event-window-from-Parquet) both run with no proprietary data — a session-scoped pytest fixture builds a tiny Hive-partitioned Parquet store at test time by feeding synthetic, obviously-fake individual_stock (TICST120) ZIPs through the real ingest pipeline (tests/synthetic_data.py, tests/conftest.py).

The 48 skips load real NEEDS files from local paths (test_real_data.py and the real-ZIP cases in test_ingest.py). They run automatically once a local NEEDS store is present (override the default root with TSE_TICK_DATA_ROOT).

Citation

If you use this software in your research, please cite it using the CITATION.cff file in the repository. A technical paper describing the library is in preparation.

License

MIT

Authors

• Kazumi Li — Schema definitions, package architecture, current maintainer
• Masataka Hayashi — Initial pandas-based prototype
• Peter Romero — Original concept and initial project design

Developed at Keio University, Nakatsuma Seminar.