gdelt-tui

Download the GDELT Global Database of Events and store it as partitioned Parquet, from a terminal UI.

pip install gdelt-tui
gdelt-tui

The TUI asks which table you want, which date range, and where to save, then shows per-file progress while it downloads and converts. q aborts after the file in flight.

The three tables

GDELT publishes the event table in three shapes. All three are supported.

Non-interactive use

Every flag works without the TUI, which is what you want in a script or cron job. Progress falls back to plain lines when stdout is not a terminal.

# last 7 days of events, the default window
gdelt-tui --dataset events

# an explicit range
gdelt-tui --dataset events --start 2024-01-01 --end 2024-01-31

# one month of history
gdelt-tui --dataset history --start 1979-01-01 --end 1979-01-31

Output layout

gdelt-data/
  raw/                                  zips, kept so you can reconvert
    events/2024/20240115.export.CSV.zip
  parquet/
    year=2024/date=2024-01-15/part-00000.parquet
    year=1979/part-00000.parquet

Hive-style partitioning, so Polars, pandas and DuckDB can scan a range without loading it all:

import polars as pl

events = pl.scan_parquet("gdelt-data/parquet", hive_partitioning=True).filter(
    pl.col("date") >= pl.date(2024, 1, 1)
)

-- DuckDB
SELECT Actor1CountryCode, count(*)
FROM read_parquet('gdelt-data/parquet/**/*.parquet', hive_partitioning=true)
WHERE year = 2024 AND QuadClass = 1
GROUP BY 1 ORDER BY 2 DESC LIMIT 10;

Library use

from datetime import date
from pathlib import Path

from gdelt_tui import BY_KEY, convert, download, fetch_index, select

events = BY_KEY["events"]
raw, out = Path("raw"), Path("parquet")

for entry in select(fetch_index(), events, date(2024, 3, 1), date(2024, 3, 2)):
    archive = raw / entry.name
    download(entry.url, archive, expected=entry.md5)
    convert(archive, events, out, period=entry.period)

Integrity

Every file carries an MD5 on the GDELT index page, so downloads are verified before the .part file is renamed into place. A file that fails its checksum is retried, then reported and skipped rather than aborting the run.

A note on the compiled wheel

The TUI is a Rust extension (ratatui via PyO3), which is what keeps the installed package to one dependency, polars, with no Python TUI libraries.

The trade-off is that this is a platform wheel, not a pure-Python package. pip install from a wheel is instant. Installing from the sdist — on musl Linux, on a platform without a published wheel, or from a source checkout — compiles the extension and therefore needs a Rust toolchain present. Building takes a couple of minutes.

License

MIT