gdeltforge 0.14.0

Reproducible sampling and cross-referencing of the full GDELT archive, on one machine.

Documentation  ·  Getting started  ·  CLI reference  ·  Recipes

pip install gdeltforge
gdeltforge --help

GDELT publishes its full archive (Events since 1979, GKG 1.0 since 2013, GKG 2.1 and Mentions since 2015) as hundreds of thousands of raw files. Drawing a sample from it that another researcher can reproduce, and whose relation to the archive is known, is harder than it should be: GDELT's search APIs answer bounded queries (the DOC API, for instance, returns up to 250 articles from the last three months), and BigQuery's tables stay in Google's warehouse, not on your own disk.

GdeltForge is a command-line pipeline that downloads that archive, converts it to Parquet, repairs GDELT's documented errors, and draws seeded samples whose design (simple random, stratified by period or by column) is stated explicitly, on a single machine. It was written for research on representation learning over GDELT events; see Background and citing.

• Downloads the full archive: Events since 1979, GKG 1.0 since 2013, GKG 2.1 and Mentions since 2015
• Stores it as Parquet, as published (convert) and repaired (clean) in separate directories
• Repairs documented GDELT errors, keeping every row by default; audits every run
• Seeded sampling: simple random, by period, filtered, stratified
• Joins sampled Events to GKG, keeping the many-to-many structure
• Runs each stage (scrape/convert/clean/sample/crossref) as its own command

Contents

• Why the Raw Archive
• Related Work
• Pipeline Design Principles
• Installation
• Project Structure
• Command Line Interface
• Usage Examples
• Logging System
• Limitations and Roadmap
• Contributing
• License
• Background and Citing
• Data Attribution

Stack

Why the Raw Archive

Of the ways GDELT offers its data (see Related Work for others), three matter here: its search APIs, BigQuery, and the raw archive, the one published as files to download.

GDELT API Limitations

GDELT's search APIs answer bounded queries about news coverage:

• the DOC API searches the last 3 months by default and returns up to 250 results per query
• the GEO API covers the last 15 minutes to 7 days
• the TV API reaches back to 2009, over television coverage

None of them returns the Events, Mentions or GKG tables themselves.

Google BigQuery Mirror Constraints

BigQuery holds the full history and answers SQL over it, within a free tier of 1 TiB of queries a month. Its tables are hosted by Google, not kept on your own disk, and some research workflows need a local, offline copy.

The Raw Bulk Archive

The raw GDELT archives contain hundreds of thousands of files (the Events directory alone about 5,000). Processing them requires:

• automated downloading
• streaming or chunked processing
• efficient columnar storage (Parquet)
• memory-safe filtering and sampling

GdeltForge performs these steps, one command per stage.

Related Work

Several other tools work with GDELT, each the better choice for some questions (a recent window, an analysis that fits in memory, SQL over the whole archive). GdeltForge focuses on two things. It treats sampling as a stage of its own: seeded designs that stream over the full archive on one machine and can be stated in a methods section. And it joins Events to GKG 2.1, which carries no event identifier, through Mentions, keeping the many-to-many relation between events and articles.

See Related Work for the other tools, what each is designed for, and when each fits better.

Pipeline Design Principles

The pipeline follows a single-responsibility, single-stage execution model. Each command performs exactly one transformation:

Each stage is explicit:

• every operation is explicit
• no hidden steps
• each module is individually testable
• any stage can be re-run without affecting others

The same stages run independently per dataset (--dataset events, gkg-v2, mentions, ...); crossref is the one stage that reads two datasets at once, joining a sampled Events output against a GKG dataset processed the same way.

Each stage consumes the output of the previous one, enabling incremental execution, streaming-friendly operations, memory-efficient processing, simple debugging, and reusable intermediate data.

Installation

pip install gdeltforge
gdeltforge --help

No config file required to get started: GdeltForge falls back to a conservative built-in default (real ./data/... paths, no row/column filtering) and writes it to config/settings.yaml the first time you run a command with nothing else configured, so it works immediately in a fresh environment like a Colab session. See Configuration to customize it instead.

# Install uv (if not already installed)
pip install uv

git clone https://github.com/Vinicius-Teixeirac/GdeltForge.git
cd GdeltForge

# Create virtual environment, install dependencies, and install GdeltForge itself
uv sync

# Activate the virtual environment
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

# Verify the CLI is installed
gdeltforge --help

cp config/settings.example.yaml config/settings.yaml

uv sync --group dev
uv run pytest

Project Structure

GdeltForge follows a standard src/ package layout, installable via uv sync / pip install.

project_root/
├── config/
│ └── settings.example.yaml # Annotated starting point; copy to settings.yaml and edit
│
├── src/gdeltforge/
│ ├── py.typed # PEP 561 marker: this package ships inline type hints
│ ├── cli.py # Argument parsing + subcommand dispatch
│ ├── launcher.py # The gdeltforge command's entry point: sizes polars' thread pool before polars loads, then runs cli
│ ├── __main__.py # python -m gdeltforge, through the same launcher
│ ├── _version.py # Generated at build time from the git tag (hatch-vcs); not hand-edited
│ │
│ ├── config/
│ │ └── default_settings.yaml # Bundled fallback config, used when no settings.yaml/--config is found at all
│ │
│ ├── data/
│ │ ├── cameo_codes.json # Bundled CAMEO/FIPS code -> name reference data, backs the `codes` command
│ │ └── places.parquet # Bundled place table: one place per (FeatureID, Lat, Long) GDELT wrote, backs clean.places
│ │
│ ├── aggregation/
│ │ └── aggregator.py # Concatenates a period's 15-minute files (GKG 2.1, Mentions, events-15min) into one per day/month/year
│ │
│ ├── conversion/
│ │ └── converter.py # CSV -> Parquet conversion logic
│ │
│ ├── crossref/
│ │ └── crossref.py # Events<->GKG join (direct for GKG 1.0, two-hop via Mentions for GKG 2.1)
│ │
│ ├── cleaning/
│ │ ├── cleaner.py # The clean stage: runs its steps on every file
│ │ ├── places.py # Loads the bundled place table for the places step
│ │ └── steps.py # The stage's steps, applied in one fixed order
│ │
│ ├── filtering/
│ │ └── filter.py # Deprecation shim: the pre-0.12 filter names, forwarding to cleaning/
│ │
│ ├── sampling/
│ │ ├── cameo_codes.py # Loads data/cameo_codes.json, groups columns by code family, backs the `codes` command
│ │ ├── indexer.py # File indexing for reproducible sampling
│ │ ├── rng.py # Random number generation helpers
│ │ └── samplers.py # Indexed, calendar, and filtered sampling
│ │
│ ├── scraping/
│ │ └── scraper.py # Downloader for Events, GKG 2.1, GKG 1.0, and Mentions
│ │
│ └── utils/
│   ├── branding.py # Terminal ANSI colors and the --version banner (brand system's terminal voice)
│   ├── concurrency.py # Sizes each worker's polars thread pools to its share of the machine, and caps concurrent reads (io.max_concurrent_reads)
│   ├── config.py # Config resolution (--config / env var / CWD / bundled default) and YAML loading
│   ├── io.py # File and chunked-IO helpers
│   └── logging.py # Central logging system
│
├── docs/assets/brand/ # Emblem, favicon, wordmark lockups, icon set, GitHub social preview
├── tests/ # pytest suite (unit tests, no network/browser required)
├── tools/ # Maintainer tools that build the place table, with the gazetteer references and repairs it cites
│
├── main.py # Backward-compatible shim: `python main.py <command>` still works
├── pyproject.toml # Package metadata, build backend, gdeltforge console-script entry point
└── README.md

Command Line Interface

Run pipeline stages using the installed console script:

gdeltforge <command> [options]

python main.py <command> [options] is kept as a backward-compatible equivalent, so existing scripts keep working unchanged. This alias only exists in a source checkout, not in the PyPI package installed above; gdeltforge itself always works either way.

The CLI intentionally does not chain stages automatically. You run each stage explicitly to maintain full control.

By default the config is read from ./config/settings.yaml (relative to wherever you run the command from). To point at a config file anywhere else on disk, use --config /path/to/settings.yaml or set the GDELTFORGE_CONFIG environment variable, useful once GdeltForge is installed and invoked from outside the repo checkout. If none of those resolve to a real file, GdeltForge falls back to a built-in default instead of failing; see Configuration.

Usage Examples

Examples of the common workflows, one stage at a time.

Scrape Raw Data

Download the entire archive (--dataset is required on every command; see CLI Reference for the other six choices):

gdeltforge scrape --dataset events

Download only files within a date range (any combination of bounds is valid):

gdeltforge scrape --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge scrape --dataset events --start-date 2022-01-01          # from date onward
gdeltforge scrape --dataset events --end-date   2015-12-31          # up to date

The filter applies to all three file types the GDELT archive provides:

Files already present in the download directory are skipped regardless of the date filter.

Downloads run concurrently (scraping.max_workers in config/settings.yaml, default 8) and are checksum-verified: the GDELT index publishes an MD5 per file, which the scraper captures and checks against each download after it completes. A mismatch is treated the same as a network failure: the file is discarded and retried up to scraping.retries times before being reported as failed, so a corrupted or truncated download never silently ends up in the dataset.

Convert CSV to Parquet

gdeltforge convert --dataset events

Extracts all CSV files from the downloaded ZIP archives and converts them to Parquet files. Each ZIP is processed independently, so conversion runs across a pool of worker processes (converter.max_workers in config/settings.yaml; null, the default, uses all available CPU cores).

paths:
  # existing paths ...
  parquet_historical_directory: "./data/events/historical"
  cleaned_historical_directory: "./data/events/cleaned_historical"

converter:
  partitioning:
    enabled: true
    rules:
      - file_type: yearly    # e.g. 1979.zip
        by: ["Year"]
      - file_type: monthly   # e.g. 200601.zip
        by: ["Year", "MonthYear"]

data/events/
├── parquet/                  # daily files (2013-present), unchanged
│   ├── 20130401.export.parquet
│   └── ...
└── historical/               # yearly/monthly files, Hive-organized
    ├── Year=1979/
    │   └── 1979.parquet
    ├── Year=2006/
    │   └── MonthYear=200601/
    │       └── 200601.parquet
    └── ...

convert also accepts --start-date/--end-date, same as scrape, narrowing which already-downloaded ZIPs get converted:

gdeltforge convert --dataset events --start-date 2020-01-01 --end-date 2020-12-31

Clean the Parquet Dataset

gdeltforge clean --dataset events

The data-quality stage. By default it repairs known GDELT errors (the events GDELT dated 1920 for 2020, with GDELT's own values kept in *_original columns) and keeps every row; rows missing a value in the columns listed under clean.columns_to_check are dropped. Every cleaned file is marked as cleaned, every run leaves an audit, and --dry-run --report measures what each step would change before anything is written. See Data Cleaning. Also accepts --start-date/--end-date, narrowing which already-converted files get read (which files get filtered, not which rows survive within them).

Sampling

All sampling modes read from the cleaned directory by default. Pass --source converted to sample from raw converted Parquet instead, skipping the clean stage entirely.

Note: sampling is already as memory-friendly as the underlying data volume allows, but --source converted will still use noticeably more RAM than the default, since it isn't working from data that's already had its missing-value rows dropped.

Indexed Sampling (Uniform Random)

gdeltforge sample --dataset events --mode indexed -n 10000 --seed 123 --out sample.parquet

This samples 10000 instances considering the entire data.

Calendar Sampling (N Rows Per Period)

gdeltforge sample --dataset events --mode calendar --per-period 20 --out daily.parquet

This samples 20 instances per day from the entire period (1979-present).

Filtered Sampling (Using JSON Filters)

5000 events whose QuadClass is in {1,2}:

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"QuadClass": [1, 2]}' \
    -n 5000 \
    --out qc12.parquet

2000 "Verbal Cooperation" events that happened in the USA (ActionGeo_CountryCode is a FIPS 2-letter code, US, not the 3-letter CAMEO code USA that Actor1CountryCode/Actor2CountryCode use):

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"], "QuadClass": [1]}' \
    -n 2000

You can select specific columns of interest, which is a memory friendly practice:

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"], "QuadClass": [1]}' \
    --columns GlobalEventID Year Actor1Code \
    -n 1000

It outputs 1000 instances following the same rule as before, but this time with only three columns.

Stratified Sampling (Fixed N Per Group)

Combines a filter with stratified reservoir sampling: draws exactly --n-per-group rows for each distinct value of a chosen column.

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"]}' \
    --stratify QuadClass \
    --n-per-group 500 \
    --out stratified.parquet

This produces a balanced dataset with 500 USA events per QuadClass value, regardless of the natural class distribution.

--stratify requires --n-per-group. The -n flag is ignored when --stratify is set.

Cross-Referencing Events with GKG

crossref enriches a sampled Events output with GKG (themes, tone, people, organizations). --gkg-version picks the join strategy: GKG 1.0 carries EventIds directly (a direct join), while GKG 2.1 carries no event ID at all and needs a two-hop join through Mentions on the source article's URL. --gkg-version auto attempts every eligible event against both generations instead of requiring one for the whole sample, useful for a sample spanning both eras (GKG 2.1/Mentions don't exist before 2015-02-18; GKG 1.0 has covered everything since 2013-04-01, and remains live today). All preserve the real many-to-many structure (one event can produce several output rows, one article covering several events contributes one row per event) instead of collapsing it.

gdeltforge scrape --dataset gkg-v2
gdeltforge scrape --dataset mentions
gdeltforge convert --dataset gkg-v2
gdeltforge convert --dataset mentions
gdeltforge clean --dataset gkg-v2
gdeltforge clean --dataset mentions

gdeltforge crossref \
    --events sample.parquet \
    --gkg-version v2 \
    --out sample_with_gkg.parquet

See Recipes for the GKG 1.0 direct-join variant and more detail.

Full Pipeline Examples

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge clean --dataset events
gdeltforge sample --dataset events --mode indexed -n 10000

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge clean --dataset events
gdeltforge sample --dataset events --mode indexed -n 5000 --seed 42

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge clean --dataset events
gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"]}' \
    -n 3000

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge clean --dataset events
gdeltforge sample --dataset events --mode calendar --per-period 30

gdeltforge scrape --dataset events && \
gdeltforge convert --dataset events && \
gdeltforge clean --dataset events && \
gdeltforge sample --dataset events --mode indexed -n 10000

foreach ($c in "scrape", "convert", "filter") {
    gdeltforge $c --dataset events
}
gdeltforge sample --dataset events --mode indexed -n 10000

gdeltforge scrape --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge convert --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge clean --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge sample --dataset events --mode indexed -n 10000 --start-date 2020-01-01 --end-date 2023-12-31

Further Examples

The complete filtered-sampling syntax reference and a full set of runnable recipes are in the documentation: see Filtered Sampling and Recipes. For which classical sampling algorithm each mode actually runs, see Sampling Theory.

Logging System

Logging is enabled through a shared logger helper:

from gdeltforge.utils.logging import get_logger
logger = get_logger(__name__)

Logging to a file:

logger = get_logger(__name__, log_to_file=True)

Limitations and Roadmap

GdeltForge is intentionally simple: one pipeline stage per command (no automatic chaining or dependency resolution), CSV -> Parquet only, and scrape/convert/clean now exit non-zero if any individual file fails.

The full, current list of limitations and the roadmap live in one place, the docs site, rather than duplicated here where they'd inevitably drift: see Limitations & Roadmap.

Contributing

Contributions are welcome. See CONTRIBUTING.md for dev setup, the branch/commit conventions this repo follows, and how to propose a change. Please also read the Code of Conduct.

See CHANGELOG.md for a history of notable changes.

Found a security issue? See SECURITY.md rather than opening a public issue.

License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.

Background and Citing

GdeltForge was written for research on representation learning for structured events, comparing tabular, graph and text representations of GDELT events with a multi-domain geometric model. Its first use was the chapter "Multi-domain Geometric Representation Learning for Structured Events" (Freitas, Sakiyama and Marcacini, BRACIS 2026, Springer LNAI 17107, doi:10.1007/978-3-032-39895-6_13). About describes what that research needed and the design decisions that followed.

If GdeltForge contributes to published work, please cite it: see CITATION.cff, or GitHub's "Cite this repository". Please also cite the GDELT Project, below.

Author: Vinícius Teixeira Carvalho Freitas, Instituto de Ciências Matemáticas e de Computação (ICMC), Universidade de São Paulo, São Carlos, Brazil.

Data Attribution

GdeltForge processes data published by the GDELT Project, which makes it available for unlimited and unrestricted use, provided any use or redistribution includes a citation and a link to their site. GdeltForge itself is an independent, unofficial tool, not affiliated with, endorsed by, or sponsored by the GDELT Project.

The place table's repaired region names (see Data Cleaning) and its country points (see Data Cleaning) come from NGA's GEOnet Names Server. Toponymic information is based on the Geographic Names Database, containing official standard names approved by the United States Board on Geographic Names and maintained by the National Geospatial-Intelligence Agency. More information is available at the Resources link at www.nga.mil. The National Geospatial-Intelligence Agency name, initials, and seal are protected by 10 United States Code § Section 425.

The place table's US state and territory points come from the U.S. Geological Survey's Geographic Names Information System (GNIS), Domestic Names National, published 2026-09-29; the data carries no use constraints, and the USGS asks for acknowledgment in products derived from it.