gdeltforge 0.10.0

Global event data, forged.

Forges the raw GDELT archive into clean, reproducibly-sampled, cross-referenced Parquet.

Documentation  ·  Getting started  ·  CLI reference  ·  Recipes

pip install gdeltforge
gdeltforge --help

GDELT's Events archive spans hundreds of millions of rows across 50+ years, but the official API caps queries at ~250 rows and BigQuery's free tier can't cover a full historical pull. GdeltForge downloads, converts, filters, and reproducibly samples the whole archive locally: checksum-verified downloads, concurrent I/O, and reservoir sampling built for a single pass over the full history. It also scrapes the Global Knowledge Graph (both current GKG 2.1 and legacy GKG 1.0) and Mentions, and can cross-reference a sampled Events output back onto GKG (themes, tone, people, organizations) with crossref.

• Full historical archive, not just the last 3 months the API allows
• Events enriched with GKG via crossref, preserving the real many-to-many structure instead of collapsing it
• Efficient columnar storage (Parquet) instead of raw CSV/ZIP
• Reproducible sampling: indexed, calendar, and filtered modes, filtered mode also supports stratified sampling (fixed N per group)
• Each stage (scrape/convert/filter/sample/crossref) runs independently and inspectably

Contents

• Challenges with Official Access Methods
• Comparison to Other GDELT Tools
• Pipeline Design Principles
• Installation
• Project Structure
• Command Line Interface
• Usage Examples
• Logging System
• Limitations and Roadmap
• Contributing
• License
• Data Attribution

Stack

Challenges with Official Access Methods

The GDELT Events Database is extremely rich, but extracting the full archive through official methods is difficult. This pipeline solves the following issues:

GDELT API Limitations

The standard GDELT APIs are optimized for recent and small queries. They impose strict limits on:

• time windows (often only the last 3 months)
• maximum rows per query (≈ 250)
• request rate limits

These constraints make it effectively impossible to retrieve the full historical dataset.

Google BigQuery Mirror Constraints

While BigQuery provides full access, it suffers from practical issues:

• free-tier quotas (1TB/month query, 10GB storage, 1GB egress) are far too small for the ~hundreds of GB required
• full-table scans are expensive without billing
• some researchers prefer local offline workflows

Raw Bulk Downloads Are Available, But Hard to Use

The raw GDELT archives contain thousands of files. Processing them requires:

• automated downloading
• streaming or chunked processing
• efficient columnar storage (Parquet)
• memory-safe filtering and sampling

This pipeline automates these steps end-to-end.

Comparison to Other GDELT Tools

GdeltForge's genuinely distinguishing feature is treating reproducible sampling as a first-class pipeline stage, not download-and-convert alone: seeded indexed, calendar, and filtered reservoir sampling over the full archive in a single streaming pass, filtered mode also supports stratified sampling. The other is crossref: GKG 2.1 carries no event ID at all, only the source article's URL, so joining it to Events means a two-hop trip through Mentions that's easy to get subtly wrong; crossref does that join with filter pushdown and keeps the real many-to-many structure intact instead of silently collapsing it.

See Comparison to Other Tools for an honest breakdown of when GdeltForge fits and when a DOC API client, BigQuery, or an existing Spark/DuckDB pipeline is the better choice.

Pipeline Design Principles

The pipeline follows a single-responsibility, single-stage execution model. Each command performs exactly one transformation:

This design is intentionally transparent and low-magic:

• every operation is explicit
• no hidden steps
• each module is individually testable
• any stage can be re-run without affecting others

The same stages run independently per dataset (--dataset events, gkg-v2, mentions, ...); crossref is the one stage that reads two datasets at once, joining a sampled Events output against a GKG dataset processed the same way.

Each stage consumes the output of the previous one, enabling incremental execution, streaming-friendly operations, memory-efficient processing, simple debugging, and reusable intermediate data.

Installation

pip install gdeltforge
gdeltforge --help

No config file required to get started: GdeltForge falls back to a conservative built-in default (real ./data/... paths, no row/column filtering) and writes it to config/settings.yaml the first time you run a command with nothing else configured, so it works immediately in a fresh environment like a Colab session. See Configuration to customize it instead.

# Install uv (if not already installed)
pip install uv

git clone https://github.com/Vinicius-Teixeirac/GdeltForge.git
cd GdeltForge

# Create virtual environment, install dependencies, and install GdeltForge itself
uv sync

# Activate the virtual environment
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

# Verify the CLI is installed
gdeltforge --help

cp config/settings.example.yaml config/settings.yaml

uv sync --group dev
uv run pytest

Project Structure

GdeltForge follows a standard src/ package layout, installable via uv sync / pip install.

project_root/
├── config/
│ └── settings.example.yaml # Annotated starting point; copy to settings.yaml and edit
│
├── src/gdeltforge/
│ ├── py.typed # PEP 561 marker: this package ships inline type hints
│ ├── cli.py # Argument parsing + subcommand dispatch (the gdeltforge entry point)
│ ├── _version.py # Generated at build time from the git tag (hatch-vcs); not hand-edited
│ │
│ ├── config/
│ │ └── default_settings.yaml # Bundled fallback config, used when no settings.yaml/--config is found at all
│ │
│ ├── data/
│ │ └── cameo_codes.json # Bundled CAMEO/FIPS code -> name reference data, backs the `codes` command
│ │
│ ├── conversion/
│ │ └── converter.py # CSV -> Parquet conversion logic
│ │
│ ├── crossref/
│ │ └── crossref.py # Events<->GKG join (direct for GKG 1.0, two-hop via Mentions for GKG 2.1)
│ │
│ ├── filtering/
│ │ └── filter.py # Filtering logic (drop invalid rows)
│ │
│ ├── sampling/
│ │ ├── cameo_codes.py # Loads data/cameo_codes.json, groups columns by code family, backs the `codes` command
│ │ ├── indexer.py # File indexing for reproducible sampling
│ │ ├── rng.py # Random number generation helpers
│ │ └── samplers.py # Indexed, daily, and filtered sampling
│ │
│ ├── scraping/
│ │ └── scraper.py # Downloader for Events, GKG 2.1, GKG 1.0, and Mentions
│ │
│ └── utils/
│   ├── branding.py # Terminal ANSI colors and the --version banner (brand system's terminal voice)
│   ├── config.py # Config resolution (--config / env var / CWD / bundled default) and YAML loading
│   ├── io.py # File and chunked-IO helpers
│   └── logging.py # Central logging system
│
├── docs/assets/brand/ # Emblem, favicon, wordmark lockups, icon set, GitHub social preview
├── tests/ # pytest suite (unit tests, no network/browser required)
│
├── main.py # Backward-compatible shim: `python main.py <command>` still works
├── pyproject.toml # Package metadata, build backend, gdeltforge console-script entry point
└── README.md

Command Line Interface

Run pipeline stages using the installed console script:

gdeltforge <command> [options]

python main.py <command> [options] is kept as a backward-compatible equivalent, so existing scripts keep working unchanged. This alias only exists in a source checkout, not in the PyPI package installed above; gdeltforge itself always works either way.

The CLI intentionally does not chain stages automatically. You run each stage explicitly to maintain full control.

By default the config is read from ./config/settings.yaml (relative to wherever you run the command from). To point at a config file anywhere else on disk, use --config /path/to/settings.yaml or set the GDELTFORGE_CONFIG environment variable, useful once GdeltForge is installed and invoked from outside the repo checkout. If none of those resolve to a real file, GdeltForge falls back to a built-in default instead of failing; see Configuration.

Usage Examples

Below are practical, beginner-friendly examples covering all common workflows.

Scrape Raw Data

Download the entire archive (--dataset is required on every command; see CLI Reference for the other six choices):

gdeltforge scrape --dataset events

Download only files within a date range (any combination of bounds is valid):

gdeltforge scrape --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge scrape --dataset events --start-date 2022-01-01          # from date onward
gdeltforge scrape --dataset events --end-date   2015-12-31          # up to date

The filter applies to all three file types the GDELT archive provides:

Files already present in the download directory are skipped regardless of the date filter.

Downloads run concurrently (scraping.max_workers in config/settings.yaml, default 8) and are checksum-verified: the GDELT index publishes an MD5 per file, which the scraper captures and checks against each download after it completes. A mismatch is treated the same as a network failure: the file is discarded and retried up to scraping.retries times before being reported as failed, so a corrupted or truncated download never silently ends up in the dataset.

Convert CSV to Parquet

gdeltforge convert --dataset events

Extracts all CSV files from the downloaded ZIP archives and converts them to Parquet files. Each ZIP is processed independently, so conversion runs across a pool of worker processes (converter.max_workers in config/settings.yaml; null, the default, uses all available CPU cores).

paths:
  # existing paths ...
  parquet_historical_directory: "./data/events/historical"
  filtered_historical_directory: "./data/events/filtered_historical"

converter:
  partitioning:
    enabled: true
    rules:
      - file_type: yearly    # e.g. 1979.zip
        by: ["Year"]
      - file_type: monthly   # e.g. 200601.zip
        by: ["Year", "MonthYear"]

data/events/
├── parquet/                  # daily files (2013-present), unchanged
│   ├── 20130401.export.parquet
│   └── ...
└── historical/               # yearly/monthly files, Hive-organized
    ├── Year=1979/
    │   └── 1979.parquet
    ├── Year=2006/
    │   └── MonthYear=200601/
    │       └── 200601.parquet
    └── ...

convert also accepts --start-date/--end-date, same as scrape, narrowing which already-downloaded ZIPs get converted:

gdeltforge convert --dataset events --start-date 2020-01-01 --end-date 2020-12-31

Filter the Parquet Dataset

gdeltforge filter --dataset events

Drops rows with missing values in the columns defined in settings.yaml. Also accepts --start-date/--end-date, narrowing which already-converted files get read (which files get filtered, not which rows survive within them).

Sampling

All sampling modes read from the filtered directory by default. Pass --source converted to sample from raw converted Parquet instead, skipping the filter stage entirely.

Note: sampling is already as memory-friendly as the underlying data volume allows, but --source converted will still use noticeably more RAM than the default, since it isn't working from data that's already had its missing-value rows dropped.

Indexed Sampling (Uniform Random)

gdeltforge sample --dataset events --mode indexed -n 10000 --seed 123 --out sample.parquet

This samples 10000 instances considering the entire data.

Calendar Sampling (N Rows Per Period)

gdeltforge sample --dataset events --mode calendar --per-period 20 --out daily.parquet

This samples 20 instances per day from the entire period (1979-present).

Filtered Sampling (Using JSON Filters)

5000 events whose QuadClass is in {1,2}:

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"QuadClass": [1, 2]}' \
    -n 5000 \
    --out qc12.parquet

2000 "Verbal Cooperation" events that happened in the USA (ActionGeo_CountryCode is a FIPS 2-letter code, US, not the 3-letter CAMEO code USA that Actor1CountryCode/Actor2CountryCode use):

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"], "QuadClass": [1]}' \
    -n 2000

You can select specific columns of interest, which is a memory friendly practice:

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"], "QuadClass": [1]}' \
    --columns GlobalEventID Year Actor1Code \
    -n 1000

It outputs 1000 instances following the same rule as before, but this time with only three columns.

Stratified Sampling (Fixed N Per Group)

Combines a filter with stratified reservoir sampling: draws exactly --n-per-group rows for each distinct value of a chosen column.

gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"]}' \
    --stratify QuadClass \
    --n-per-group 500 \
    --out stratified.parquet

This produces a balanced dataset with 500 USA events per QuadClass value, regardless of the natural class distribution.

--stratify requires --n-per-group. The -n flag is ignored when --stratify is set.

Cross-Referencing Events with GKG

crossref enriches a sampled Events output with GKG (themes, tone, people, organizations). --gkg-version picks the join strategy: GKG 1.0 carries EventIds directly (a direct join), while GKG 2.1 carries no event ID at all and needs a two-hop join through Mentions on the source article's URL. --gkg-version auto attempts every eligible event against both generations instead of requiring one for the whole sample, useful for a sample spanning both eras (GKG 2.1/Mentions don't exist before 2015-02-18; GKG 1.0 has covered everything since 2013-04-01, and remains live today). All preserve the real many-to-many structure (one event can produce several output rows, one article covering several events contributes one row per event) instead of collapsing it.

gdeltforge scrape --dataset gkg-v2
gdeltforge scrape --dataset mentions
gdeltforge convert --dataset gkg-v2
gdeltforge convert --dataset mentions
gdeltforge filter --dataset gkg-v2
gdeltforge filter --dataset mentions

gdeltforge crossref \
    --events sample.parquet \
    --gkg-version v2 \
    --out sample_with_gkg.parquet

See Recipes for the GKG 1.0 direct-join variant and more detail.

Full Pipeline Examples

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge filter --dataset events
gdeltforge sample --dataset events --mode indexed -n 10000

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge filter --dataset events
gdeltforge sample --dataset events --mode indexed -n 5000 --seed 42

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge filter --dataset events
gdeltforge sample \
    --dataset events \
    --mode filtered \
    --filter '{"ActionGeo_CountryCode": ["US"]}' \
    -n 3000

gdeltforge scrape --dataset events
gdeltforge convert --dataset events
gdeltforge filter --dataset events
gdeltforge sample --dataset events --mode calendar --per-period 30

gdeltforge scrape --dataset events && \
gdeltforge convert --dataset events && \
gdeltforge filter --dataset events && \
gdeltforge sample --dataset events --mode indexed -n 10000

foreach ($c in "scrape", "convert", "filter") {
    gdeltforge $c --dataset events
}
gdeltforge sample --dataset events --mode indexed -n 10000

gdeltforge scrape --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge convert --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge filter --dataset events --start-date 2020-01-01 --end-date 2023-12-31
gdeltforge sample --dataset events --mode indexed -n 10000 --start-date 2020-01-01 --end-date 2023-12-31

Further Examples

The complete filtered-sampling syntax reference and a full set of runnable recipes are in the documentation: see Filtered Sampling and Recipes. For which classical sampling algorithm each mode actually runs, see Sampling Theory.

Logging System

Logging is enabled through a shared logger helper:

from gdeltforge.utils.logging import get_logger
logger = get_logger(__name__)

Logging to a file:

logger = get_logger(__name__, log_to_file=True)

Limitations and Roadmap

GdeltForge is intentionally simple: one pipeline stage per command (no automatic chaining or dependency resolution), CSV -> Parquet only, and scrape/convert/filter now exit non-zero if any individual file fails.

The full, current list of limitations and the roadmap live in one place, the docs site, rather than duplicated here where they'd inevitably drift: see Limitations & Roadmap.

Contributing

Contributions are welcome. See CONTRIBUTING.md for dev setup, the branch/commit conventions this repo follows, and how to propose a change. Please also read the Code of Conduct.

See CHANGELOG.md for a history of notable changes.

Found a security issue? See SECURITY.md rather than opening a public issue.

License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.

Data Attribution

GdeltForge processes data published by the GDELT Project, which makes it available for unlimited and unrestricted use, provided any use or redistribution includes a citation and a link to their site. GdeltForge itself is an independent, unofficial tool, not affiliated with, endorsed by, or sponsored by the GDELT Project.