owid-catalog

A Pythonic library for working with OWID data.

The owid-catalog library is the foundation of Our World in Data's data management system. It provides:

1. Data APIs: Access OWID's published data through unified client interfaces
2. Data Structures: Enhanced pandas DataFrames with rich metadata support

Installation

pip install owid-catalog

Quick Examples

Accessing OWID Data

from owid.catalog import fetch, search

# Search for charts (default)
charts = search("population")
tb = charts[0].fetch()

# Fetch data from OWID Chart at ourworldindata.org/grapher/life-expectancy
tb = fetch("life-expectancy")

# Search for tables
tables = search("population", kind="table", namespace="un")
tb = tables[0].fetch()

# Search indicators (using semantic search)
search("renewable energy", kind="indicator")

Working with Data Structures

from owid.catalog import Table
from owid.catalog import processing as pr

# Tables are pandas DataFrames with metadata
tb = Table(df, metadata={"short_name": "population"})

# Metadata propagates through operations
tb_filtered = tb[tb["year"] > 2000]  # Keeps metadata
tb_merged = pr.merge(tb1, tb2, on="country")  # Merges metadata

Documentation

For detailed documentation, see:

• API Reference: ChartsAPI, IndicatorsAPI, TablesAPI
• Data Structures: Dataset, Table, Variable, metadata handling
• Full Documentation: Complete library documentation

Architecture

graph TB
etl -->|reads| snapshot[upstream datasets]
etl -->|generates| s3[data catalog]
catalog[owid-catalog] -->|queries| s3

This library is part of OWID's ETL project, which contains recipes for all datasets we publish.

Development

You need Python 3.10+, uv and make installed. Clone the repo, then you can simply run:

# run all unit tests and CI checks
make test

# watch for changes, then run all checks
make watch

Changelog

v1.2.0

• Remove legacy Source metadata (origins only)
  • Removed Source class from owid.catalog.core.meta
  • Removed sources field from VariableMeta and DatasetMeta (use origins instead)
  • Removed if_source_exists parameter from Dataset.update_metadata (use if_origins_exist)
  • Removed get_unique_sources_from_indicators helper
  • Removed sources aggregation from combine_indicators_metadata

v1.1.0

• Remove processing log feature
  • Removed ProcessingLog and processing_log module from owid.catalog.core
  • Removed combine_indicators_processing_logs helper
  • Removed update_log / amend_log methods on Indicator
  • Removed processing-log tracking from Table arithmetic operations (__add__, __sub__, __mul__, etc.)
  • Removed processing_log field from VariableMeta

v1.0.1

• ResponseSet ergonomics
  • Remove deprecated ResponseSet.results property (use .items instead)
  • Add .to_dict() method for serializing results to plain dicts (useful for AI/LLM context windows)
  • Add all_fields parameter to .to_frame() to temporarily override display mode without mutating instance state

v1.0.0

• New unified Client API
  • owid.catalog.Client as single entry point with ChartsAPI, IndicatorsAPI, TablesAPI
  • Quick access via search() and fetch() convenience functions
  • Rich result types: ChartResult, IndicatorResult, TableResult with ResponseSet container
• Charts API
  • Fetch chart data by slug, URL, or slug with query params
  • Parse chart slugs from grapher/explorer URLs via parse_chart_slug()
  • Explorer best-effort fetching with graceful error handling
  • set_ui_advanced() / set_ui_basic() for display configuration
• Tables API
  • Search catalog by table, namespace, version, dataset, and channel
  • Fetch tables directly by catalog path
  • Embedded catalog index with local caching
• Indicators API
  • Semantic search via search.owid.io vector embeddings
  • Sort by relevance (similarity + popularity blend) or similarity only
  • fetch() for single-column indicator or fetch_table() for the full table
• Search & discovery
  • Fuzzy, exact, contains, and regex matching modes
  • .latest() filtering to keep only newest versions
  • Popularity scores (0.0-1.0) from analytics views, results sorted by popularity
  • refresh_index parameter to force catalog index reload
• Data structures integration
  • All fetch() methods return owid.catalog.Table with full metadata
  • CatalogPath helper for parsing catalog paths
  • Lazy loading with load_data=False for deferred data access
• Library reorganization
  • Restructured into owid.catalog.core (data structures) and owid.catalog.api (remote access)
  • catalog.find() deprecated in favor of Client().tables.search() (backwards compat maintained)
  • Legacy code moved to owid.catalog.api.legacy
  • New dependencies: pydantic v2.0+
• Private data support
  • Private datasets served from separate R2 bucket
  • API can fetch private data from private bucket
• Performance
  • Vectorized operations replacing iterrows() in TablesAPI
  • Embedded catalog index loading (removed ETLCatalog dependency)
  • Modularized search into helper methods
• Other
  • Thumbnail display in ResponseSet for chart results
  • JSON output format support
  • Comprehensive exception handling: ChartNotFoundError, LicenseError
  • API URLs immutable with Pydantic Field(frozen=True)