owid-catalog

A Pythonic library for working with OWID data.

The owid-catalog library is the foundation of Our World in Data's data management system. It provides:

1. Data APIs: Access OWID's published data through unified client interfaces
2. Data Structures: Enhanced pandas DataFrames with rich metadata support

Installation

pip install owid-catalog

Quick Examples

Accessing OWID Data

from owid.catalog import fetch, search

# Search for charts (default)
charts = search("population")
tb = charts[0].fetch()

# Fetch data from OWID Chart at ourworldindata.org/grapher/life-expectancy
tb = fetch("life-expectancy")

# Search for tables
tables = search("population", kind="table", namespace="un")
tb = tables[0].fetch()

# Search indicators (using semantic search)
search("renewable energy", kind="indicator")

Working with Data Structures

from owid.catalog import Table
from owid.catalog import processing as pr

# Tables are pandas DataFrames with metadata
tb = Table(df, metadata={"short_name": "population"})

# Metadata propagates through operations
tb_filtered = tb[tb["year"] > 2000]  # Keeps metadata
tb_merged = pr.merge(tb1, tb2, on="country")  # Merges metadata

Documentation

For detailed documentation, see:

• API Reference: ChartsAPI, IndicatorsAPI, TablesAPI
• Data Structures: Dataset, Table, Variable, metadata handling
• Full Documentation: Complete library documentation
• Agent skill: guide for AI agents using the library for data analysis

Architecture

graph TB
etl -->|reads| snapshot[upstream datasets]
etl -->|generates| s3[data catalog]
catalog[owid-catalog] -->|queries| s3

This library is part of OWID's ETL project, which contains recipes for all datasets we publish.

Development

You need Python 3.11+, uv and make installed. Clone the repo, then you can simply run:

# run all unit tests and CI checks
make test

# watch for changes, then run all checks
make watch

Maintainer notes — how the version is bumped, how a release reaches PyPI, and which checks actually cover this directory — are in DEVELOPMENT.md.

Changelog

v1.3.0

• Dataset.read() returns strings as pandas' default str dtype (pyarrow-backed, NaN for missing values) instead of string[pyarrow] (pd.NA for missing values), through owid-repack>=0.2.0. Strings read from a dataset now behave like strings pandas 3 creates: merging on them no longer turns the key into object, and comparisons give False rather than <NA> for missing values. Check for missing values with pd.isna(), not is pd.NA. ds[...] and read(safe_types=False) are unchanged
• description_key items that YAML parses as mappings (a plain item containing : ) raise a ValueError that names the indicator and says to quote the item, instead of a bare assertion
• Tests that call live services (the catalog, ourworldindata.org, Datasette) are marked integration: make unittest skips them and make test-integration runs them

v1.2.9

• ResponseSet(_ui_advanced=True) keeps the flag: Pydantic dropped constructor arguments for private attributes, so search results never got the advanced display
• Dev tooling: ruff 0.16.10, ty 0.0.85

v1.2.8

• pandas 3 support: pandas>=2.2.3,<4 (was <3), and requires owid-datautils>=0.6.12 and owid-repack>=0.1.12, which allow pandas 3 too
• Behavior kept from pandas 2 on pandas 3
  • Indicator.replace and Table.replace on a categorical replace its categories, as pandas 2 did: a new value renames the category, an existing one merges into it, and NaN removes it. pandas 3 raises on a value that isn't already a category
  • Table.groupby().apply() passes the grouping columns to the function, since pandas 3 removed include_groups=True
  • Table.set_index(verify_integrity=True) checks uniqueness itself, since pandas 3 deprecates the argument; Table.format relies on it
  • Indicator.fillna() without a value raises, as in pandas 2; pandas 3 silently fills nothing
• Fixes for the pandas str dtype: search results map missing title/description to None, display truncation also covers str columns, and read_csv with categorical dtypes parses in one pass

v1.2.7

• Dependency security updates (tornado 6.5.9, urllib3 2.8.0)

v1.2.6

• Documentation rendered from metadata (new core.docs module)
  • Dataset.readme() renders a Markdown README for a dataset: its description, one block per table (citation, indicators, sources), its changelog, the processing note, license and download links
  • New Table.sources table of the distinct origins behind a table's columns; Table.codebook is rebuilt on the same helpers, and its source column uses the same labels
• Dataset changelog: new DatasetMeta.changelog, a list of ChangelogEntry(date, changes), set from dataset.changelog in .meta.yml. Dates are stored as YYYY-MM-DD strings, and a malformed entry raises where it is set. The README renders it as its own "Changelog" section, and it never enters the description
• Schema.org JSON-LD
  • Descriptions over Google Dataset Search's 5,000-character limit raise instead of producing a record Google rejects
  • Each table's nested Dataset repeats the creator, since Google validates nested Datasets on their own
  • dataset_keywords() is public (was _keywords)
• Docstrings say "data producer" instead of "data provider"

v1.2.5

• Remove the leftover multidim channel from CHANNEL in core.datasets and core.paths. MDIM configs were written as datasets under data/multidim/ for a short while in 2025; they have lived outside the data catalog since, so the channel only made the publish job look for an empty folder and write an empty index file on every deploy
• Render metadata Jinja in a SandboxedEnvironment, so a producer-supplied value containing << or <% cannot walk __class__/__subclasses__ on whatever runs the ETL
• Fail loudly on a character-exploded description_key (a markdown string that was iterated character by character): Markdown is now a str subclass that raises TypeError on iteration, alongside a new validate_description_key_list() sanity check
• New s3_utils.object_exists() to check for an object without downloading it

v1.2.4

• Python support
  • Add Python 3.14, drop Python 3.10 (requires-python = ">=3.11, <3.15")
  • Drop the typing_extensions fallbacks for Self, Required and NotRequired
• description_key becomes free-form markdown
  • VariableMeta.description_key is now a markdown string
  • A list of bullet points is still accepted (items may carry per-item Jinja) and is converted to a markdown list after rendering — the grapher only ever sees a string
  • New description_key_to_string() in owid.catalog.core.meta, reproducing how the grapher rendered those lists before
• Display metadata
  • Add display.timeInterval (day, week, month, quarter, year, decade)
  • Remove the deprecated display.yearIsDay

v1.2.3

• combine_indicators_processing_level tolerates unrendered Jinja templates in processing_level instead of asserting on an unknown level — when a template is combined with a literal it overstates rather than understates the result, since processing_level feeds licensing downstream
• s3_utils.upload() accepts content_type and cache_control
• Catalog JSON-LD (schema_org): license and keyword fixes, citation dropped
• Dependency bumps for security advisories

v1.2.2

• Add owners to DatasetMeta
• New Dataset.update_metadata_from_dict() and yaml_metadata.update_metadata_from_dict(), for callers that already hold parsed metadata rather than a YAML path (used by the Owl runner)
• New schema_org module emitting schema.org JSON-LD for catalog datasets and tables, with follow-ups: stable short landing-page URLs, table descriptions filled from existing metadata, an explicit dataset description requirement, and unrendered Jinja templates kept out of the output
• YAML variable checks accept a long-format base name as a match for pivoted {base}__{dim}_{value} columns, so a long_to_wide override block is no longer flagged as a typo
• Republished to PyPI after the ty upgrade, with no functional change of its own

v1.2.1

• Send User-Agent: owid-catalog/<version> (python <x.y.z>) on every outbound HTTP call, via a shared requests.Session in owid.catalog.api.utils (plus STORAGE_OPTIONS for the pandas reads that cannot take a session)
• prune_dict() keeps explicitly-empty values for keys listed in KEEP_IF_EMPTY: chartTypes: [] is a meaningful "render no chart-type toggles" override, not the same as an absent key

v1.2.0

• Remove legacy Source metadata (origins only)
  • Removed Source class from owid.catalog.core.meta
  • Removed sources field from VariableMeta and DatasetMeta (use origins instead)
  • Removed if_source_exists parameter from Dataset.update_metadata (use if_origins_exist)
  • Removed get_unique_sources_from_indicators helper
  • Removed sources aggregation from combine_indicators_metadata

v1.1.0

• Remove processing log feature
  • Removed ProcessingLog and processing_log module from owid.catalog.core
  • Removed combine_indicators_processing_logs helper
  • Removed update_log / amend_log methods on Indicator
  • Removed processing-log tracking from Table arithmetic operations (__add__, __sub__, __mul__, etc.)
  • Removed processing_log field from VariableMeta

v1.0.1

• ResponseSet ergonomics
  • Remove deprecated ResponseSet.results property (use .items instead)
  • Add .to_dict() method for serializing results to plain dicts (useful for AI/LLM context windows)
  • Add all_fields parameter to .to_frame() to temporarily override display mode without mutating instance state

v1.0.0

• New unified Client API
  • owid.catalog.Client as single entry point with ChartsAPI, IndicatorsAPI, TablesAPI
  • Quick access via search() and fetch() convenience functions
  • Rich result types: ChartResult, IndicatorResult, TableResult with ResponseSet container
• Charts API
  • Fetch chart data by slug, URL, or slug with query params
  • Parse chart slugs from grapher/explorer URLs via parse_chart_slug()
  • Explorer best-effort fetching with graceful error handling
  • set_ui_advanced() / set_ui_basic() for display configuration
• Tables API
  • Search catalog by table, namespace, version, dataset, and channel
  • Fetch tables directly by catalog path
  • Embedded catalog index with local caching
• Indicators API
  • Semantic search via search.owid.io vector embeddings
  • Sort by relevance (similarity + popularity blend) or similarity only
  • fetch() for single-column indicator or fetch_table() for the full table
• Search & discovery
  • Fuzzy, exact, contains, and regex matching modes
  • .latest() filtering to keep only newest versions
  • Popularity scores (0.0-1.0) from analytics views, results sorted by popularity
  • refresh_index parameter to force catalog index reload
• Data structures integration
  • All fetch() methods return owid.catalog.Table with full metadata
  • CatalogPath helper for parsing catalog paths
  • Lazy loading with load_data=False for deferred data access
• Library reorganization
  • Restructured into owid.catalog.core (data structures) and owid.catalog.api (remote access)
  • catalog.find() deprecated in favor of Client().tables.search() (backwards compat maintained)
  • Legacy code moved to owid.catalog.api.legacy
  • New dependencies: pydantic v2.0+
• Private data support
  • Private datasets served from separate R2 bucket
  • API can fetch private data from private bucket
• Performance
  • Vectorized operations replacing iterrows() in TablesAPI
  • Embedded catalog index loading (removed ETLCatalog dependency)
  • Modularized search into helper methods
• Other
  • Thumbnail display in ResponseSet for chart results
  • JSON output format support
  • Comprehensive exception handling: ChartNotFoundError, LicenseError
  • API URLs immutable with Pydantic Field(frozen=True)