polars-pycountry

ISO 3166 and ISO 4217 code lookup for Polars, as a native Rust expression plugin.

Turning a country column into codes looks like a dictionary lookup until you meet real data. US, USA, 840, United States, United States of America and 🇺🇸 are all the same country; Côte d'Ivoire and Cote d'Ivoire are not the same string; CA is Canada or California depending on the column next to it. The reference answer in Python is pycountry — a good library, but a Python one. Inside Polars it can only be driven through Expr.map_elements, one interpreter round-trip per row.

This package reads the same iso-codes tables pycountry ships, implements its lookup rules in Rust, and exposes them as ordinary Polars expressions. It is built to produce identical output to pycountry, not merely similar output — see Correctness.

import polars as pl
import polars_pycountry as pc

df = pl.DataFrame({
    "country": ["USA", "united kingdom", "276", "🇫🇷", "Nowhere", None],
})

df.with_columns(
    pc.alpha_2("country").alias("alpha_2"),
    pc.alpha_3("country").alias("alpha_3"),
    pc.name("country").alias("name"),
    pc.flag("country").alias("flag"),
)

┌────────────────┬─────────┬─────────┬────────────────┬──────┐
│ country        ┆ alpha_2 ┆ alpha_3 ┆ name           ┆ flag │
╞════════════════╪═════════╪═════════╪════════════════╪══════╡
│ USA            ┆ US      ┆ USA     ┆ United States  ┆ 🇺🇸   │
│ united kingdom ┆ GB      ┆ GBR     ┆ United Kingdom ┆ 🇬🇧   │
│ 276            ┆ DE      ┆ DEU     ┆ Germany        ┆ 🇩🇪   │
│ 🇫🇷             ┆ FR      ┆ FRA     ┆ France         ┆ 🇫🇷   │
│ Nowhere        ┆ null    ┆ null    ┆ null           ┆ null │
│ null           ┆ null    ┆ null    ┆ null           ┆ null │
└────────────────┴─────────┴─────────┴────────────────┴──────┘

Install

pip install polars-pycountry
# or
uv add polars-pycountry

Prebuilt wheels cover Linux (glibc and musl, x86_64 and aarch64), macOS (Intel and Apple Silicon), and Windows (x64 and arm64). There is one wheel per platform rather than one per Python version, because the extension is built against the stable ABI. An sdist is published too, so anything else builds from source given a Rust toolchain — see CONTRIBUTING.md.

pycountry is not a runtime dependency. The ISO tables are compiled into the extension; pycountry is installed only for the test suite, which asserts the two agree.

Usage

Three families of expressions, all taking a string column.

Countries — ISO 3166-1

The input may be any indexed field: alpha-2, alpha-3, numeric, name, official name, common name, or the flag emoji. Matching is case-insensitive.

Subdivisions — ISO 3166-2

The value may be a full code ("US-CA"), a bare code with a country ("CA"), or a name ("California").

Pass country whenever you have it. Subdivision names are not unique across the standard — "Central" names a region in a dozen countries — so an unscoped name resolves to whichever record ISO lists first.

The argument follows the usual Polars convention: a bare string names a column, so a fixed country needs pl.lit(...). Either form accepts anything the country lookup does — "US", "USA", "840", "United States":

df.with_columns(
    pc.subdivision_code("state", country=pl.col("country")),  # per row
    pc.subdivision_code("state", country=pl.lit("US")),  # fixed
)

A country that does not itself resolve yields null rather than falling back to a global search, so a bad scope cannot quietly return another country's state.

Currencies — ISO 4217

pc.currency, pc.currency_alpha_3, pc.currency_numeric, pc.currency_name. Input may be the code, the number, or the name. Exact only — pycountry gives search_fuzzy to countries and subdivisions, not to currencies, so there is no reference to match a fuzzy currency search against.

Nulls

Every expression returns null when nothing matched, and null for null input. Blank and whitespace-only input is null too — see fuzzy matching for why that is a deliberate divergence from pycountry.

Null is also a real answer, not only a failure: most countries have no official_name and almost none have a common_name, so pl.coalesce(pc.common_name(c), pc.name(c)) is the usual way to get a display name.

Expression namespace

Importing the package registers a .country namespace:

df.with_columns(pl.col("country").country.alpha_2())
df.filter(pl.col("country").country.alpha_3() == "DEU")

Scalars

For code that isn't holding a DataFrame — the same Rust core, no Polars round-trip. Each returns a dict, or None:

pc.lookup_country("USA")
# {'alpha_2': 'US', 'alpha_3': 'USA', ..., 'matched_on': 'alpha_3', 'score': None}

pc.lookup_country("Texas", fuzzy=True)["alpha_2"]  # 'US'
pc.lookup_subdivision("California", country="US")  # {'code': 'US-CA', ...}
pc.lookup_currency("840")  # {'alpha_3': 'USD', ...}
pc.search_countries("guinea", limit=3)  # ranked, with scores

Matching

Exact, by default

The default is pycountry.countries.lookup: case-insensitive, across every field the standard indexes, and nothing else. It is accent-sensitive, because pycountry's indices are — "CÔTE D'IVOIRE" resolves and "Cote d'Ivoire" does not.

Numeric codes are three digits wide and the leading zero is part of the value, so an integer column needs padding before it will match:

pl.col("n").cast(pl.String).str.zfill(3)  # 4 -> "004"

Fuzzy matching

fuzzy=True lets an exact miss fall back to pycountry's search_fuzzy, which folds accents away, matches partial names and initials, and reaches countries through their subdivisions:

pc.alpha_2("country", fuzzy=True)
# "Cote d'Ivoire" -> CI      (accents folded)
# "USA"           -> US      (initials of "United States of America")
# "Texas"         -> US      (via the subdivision)
# "Bavaria"       -> null    (the standard says "Bayern"; this is not a translation layer)

It is off by default because it is a guess, and a wrong guess is harder to notice than a null. "Republic of Korea" resolves to KP, not KR — no ISO field spells the name that way, so the subdivision pass decides it. pycountry answers KP too; the point is that fuzzy answers deserve review, which is what pc.match is for:

df.with_columns(pc.match("country", fuzzy=True).alias("m")).unnest("m")
# matched_on: "alpha_3", "name", "numeric", ... or "fuzzy"
# score:      null for an exact match; the fuzzy points otherwise

Filtering on pl.col("score").is_null() separates the answers you can trust from the ones worth eyeballing.

One deliberate divergence: pycountry.countries.search_fuzzy("") matches every country — the empty string is a substring of every name, at offset 0 — and ranks Andorra first. In a DataFrame that turns every blank cell into a confident wrong answer, so blank input is null here.

Historic countries — ISO 3166-3

historic=True consults the withdrawn-country table when ISO 3166-1 has no match. Current countries always win, so a reused code like AI still resolves to Anguilla rather than to French Afars and Issas:

pc.extract("country", historic=True)
# "YUCS" -> alpha_2 YU, withdrawal_date 2003-07-23, historic true

alpha_4 and withdrawal_date are ISO 3166-3 fields, so they are populated only for a historic match. They are in the struct either way — a keyword never changes its shape.

Performance

Measured with just bench:

_{AMD Ryzen 9 3950X (16 cores / 32 threads), 31 GB RAM, Linux 6.18 (WSL2), Python 3.12.13, Polars 1.44.0. Median of six runs; the parallel=True exact figure varies by about ±13% between runs, because at 200,000 rows it completes in ~3 ms.}

The two row counts differ on purpose. pycountry.countries.lookup is a dict probe, so 200,000 rows of it take a fraction of a second; search_fuzzy scans every country and every subdivision, in Python, with no cache, at ~8 ms per call — 200,000 rows of that would take about half an hour. Both sides are measured on the same count, so each ratio still compares like with like.

What the fuzzy number depends on

Fuzzy matching is memoized: each call resolves the distinct values in the column, not the rows. Real columns repeat themselves heavily, so the cost usually lands closer to the cardinality than to the row count — but it is worth knowing which regime you are in:

_{Not part of just bench: the low-cardinality row draws from the same 14-value vocabulary the fuzzy benchmark uses, the high-cardinality row from every distinct ISO 3166-2 subdivision name. Median of five runs.}

Above 100,000 rows the distinct values are resolved once up front, in parallel, and the rows are filled from the result. Without that each thread would build its own cache and re-run the same searches, which made parallel=True slower than parallel=False on a high-cardinality column.

Measure a release build. just bench builds one; a plain maturin develop is unoptimized and far slower on this workload, which measures the profile rather than the code.

Columns of 100k rows or more are split across rayon threads; pass parallel=False to force single-threaded. The threshold sits above the streaming engine's morsel size, so when Polars is already calling the plugin from several of its own worker threads each call stays single-threaded rather than nesting a fan-out inside it.

A caveat worth stating plainly: if your column has few distinct values, a dict built over Series.unique() plus replace_strict can still beat any per-row approach, including this one. This package wins on high-cardinality columns, on the fuzzy path, and on code you would rather not write.

Correctness

The point of this package is not "fast country lookup" — it is "fast country lookup you can swap in without your results moving". tests/test_parity.py asserts agreement with pycountry over four corpora:

1. Every indexed value in ISO 3166-1 — each alpha-2, alpha-3, numeric code, name, official name, common name and flag in the standard, in original, upper and lower case. If the two agree on every string the table contains, they agree on every exact lookup that can succeed.
2. Every ISO 3166-2 code — all ~5,000 of them, checked against pycountry.subdivisions.get.
3. Every ISO 4217 value — codes, numbers and names.
4. A fuzzy corpus — every country name and subdivision name, plus hand-written messy input, compared against pycountry.countries.search_fuzzy result by result, including the ranking.

Both sides are pointed at the same table files, so a disagreement can only be an algorithm difference — never two different snapshots. tests/test_data.py separately asserts the vendored tables are the ones the installed pycountry ships.

The known, deliberate divergences are blank input (null rather than "every country"), and no fuzzy search over currencies or the historic table. Everything else agreeing is the contract; if you find an input where this package and pycountry disagree, that is a bug here. Please open an issue with the input.

The ISO tables

A snapshot of the iso-codes tables is compiled into the binary, so there is no network access, no cache directory, and no first-call latency spike. pc.iso_version() reports the upstream release in use, and pc.table_sizes() how many records each table holds.

To supply your own tables at startup, point POLARS_COUNTRY_DATA at a directory holding iso3166-1.json, iso3166-2.json, iso3166-3.json and iso4217.json:

export POLARS_COUNTRY_DATA=/path/to/iso-codes

It is read once, on first use, so set it before the first lookup.

Refreshing without a restart

The standard changes — countries are added, subdivisions renamed, codes withdrawn. A long-lived process would otherwise be stuck with whatever tables it read when it resolved its first country, so two functions replace them in place:

# Download the current tables and load them into this process.
pc.refresh_iso_data()

# ...and keep a copy, so the next run need not go back to the network.
pc.refresh_iso_data(save_to="iso-codes/")

# Or load ones you already have.
pc.load_iso_data("iso-codes/")

Both take effect for every lookup that starts after they return. A query already in flight keeps the tables it began with, so no single column is ever resolved against two different snapshots.

refresh_iso_data is the only function here that touches the network, and only when you call it — importing the package still does nothing. Point it at an internal mirror with pc.refresh_iso_data(url=...) if outbound access is restricted.

A missing file, an unparseable table, or one too small to be a complete copy raises ValueError and leaves the working tables untouched. That last check earns its keep: a truncated download or an HTML error page is still valid something, and without a floor on the record count the only symptom would be countries quietly going missing.

Compatibility

If a future Polars release bumps the plugin ABI, this package fails loudly at load rather than miscomputing.

Contributing

Contributions are welcome — see CONTRIBUTING.md for the development loop, the parity requirement, and how to refresh the ISO tables. docs/architecture/overview.md explains how this implementation maps onto pycountry's, which is worth reading before changing the matching rules.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

The vendored ISO tables are not covered by that dual license. They come from the iso-codes project and are licensed LGPL-2.1-or-later, which has obligations that attach to any binary embedding them. See NOTICE before redistributing.