Pandas Maxmind

Provides fast and convenient geolocation bindings for Pandas Dataframes. Uses numpy ndarray's internally to speed it up compared to naively applying function per column. Based on the maxminddb-rust.

Features

• Supports both MMAP and in-memory implementations
• Supports parallelism (useful for very big datasets)
• Comes with pre-built wheels, no need to install and maintain external C-library to get (better than) C-performance

Installation

1. Minimal supported Python is 3.8
2. pip install pandas_maxminddb
3. The preferred way is to use precompiled binary wheel, as this requires no toolchain and is fastest.
4. If you want to build from source any platform Rust has target for is supported.

Pre-built wheels

The wheels are built against following numpy and pandas distributions:

• If you're on Windows / macOS / Linux there is no need to do anything extra.
• If you use ARMv7 (RaspberryPi and such) use PiWheels --extra-index-url=https://www.piwheels.org/simple, install libatlas-base-dev for numpy.
• If you use musl-based distro like Alpine use Alpine-wheels --extra-index-url https://alpine-wheels.github.io/index , install libstdc++ for pandas.

Refer to the build workflow for details.

Usage

By importing pandas_maxminddb you add Pandas geo extension which allows you to add columns in-place. This example uses context manager for reader lifetime:

import pandas as pd
from pandas_maxminddb import open_database

ips = pd.DataFrame(data={
    'ip': ["75.63.106.74", "132.206.246.203", "94.226.237.31", "128.119.189.49", "2.30.253.245"]})
with open_database('./GeoLite.mmdb/GeoLite2-City.mmdb') as reader:
    ips.geo.geolocate('ip', reader, ['country', 'city', 'state', 'postcode'])
ips

Without context manager

You can also instantiate reader yourself, eg:

import pandas as pd
from pandas_maxminddb import ReaderMem, ReaderMmap

reader = ReaderMem('./GeoLite.mmdb/GeoLite2-City.mmdb')
ips = pd.DataFrame(data={
    'ip': ["75.63.106.74", "132.206.246.203", "94.226.237.31", "128.119.189.49", "2.30.253.245"]})
ips.geo.geolocate('ip', reader, ['country', 'city', 'state', 'postcode'])
ips

Parallelism

If dataset is big enough, and you have extra cores you might benefit from using them. Currently only ReaderMem is supported:

import pandas as pd
from pandas_maxminddb import ReaderMem

reader = ReaderMem('./GeoLite.mmdb/GeoLite2-City.mmdb')
ips = pd.DataFrame(data={
    'ip': ["75.63.106.74", "132.206.246.203", "94.226.237.31", "128.119.189.49", "2.30.253.245"]})
ips.geo.geolocate('ip', reader, ['country', 'city', 'state', 'postcode'], parallel=True)
ips

Benchmarks

• Tested on M1 Max with 1024 chunk size on 100k dataset, refer to benchmark

Extending

Due to Dataframe columns being flat arrays and geolocation data coming in a hierarchical format you might need to provide more mappings to serve your particular use-case. In order to do that follow Development section to setup your environment and then:

1. Add column name to the geo_column.rs
2. Add column mapping to the geolocate.rs

Development

Setting up environment

• git clone --recurse-submodules git@github.com:andrusha/pandas-maxminddb.git
• PYTHON_CONFIGURE_OPTS="--enable-shared" asdf install
• PYTHON_CONFIGURE_OPTS="--enable-shared" python -m venv .venv
• source .venv/bin/activate
• pip install nox
• nox -s test
• PYTHONPATH=.venv/lib/python3.8/site-packages cargo test --no-default-features

libmaxminddb

In order to run nox -s bench properly you would need libmaxminddb installed as per maxminddb instructions prior to installing Python package, so that C-extension could be benchmarked properly.

On macOS this would require following:

• brew instal libmaxminddb
• PATH="/opt/homebrew/Cellar/libmaxminddb/1.7.1/bin:$PATH" LDFLAGS="-L/opt/homebrew/Cellar/libmaxminddb/1.7.1/lib" CPPFLAGS="-I/opt/homebrew/Cellar/libmaxminddb/1.7.1/include" pip install maxminddb --force-reinstall --verbose --no-cache-dir