numpy maxminddb python bindings
Project description
Pandas Maxmind
Provides fast and convenient geolocation bindings for Pandas Dataframes. Uses numpy ndarray's internally to speed it up compared to naively applying function per column. Based on the maxminddb-rust.
Installation
- Minimal supported Python is 3.8
pip install pandas_maxminddb
- The preferred way is to use precompiled binary wheel, as this requires no toolchain and is fastest.
- If you want to build from source any platform Rust has target for is supported.
Pre-build wheels
The wheels are built against following numpy
and pandas
distributions:
- If you're on Windows / macOS / Linux there is no need to do anything extra.
- If you use ARMv7 (RaspberryPi and such)
use PiWheels
--extra-index-url=https://www.piwheels.org/simple
, installlibatlas-base-dev
for numpy. - If you use musl-based distro like Alpine
use Alpine-wheels
--extra-index-url https://alpine-wheels.github.io/index
, installlibstdc++
for pandas.
Refer to the build workflow for details.
Py | win x86 | win x64 | macOS x86_64 | macOS AArch64 | linux x86_64 | linux i686 | linux AArch64 | linux ARMv7 | musl linux x86_64 |
---|---|---|---|---|---|---|---|---|---|
3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚫 | ✅ |
3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚫 |
3.10 | 🚫 | ✅ | ✅ | ✅ | ✅ | ✅ | 🚫 | 🚫 | ✅ |
Usage
By importing pandas_maxminddb
you add Pandas geo
extension which allows you to add columns
in-place
import pandas as pd
from pandas_maxminddb import open_database
ips = pd.DataFrame(data={
'ip': ["75.63.106.74", "132.206.246.203", "94.226.237.31", "128.119.189.49", "2.30.253.245"]})
with open_database('./GeoLite.mmdb/GeoLite2-City.mmdb') as reader:
ips.geo.geolocate('ip', reader, ['country', 'city', 'state', 'postcode'])
ips
ip | city | postcode | state | country | |
---|---|---|---|---|---|
0 | 75.63.106.74 | Houston | 77070 | TX | US |
1 | 132.206.246.203 | Montreal | H3A | QC | CA |
2 | 94.226.237.31 | Kapellen | 2950 | VLG | BE |
3 | 128.119.189.49 | Northampton | 01060 | MA | US |
4 | 2.30.253.245 | London | SW15 | ENG | GB |
Benchmarks
Name (time in ms) | Min | Max | Mean | StdDev | Median | IQR | Outliers | OPS | Rounds | Iterations |
---|---|---|---|---|---|---|---|---|---|---|
test_benchmark_pandas_maxminddb | 273.2588 (1.0) | 284.8850 (1.0) | 280.4760 (1.0) | 4.5448 (1.0) | 281.6831 (1.0) | 5.9721 (1.0) | 1;0 | 3.5654 (1.0) | 5 | 1 |
test_benchmark_c_maxminddb | 986.0314 (3.61) | 1,002.4413 (3.52) | 995.7461 (3.55) | 8.3891 (1.85) | 1,001.3420 (3.55) | 15.1085 (2.53) | 2;0 | 1.0043 (0.28) | 5 | 1 |
test_benchmark_python_maxminddb | 9,011.4650 (32.98) | 9,286.9635 (32.60) | 9,081.2087 (32.38) | 117.9029 (25.94) | 9,020.5363 (32.02) | 114.9376 (19.25) | 1;0 | 0.1101 (0.03) | 5 | 1 |
Extending
Due to Dataframe columns being flat arrays and geolocation data coming in a hierarchical format you might need to provide more mappings to serve your particular use-case. In order to do that follow Development section to setup your environment and then:
- Add column name to the geo_column.rs
- Add column mapping to the geolocate.rs
Development
Setting up environment
git clone --recurse-submodules git@github.com:andrusha/pandas-maxminddb.git
PYTHON_CONFIGURE_OPTS="--enable-shared" asdf install
PYTHON_CONFIGURE_OPTS="--enable-shared" python -m venv .venv
source .venv/bin/activate
pip install nox
nox -s test
PYTHONPATH=.venv/lib/python3.8/site-packages cargo test --no-default-features
libmaxminddb
In order to run nox -s bench
properly you would
need libmaxminddb installed as
per maxminddb instructions prior to
installing Python package, so that C-extension could be benchmarked properly.
On macOS this would require following:
brew instal libmaxminddb
PATH="/opt/homebrew/Cellar/libmaxminddb/1.7.1/bin:$PATH" LDFLAGS="-L/opt/homebrew/Cellar/libmaxminddb/1.7.1/lib" CPPFLAGS="-I/opt/homebrew/Cellar/libmaxminddb/1.7.1/include" pip install maxminddb --force-reinstall --verbose --no-cache-dir
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pandas_maxminddb-0.1.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 005b14c128504bc52d2185084ae94fc392bc84658a63ac010ce35621773811bf |
|
MD5 | 5860861196d3e696d6796b4a844c0bc6 |
|
BLAKE2b-256 | 208cb7c04b5291f74ace3accc33789386e7247f2e8f23aa904033dfceb4a28d0 |
Hashes for pandas_maxminddb-0.1.1-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3fd7c3511ee5aba389341c343c8f4c1462f0a3c236e28b7c99fbf9b10326630 |
|
MD5 | 571eccff0f540dd166d59305213929f7 |
|
BLAKE2b-256 | f5f6eb9fa57f6481ba5a260144739f01de4c8cb5d49e03ae0348a44035b20a2b |
Hashes for pandas_maxminddb-0.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9d31e0eb048bee35ba9c44f23be41cb7a7c858960974aa31cfd39f24e84ff2c |
|
MD5 | 3779b6b2b838aa05ff46ed4dc2d15d67 |
|
BLAKE2b-256 | 885d1109e7ff9807cb673fb87906cff9f39831240545a1eeff0c857dd1401ac8 |
Hashes for pandas_maxminddb-0.1.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59301e0b8326f44b9b684a97b6f45c15b25508eda03f3331292e28fcc15ddf27 |
|
MD5 | 3c7195c3723d9638792071da5b1939a0 |
|
BLAKE2b-256 | f030281287337b23f49dfd1a760d29994adc8d883b39ee4891e43938ffaf80df |
Hashes for pandas_maxminddb-0.1.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeb75c4c90aeaa6ca0bfec109bf97feb1a8d599ddd3ea1c03c77e2f305d0e18a |
|
MD5 | a5d3fcd94692c117bb023d33a2aa72a8 |
|
BLAKE2b-256 | d9b84f9bd5eab8ec61cc8cec2c531bda6bac4a526f11f6fa5c632644368ae9fb |
Hashes for pandas_maxminddb-0.1.1-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12f6bb0436c0800e70555be9d3b4ed09b407cbd8b1f607711a2f8b4e8f669a99 |
|
MD5 | efc4e0985be156d065a6f0cd83b84c0f |
|
BLAKE2b-256 | 509d0c881634ddbd440ce5a717a70a04b6df6470f79ac6e7dd38df1039556677 |
Hashes for pandas_maxminddb-0.1.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1257038b314981d8842e346015521535b4787aeabed98fc444def63ad58914fc |
|
MD5 | 6338a737fa45abd6dbae2247137b426e |
|
BLAKE2b-256 | 5dd339f967dac2b53c4e796dbb162dab6395b3a0903254fe0ed8e57a0382b745 |
Hashes for pandas_maxminddb-0.1.1-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ff4b5f51fc56e467f2f2ea5f0bea3d4dc8b842602e0ff5c8272972ca12dcb69 |
|
MD5 | df2fe4a7e5957b1e4b7a92c5ff8454e7 |
|
BLAKE2b-256 | a34dda9052c5e6900e57cc222ff7ebad4156197995f74466ed4b1d87e3cc7b76 |
Hashes for pandas_maxminddb-0.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 139976f3b7eaefab5c01e014da4865e2c278b24e83dd439aa69859713bd5adf0 |
|
MD5 | 747dc9f21306f1c3b54b03feb5b34e5b |
|
BLAKE2b-256 | 27b30905352c21c484b8a1c2d3dc90427560921f36c965f826c6ff8eed2b1f59 |
Hashes for pandas_maxminddb-0.1.1-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bca5b636f88ffe472b74fbd1d936c79a4d1ae6b29150c45a432b903c53d72aca |
|
MD5 | ee68ae41feba34bc552cf08b38e773f6 |
|
BLAKE2b-256 | 2e344f551075ffce2040c8c87ccbd32ae2c35a4e24dcd67a1775fc109d61b560 |
Hashes for pandas_maxminddb-0.1.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67dbdf181f5c2f1b6e044196907d44bc575a3450de77bfb44d213ab4dc6846be |
|
MD5 | 474003cd333f963727c5e60302ccae54 |
|
BLAKE2b-256 | e8909a5454c848b32671c3541cf5b5f7fc7e11ba696ce4ec42f3268d23b98882 |
Hashes for pandas_maxminddb-0.1.1-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8de5f33a05ade6d035e368fb1ed9a5acb36a24785f506e595498cc8058411fc5 |
|
MD5 | c2573681a524ba2cf5a412a493b4afdd |
|
BLAKE2b-256 | b484143fc4774058266dd24c467e3e43b83cf1ee6566133e0a26c4d53c91bc16 |
Hashes for pandas_maxminddb-0.1.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8fb1bcfcda67632f3220371becd0ba1dfa110206c0b71a7536d4ecaf4f3f96d8 |
|
MD5 | 49f709f9e1f26f15c328065bc8520751 |
|
BLAKE2b-256 | 51c54738aeda82ad3a50536c26ca3af12e86c947d19b9d3450e4266af8375a74 |
Hashes for pandas_maxminddb-0.1.1-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b3cf012d55e1a6a1d171b38f245abc10325ae3a8c49a2a9148218c2e580226d |
|
MD5 | c5032ed108572738c7902c5732d2720a |
|
BLAKE2b-256 | ca38876439d665f827102ecd6d4c6dc0c96b20f3c6395da979df698b574c0e90 |
Hashes for pandas_maxminddb-0.1.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22d0e6abddc6d4baeab1515ec0e40272080b48a708887b593b475f0e3dee115a |
|
MD5 | d9737d9d8666e9d139c5d06a1cfd81da |
|
BLAKE2b-256 | 93883bf95e4ad1e91ee035249e108bfd0d28802bf4143109cab82d50a980b0f2 |
Hashes for pandas_maxminddb-0.1.1-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc13075a5091a185756889b52086a2ea474f25b34e37584504fe9b3046eefc5c |
|
MD5 | d657d8eb44972c3eb6967e7b2b6a728f |
|
BLAKE2b-256 | 45ec526da1e708e049e1bedd6fc82b6c91020ae6f068b8ac07c9a92c6126659e |
Hashes for pandas_maxminddb-0.1.1-cp38-cp38-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36e181f7624739da9895fb70382d379afaa19d3a72f57f5521c28584d12edeb2 |
|
MD5 | b95c89388cae96fd14ca46e60548b30c |
|
BLAKE2b-256 | b6ffcb6ea790221c65916f709c22069876c1795e64be990dcba467507ab9b714 |
Hashes for pandas_maxminddb-0.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c18f3693cd4e0ad5362f0cc9127d6f275f5672142093047ea9525354d46405e8 |
|
MD5 | 162b154014d624817c2ddcd5acf0140a |
|
BLAKE2b-256 | 21fb13b9901a45797bd6a1b113f967750eac2dbbd0fb4cb6e8b8e3b398caf544 |
Hashes for pandas_maxminddb-0.1.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 53421a4f168e10d88eaca4ade61bd8077ae392e47721649680d5d889eeefecc3 |
|
MD5 | 45b12232bec7c0839dd58be7d39110c1 |
|
BLAKE2b-256 | 9733ff95a3a8490382e24972d532ba786c8cb9b2011a2b2f74956d713722a7c5 |
Hashes for pandas_maxminddb-0.1.1-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3405653ba9a7bba5d153d97089c3cb7061ce1c4653c6a36d8577fc3cb6fedcee |
|
MD5 | 8dfdd67a821f51df8fd7d7d5ce94770d |
|
BLAKE2b-256 | 0ef0f557c4938533ba29f2da37d39270157c4fcf2a84d54a5aeb931cea8a168c |
Hashes for pandas_maxminddb-0.1.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 175cc5b3c7c8a51d93c489a8f1ac3adc23e00cb3cb7648cc2272ae8ddf0576c7 |
|
MD5 | 7f5c6a5e15eaa092effeb4936baa2f14 |
|
BLAKE2b-256 | 1afbb2c3933bd45f9eb875da06fea228293ce670024d84545c0ac6a41947f9e2 |
Hashes for pandas_maxminddb-0.1.1-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41e79b94b022f7bfc02a3d1d0c6174abf0520d240d6f3afe4c8f1e8d9cc068f9 |
|
MD5 | 06374a70ae6f1b60f40abf18984a3772 |
|
BLAKE2b-256 | e31f99e98f742f01f3b8a94caeaa67c1ef47a325b50b4caf01e0a75ea7f7d125 |