mmdb-writer 0.2.7

Make `mmdb` format ip library file which can be read by maxmind official language reader

• MaxMind-DB-Writer-python
  • Install
  • Usage
  • Examples
  • Using the Java Client
    • TLDR
    • Underlying Principles
  • Type Enforcement
  • Reference:

MaxMind-DB-Writer-python

Make mmdb format ip library file which can be read by maxmind official language reader

The official perl writer was written in perl, which was difficult to customize. So I implemented the MaxmindDB format ip library in python language.

MaxMind has now released an official Go version of the MMDB writer. If you prefer using Go, you can check out the official Go implementation mmdbwriter. This project still provides a Python alternative for those who need it.

Looking for better performance? The feature/rust branch is a beta Rust rewrite via PyO3. It uses a simplified API (no netaddr dependency) and is 6–8× faster overall, with insert speeds up to 5× faster than this pure-Python version.

Install the beta directly from GitHub:

pip install "git+https://github.com/vimt/MaxMind-DB-Writer-python.git@feature/rust"

Note: the beta requires a Rust toolchain to build from source, or pre-built wheels when available.

Install

pip install -U mmdb_writer

Usage

from netaddr import IPSet

from mmdb_writer import MMDBWriter

writer = MMDBWriter()

writer.insert_network(IPSet(['1.1.0.0/24', '1.1.1.0/24']), {'country': 'COUNTRY', 'isp': 'ISP'})
writer.to_db_file('test.mmdb')

import maxminddb

m = maxminddb.open_database('test.mmdb')
r = m.get('1.1.1.1')
assert r == {'country': 'COUNTRY', 'isp': 'ISP'}

Examples

see csv_to_mmdb.py Here is a professional and clear translation of the README.md section from Chinese into English:

Using the Java Client

If you are using the Java client, you need to be careful to set the int_type parameter so that Java correctly recognizes the integer type in the MMDB file.

Example:

from mmdb_writer import MMDBWriter

writer = MMDBWriter(int_type='i32')

Alternatively, you can explicitly specify data types using the Type Enforcement section.

Underlying Principles

In Java, when deserializing to a structure, the numeric types will use the original MMDB numeric types. The specific conversion relationships are as follows:

mmdb type	java type
float	Float
double	Double
int32	Integer
uint16	Integer
uint32	Long
uint64	BigInteger
uint128	BigInteger

When using the Python writer to generate an MMDB file, by default, it converts integers to the corresponding MMDB type based on the size of the int. For instance, int(1) would convert to uint16, and int(2**16+1) would convert to uint32. This may cause deserialization failures in Java clients. Therefore, it is necessary to specify the int_type parameter when generating MMDB files to define the numeric type accurately.

Type Enforcement

MMDB supports a variety of numeric types such as int32, uint16, uint32, uint64, uint128 for integers, and f32, f64 for floating points, while Python only has one integer type and one float type (actually f64).

Therefore, when generating an MMDB file, you need to specify the int_type parameter to define the numeric type of the MMDB file. The behaviors for different int_type settings are:

int_type	Behavior
auto (default)	Automatically selects the MMDB numeric type based on the value size. Rules: int32 for value < 0 uint16 for 0 <= value < 2^16 uint32 for 2^16 <= value < 2^32 uint64 for 2^32 <= value < 2^64 uint128 for value >= 2^64.
i32	Stores all integer types as int32.
u16	Stores all integer types as uint16.
u32	Stores all integer types as uint32.
u64	Stores all integer types as uint64.
u128	Stores all integer types as uint128.

If you want to use different int types for different scenarios, you can use type wrapping:

from mmdb_writer import MMDBWriter, MmdbI32, MmdbF32

writer = MMDBWriter()
# the value of field "i32" will be stored as int32 type
writer.insert_network(IPSet(["1.0.0.0/24"]), {"i32": MmdbI32(128), "f32": MmdbF32(1.22)})

Reference:

• MaxmindDB format
• geoip-mmdb