MariaDB Binlog Indexer
This tool helps to index mariadb binlogs and store them in compact format to query faster.
File Format
- DuckDB - Store all the metadata of binlogs.
- Parquet - Store the actual queries.
DB Schema
┌─────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │ null │ key │ default │ extra │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├─────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ binlog │ VARCHAR │ YES │ NULL │ NULL │ NULL │
│ db_name │ VARCHAR │ YES │ NULL │ NULL │ NULL │
│ table_name │ VARCHAR │ YES │ NULL │ NULL │ NULL │
│ timestamp │ INTEGER │ YES │ NULL │ NULL │ NULL │
│ type │ VARCHAR │ YES │ NULL │ NULL │ NULL │
│ row_id │ INTEGER │ YES │ NULL │ NULL │ NULL │
│ event_size │ INTEGER │ YES │ NULL │ NULL │ NULL │
└─────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┘
Usage
Before starting up please note down,
- All the timestamps are in seconds UTC
- Available query types are
SELECT,INSERT,UPDATE,DELETE,OTHER
Also, check the python docstrings for more details.
Initialize Indexer
from mariadb_binlog_indexer import Indexer
indexer = Indexer(
base_path="<path to store indexer related files>",
db_name="metadata.db", # The name of the database to store metadata
)
Index New Binlog
indexer.add(
binlog_path="<path to binlog>",
batch_size=10000, # The batch size to insert binlog in duckdb
)
Remove Indexes of Binlog
indexer.remove(
binlog_path="<path to binlog>",
)
Generate Timeline
This function will provide a summary of binlog event in a given time range. It will split the time range into 30 parts and provide event related information for each part.
indexer.get_timeline(
start_timestamp=1746534427, # The start timestamp in seconds UTC
end_timestamp=1756534427, # The end timestamp in seconds UTC,
type="INSERT", # Optional
database="test_db", # Optional
)
Get Row Ids
This function will help to get the row ids of each binlog based on our request. The purpose of this function is to help finding required row ids beforehand to implement pagination on the other end.
Only reason of doing this is to reduce the search in the parquet files.
indexer.get_row_ids(
start_timestamp=1746534427, # The start timestamp in seconds UTC
end_timestamp=1756534427, # The end timestamp in seconds UTC,
type="INSERT", # Optional
database="test_db", # Optional
table="test_table", # Optional
search_str="test", # Optional
)
Get Queries from Parquet Files
This function will help to get the queries from parquet files.
indexer.get_queries(
row_ids={
"binlog_1": [101, 102, 103],
"binlog_2": [104, 105, 106],
},
database="test_db", # Optional
)
You need to provide database name as filtering purpose only.
Metadata
Release files for mariadb-binlog-indexer 0.0.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_x86_64.whl | Python 3 | none | Linux glibc 2.17+ x86-64 | Details for mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_x86_64.whl |
| mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_aarch64.whl | Python 3 | none | Linux glibc 2.17+ ARM64 | Details for mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_aarch64.whl |
Total release size: 56.1 MB
Release files / mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_x86_64.whl
| Download URL | mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_x86_64.whl |
|---|---|
| Size | 28.8 MB |
| Tags | Linux glibc 2.17+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
5530326e8213ed4a5d2813065eb3d0bef3f0e08b4abbc062be5302084e129b59
|
|
BLAKE2b-256 checksum How to use checksums |
e2d20415f340639aaf52c32ab781aa99b06bf846b82cd01f43cfd60ecd241ba9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.8.10
|
Release files / mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_aarch64.whl
| Download URL | mariadb_binlog_indexer-0.0.20-py3-none-manylinux2014_aarch64.whl |
|---|---|
| Size | 27.3 MB |
| Tags | Linux glibc 2.17+ ARM64 Python 3 |
|
SHA-256 checksum How to use checksums |
f444f39390f1bf4da472b123db6a025c7d4d277a9d7be92f1a182f90536ad36c
|
|
BLAKE2b-256 checksum How to use checksums |
5e65f779af8bd10bdfa93a4b16e03660bfa128a206bde37ebe30e894ccd8ccd4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.8.10
|
Security for mariadb-binlog-indexer 0.0.20
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.