A collection of Python utility functions for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, and minimising locking
Project description
pg-bulk-ingest
A Python utility function for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, allowing concurrent reads as much as possible.
Allowing concurrent writes is not an aim of pg-bulk-ingest. It is designed for use in ETL pipelines where PostgreSQL is used as a data warehouse, and the only writes to the table are from pg-bulk-ingest. It is assumed that there is only one pg-bulk-ingest running against a given table at any one time.
Features
pg-bulk-ingest exposes a single function as its API that:
- Creates the tables if necessary
- Migrates any existing tables if necessary, minimising locking
- Ingests data in batches, where each batch is ingested in its own transaction
- Handles "high-watermarking" to carry on from where a previous ingest finished or errored
- Optionally performs an "upsert", matching rows on primary key
- Optionally deletes all existing rows before ingestion
- Optionally calls a callback just before each batch is visible to other database clients
Visit the pg-bulk-ingest documentation for usage instructions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file pg_bulk_ingest-0.0.54.tar.gz
.
File metadata
- Download URL: pg_bulk_ingest-0.0.54.tar.gz
- Upload date:
- Size: 9.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b57ab9af73ef641ad03da45ecd8754e8413cae9b394d1725176ac5dfc779319 |
|
MD5 | e9f4635edd0c8dab4782a135a60c6410 |
|
BLAKE2b-256 | 26024adc44d466dc8ecb053b9cde6f92781ecfa61b18ff0cd0402f09e78b1c8f |
File details
Details for the file pg_bulk_ingest-0.0.54-py3-none-any.whl
.
File metadata
- Download URL: pg_bulk_ingest-0.0.54-py3-none-any.whl
- Upload date:
- Size: 8.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c87f916c130761f55b084e01d265a3523b3ab4afbca639f5a9c7b556ad4ef7c4 |
|
MD5 | 5bbd8d328049c69f16a220f48edbbcec |
|
BLAKE2b-256 | d8595f626142cebdbf909df5efb95e1d11692b95af3dbbafac22c90e70357e0d |