Skip to main content

A collection of Python utility functions for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, and minimising locking

Project description

pg-bulk-ingest

PyPI package Test suite Code coverage

A Python utility function for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, allowing concurrent reads as much as possible.

Allowing concurrent writes is not an aim of pg-bulk-ingest. It is designed for use in ETL pipelines where PostgreSQL is used as a data warehouse, and the only writes to the table are from pg-bulk-ingest. It is assumed that there is only one pg-bulk-ingest running against a given table at any one time.

Features

pg-bulk-ingest exposes a single function as its API that:

  • Creates the tables if necessary
  • Migrates any existing tables if necessary, minimising locking
  • Ingests data in batches, where each batch is ingested in its own transaction
  • Handles "high-watermarking" to carry on from where a previous ingest finished or errored
  • Optionally performs an "upsert", matching rows on primary key
  • Optionally deletes all existing rows before ingestion
  • Optionally calls a callback just before each batch is visible to other database clients

Visit the pg-bulk-ingest documentation for usage instructions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pg_bulk_ingest-0.0.54.tar.gz (9.4 kB view details)

Uploaded Source

Built Distribution

pg_bulk_ingest-0.0.54-py3-none-any.whl (8.8 kB view details)

Uploaded Python 3

File details

Details for the file pg_bulk_ingest-0.0.54.tar.gz.

File metadata

  • Download URL: pg_bulk_ingest-0.0.54.tar.gz
  • Upload date:
  • Size: 9.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for pg_bulk_ingest-0.0.54.tar.gz
Algorithm Hash digest
SHA256 1b57ab9af73ef641ad03da45ecd8754e8413cae9b394d1725176ac5dfc779319
MD5 e9f4635edd0c8dab4782a135a60c6410
BLAKE2b-256 26024adc44d466dc8ecb053b9cde6f92781ecfa61b18ff0cd0402f09e78b1c8f

See more details on using hashes here.

File details

Details for the file pg_bulk_ingest-0.0.54-py3-none-any.whl.

File metadata

File hashes

Hashes for pg_bulk_ingest-0.0.54-py3-none-any.whl
Algorithm Hash digest
SHA256 c87f916c130761f55b084e01d265a3523b3ab4afbca639f5a9c7b556ad4ef7c4
MD5 5bbd8d328049c69f16a220f48edbbcec
BLAKE2b-256 d8595f626142cebdbf909df5efb95e1d11692b95af3dbbafac22c90e70357e0d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page