pg-bulk-ingest

PyPI package Test suite Code coverage

A Python utility function for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, allowing concurrent reads as much as possible.

Allowing concurrent writes is not an aim of pg-bulk-ingest. It is designed for use in ETL pipelines where PostgreSQL is used as a data warehouse, and the only writes to the table are from pg-bulk-ingest. It is assumed that there is only one pg-bulk-ingest running against a given table at any one time.

Features

pg-bulk-ingest exposes a single function as its API that:

  • Creates the tables if necessary
  • Migrates any existing tables if necessary, minimising locking
  • Ingests data in batches, where each batch is ingested in its own transaction
  • Handles "high-watermarking" to carry on from where a previous ingest finished or errored
  • Optionally performs an "upsert", matching rows on primary key
  • Optionally deletes all existing rows before ingestion
  • Optionally calls a callback just before each batch is visible to other database clients

Visit the pg-bulk-ingest documentation for usage instructions.

Project links

Data verified by PyPI on May 1, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for uktrade from gravatar.com uktrade

Credits

Author: Department for Business and Trade

GitHub Statistics

Data verified by PyPI on May 1, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License

Requires

Python >=3.8.2

Provides Extra

ci ci-sqlalchemy1-with-pg-arrow ci-sqlalchemy1-without-pg-arrow ci-sqlalchemy2-with-pg-arrow ci-sqlalchemy2-without-pg-arrow dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pg_bulk_ingest-0.1.3.tar.gz (23.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

pg_bulk_ingest-0.1.3-py3-none-any.whl (23.9 kB view details)

Uploaded Python 3

File details

Details for the file pg_bulk_ingest-0.1.3.tar.gz.

File metadata

  • Download URL: pg_bulk_ingest-0.1.3.tar.gz
  • Upload date:
  • Size: 23.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for pg_bulk_ingest-0.1.3.tar.gz
Algorithm Hash digest
SHA256 df18d58a4f27d3e5ac6b1e97c4deb23305210eae9ab8c3b987967448853c0ba5
MD5 d4a4d8e66e09999af68efdcdd93809c9
BLAKE2b-256 4f0486a4c07e9ccc9023c7a4d950217103ea7569d9b6e439c3e1f7dd1d89f261

See more details on using hashes here.

Provenance

The following attestation bundles were made for pg_bulk_ingest-0.1.3.tar.gz:

Publisher: deploy-package-to-pypi.yml on uktrade/pg-bulk-ingest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pg_bulk_ingest-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: pg_bulk_ingest-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 23.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for pg_bulk_ingest-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 112a7a6827a818d8f7e5995f515a0e641a8c86f0b5a4cc91b7897639c80eff0a
MD5 8ae24605d449c5048c57d2e95749f1aa
BLAKE2b-256 e182c7d75a9f9156497e5f2fbaafa9c7891c2d040500a700d5cde6c6c6bd10a0

See more details on using hashes here.

Provenance

The following attestation bundles were made for pg_bulk_ingest-0.1.3-py3-none-any.whl:

Publisher: deploy-package-to-pypi.yml on uktrade/pg-bulk-ingest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.1.3 This release

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

0.0.58

2 files

0.0.57

2 files

0.0.56

2 files

0.0.55

2 files

0.0.54

2 files

0.0.53

2 files

0.0.52

2 files

0.0.51

2 files

0.0.50

2 files

0.0.49

2 files

0.0.48

2 files

0.0.47

2 files

0.0.46

2 files

0.0.45

2 files

0.0.44

2 files

0.0.43

2 files

0.0.42

2 files

0.0.41

2 files

0.0.40

2 files

0.0.39

2 files

0.0.38

2 files

0.0.37

2 files

0.0.36

2 files

0.0.35

2 files

0.0.34

2 files

0.0.33

2 files

0.0.32

2 files

0.0.31

2 files

0.0.30

2 files

0.0.29

2 files

0.0.28

2 files

0.0.27

2 files

0.0.26

2 files

0.0.24

2 files

0.0.23

2 files

0.0.22

2 files

0.0.21

2 files

0.0.20

2 files

0.0.19

2 files

0.0.18

2 files

0.0.17

2 files

0.0.16

2 files

0.0.15

2 files

0.0.14

2 files

0.0.13

2 files

0.0.12

2 files

0.0.11

2 files

0.0.10

2 files

0.0.9

2 files

0.0.8

2 files

0.0.7

2 files

0.0.6

2 files

0.0.5

2 files

0.0.4

2 files

0.0.3

2 files

0.0.2

2 files

0.0.1

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page