RTB House apps pg_stream_copy
Project description
PG Stream Copy
Fast COPY TO postgresql table directly from python by converting input data to bytes and stream to psycopg2 cursor using COPY <table> FROM STDIN BINARY
Benchmark:
The test with 1 mln rows of different column types on docker environment gave results:
- ~21.5s for
pg_stream_copy
- ~54s for
psycopg2.extras.execute_values
Usage:
from datetime import date
from psycopg2 import connect
from pg_stream_copy import Schema, WriterEncoder
conn = connect('postgresql://postgres@localhost')
cursor = conn.cursor()
table_name = 'public.example_table'
cursor.execute(f'''
CREATE TABLE {table_name} (
_smallint SMALLINT NULL,
_integer INTEGER NULL,
_bigint BIGINT NULL,
_float DOUBLE PRECISION NULL,
_numeric NUMERIC NULL,
_character_varying CHARACTER VARYING NULL,
_date DATE NULL
);
''')
schema = Schema.load_from_table(cursor, table_name)
with WriterEncoder(cursor, table_name, schema) as writer_encoder:
writer_encoder.append_tuple((2, 3, 4, 2.34, 'foo bar', date(2019, 2, 1)))
writer_encoder.append_dict({
'_smallint': 200,
'_integer': 300,
'_bigint': 400,
'_float': 234,
'_numeric': Decimal("-12.34")
'_character_varying': 'bar baz',
'_date': date(2019, 2, 3),
})
conn.commit()
conn.close()
Supported PostgreSQL types:
- boolean
- smallint
- integer
- bigint
- double precision
- numeric
- character varying
- text
- date
- timestamp
- timestamp with time zone
- json
- jsonb
Development:
# prepare env
docker-compose run py bash
python -m venv venv
pip install -e .[dev,e2e]
# run tests
pytest tests/
If you need to test different PostgreSQL and Python version, you can use env vars:
PYTHON_VERSION=3.7 PG_VERSION=10 docker-compose build
PYTHON_VERSION=3.7 PG_VERSION=10 docker-compose run py ...
or use CI script:
PG_VERSION=11 PYTHON_VERSION=3.7 ./bin/tests.sh
currently pg_stream_copy is supporting Python 3.7 and 3.8 and PostgreSQL v10, v11, v12
Changelog
1.2.0
- Added new data type - numeric
1.1.0
- Added new data types - text, json, jsonb
1.0.3
- Added README to pypi.org
- Added LICENCE, CHANGELOG
1.0.2
- Updated pypi token
1.0.1
- Added Travis CI
1.0.0
- Initial version
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pg_stream_copy-1.5.4.tar.gz
(12.2 kB
view details)
Built Distribution
File details
Details for the file pg_stream_copy-1.5.4.tar.gz
.
File metadata
- Download URL: pg_stream_copy-1.5.4.tar.gz
- Upload date:
- Size: 12.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.8.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e49560e8f2f33bb8431a5f7b31228e4d056deee8bb63ea379ae8c18dd5a32cf |
|
MD5 | acfaa12f28d5aef9d8a2c077160dd3ef |
|
BLAKE2b-256 | 3a897584b6441acf3139119470931d8f284f792fd9232f750ea1502501d9feab |
File details
Details for the file pg_stream_copy-1.5.4-py3-none-any.whl
.
File metadata
- Download URL: pg_stream_copy-1.5.4-py3-none-any.whl
- Upload date:
- Size: 16.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.8.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80b754672e3a02fb44e459cd3c5a5b0448c789284cc035fe753c99644894c496 |
|
MD5 | 46789159479f8696db3d996a8ad85386 |
|
BLAKE2b-256 | 79fe242755980facb0015bb20c255b48988f1779843d38d04d314cda155ff094 |