Stream results of multi statement PostgreSQL queries from Python without a server-side cursor
Project description
streampq
Stream results of multi-statement PostgreSQL queries from Python without server-side cursors. Has benefits over some other Python PostgreSQL libraries:
-
Streams results from complex multi-statement queries even though SQL doesn't allow server-side cursors for such queries - suitable for large amounts of results that don't fit in memory.
-
CTRL+C (SIGINT) by default behaves as expected even during slow queries - a
KeyboardInterrupt
is raised and quickly bubbles up through streampq code. Unless client code prevents it, the program will exit. -
Every effort is made to cancel queries on
KeyboardInterrupt
,SystemExit
, or errors - the server doesn't continue needlessly using resources.
Installation
pip install streampq
The libpq
binary library is also required. This is typically either already installed, or installed by:
- macOS + brew:
brew install libpq
- Linux (Debian):
apt install libpq5
- Linux (Red Hat):
yum install postgresql-libs
Usage
from streampq import streampq_connect
# libpq connection paramters
# https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
#
# Any can be ommitted and environment variables will be used instead
# https://www.postgresql.org/docs/current/libpq-envars.html
connection_params = (
('host', 'localhost'),
('port', '5432'),
('dbname', 'postgres'),
('user', 'postgres'),
('password', 'password'),
)
# SQL statement(s) - if more than one, separate by ;
sql = '''
SELECT * FROM my_table WHERE my_col = {first};
SELECT * FROM my_other_table WHERE my_other_col = {second};
'''
with streampq_connect(connection_params) as query:
for (columns, rows) in query(sql, literals=(
('first', 'a value'),
('second', 'another value'),
)):
for row in rows:
pass
Exceptions
Exceptions derive from streampq.StreamPQError
. If there is any more information available on the error, it's added as a string in its args
property. This is included in the string representation of the exception by default.
Exception hierarchy
-
StreamPQError
Base class for all explicitly-thrown exceptions
-
ConnectionError
An error occurred while attempting to connect to the database.
-
QueryError
An error occurred while attempting to run a query. Typically this is due to a syntax error or a missing column.
-
CancelError
An error occurred while attempting to cancel a query.
-
CommunicationError
An error occurred communicating with the database after successful connection.
-
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for streampq-0.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97b5b2894af17e9c80b9c863a6291ba06f0e78397dedac35efbd68a4c10fd237 |
|
MD5 | cc0b483453ceea4df6417865fd1b1fc4 |
|
BLAKE2b-256 | d2b9db315542ce4fab66e7ca738bd952899737b5565c04d33b93bcbc59832fd1 |