Stream results of multi statement PostgreSQL queries from Python
Project description
streampq
Stream results of multi-statement PostgreSQL queries from Python without a server-side cursor.
For complex situations where multiple statements are needed, but also where results are too big to store in memory at once. Existing Python PostgreSQL drivers don't seem to handle this case well - typically they require a server-side cursor, but SQL syntax doesn't allow a server-side cursor to "wrap" multiple SQL statements. This library side steps that problem by using libpq's single-row mode.
CTRL+C is respected during queries: even in the middle of a slow query, a KeyboardInterrupt is raised and bubbles up as is typical in Python programs. Other Python PostgreSQL drivers don't do this - CTRL+C can appear to be ignored. This is done by using libpq's Asynchronous Command Processing to wait for query results using Python's selectors module rather than in blocking libpq functions.
Installation
pip install streampq
The libpq
binary library is also required. This is typically either already installed, or installed by:
- macOS + brew:
brew install libpq
- Linux (Debian):
apt install libpq5
- Linux (Red Hat):
yum install postgresql-libs
Usage
from streampq import streampq_connect
# libpq connection paramters
# https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
#
# Any can be ommitted and environment variables will be used instead
# https://www.postgresql.org/docs/current/libpq-envars.html
connection_params = (
('host', 'localhost'),
('port', '5432'),
('dbname', 'postgres'),
('user', 'postgres'),
('password', 'password'),
)
# SQL statement(s) - if more than one, separate by ;
sql = '''
SELECT * FROM my_table;
SELECT * FROM my_other_table;
'''
with streampq_connect(connection_params) as query:
for (columns, rows) in query(sql):
for row in rows:
pass
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.