No project description provided
Project description
flaco
The easiest and perhaps most memory efficient way to get PostgreSQL data (more flavors to come?) into Arrow (IPC/Feather) or Parquet files.
If you're trying to load data directly into Pandas then you may find that evan a 'real' 100MB can cause bloat upwards of 1GB. Expanding this can cause significant bottle necks in processing data efficiently.
Since Arrow supports efficient and even larger-than-memory processing, as with dask, duckdb, or others. Just getting data onto disk is sometimes the hardest part; this aims to make that easier.
NOTE: This is still a WIP, and is purpose built for my needs. I intend to generalize it more to be useful towards a wider audience. Issues and pull requests welcome!
Example
from flaco import read_sql_to_file, FileFormat
uri = "postgresql://postgres:postgres@localhost:5432/postgres"
stmt = "select * from my_big_table"
read_sql_to_file(uri, stmt, 'output.data', FileFormat.Parquet)
# Then with pandas...
import pandas as pd
df = pd.read_parquet('output.data')
# pyarrow... (memory mapped file, where potentially larger than memory)
import pyarrow as pa
with pa.memory_map('output.data', 'rb') as source:
table = pa.ipc.open_file(source).read_all() # mmap pyarrow.Table
# DuckDB...
import duckdb
cur = duckdb.connect()
cur.execute("select * from read_parquet('output.data')")
# Or anything else which works with Arrow and/or Parquet files
License
Why did you choose such lax licensing? Could you change to a copy left license, please?
...just kidding, no one would ask that. This is dual licensed under Unlicense or MIT, at your discretion.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for flaco-0.6.0rc1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8473cd7ae402a8bdebf58889370cc352249b2cb78aec4dd0d43f1182616b233f |
|
MD5 | f0f8bb02b5240956011d70412218ef32 |
|
BLAKE2b-256 | 3f57c1134bd068c01304dcb63e7354c018c1a07bfe27360fb51608a39e64cc3b |
Hashes for flaco-0.6.0rc1-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0e76b479ada3816fda14d6eb669493016ab954f8a3dee1284d6e994c6ceb8d1 |
|
MD5 | 50a9621cefb9efcda55aa3c3c7394ac5 |
|
BLAKE2b-256 | 05b2bb9764d2dfc1bf8dd667d45537ce7060f69db85ecc7bae5f1822189c84a5 |
Hashes for flaco-0.6.0rc1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5856c4bcfce80e178af5628a2537c537ea16ba3098b9e570f3efe9833999e750 |
|
MD5 | ab3d4e7cef369fb1a12f72e9b1d77a52 |
|
BLAKE2b-256 | d2d3ff5dedf43b3fd7feb8f5203fa8ed765455b567502a42bb69e951a70f7f07 |
Hashes for flaco-0.6.0rc1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca338cd46fec4f2977a8ab245a1e8a84cdb7eeb9a2ff7264613f2b3289aba6bb |
|
MD5 | 80f1a8ab43aeaf62443a4819fcf20063 |
|
BLAKE2b-256 | ca49180b8cd4ffd4bb5981a35c72a8fd3b002b2f8b8c7cdb7beab8bea51ee942 |
Hashes for flaco-0.6.0rc1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb000c57917f4eab02220aea956a270bdafffddcd334eca0ec0cdcbd35ba1325 |
|
MD5 | da45d0a008d936666302b1f84dee01ed |
|
BLAKE2b-256 | bafaa7455280a3026d3cc4672026b6e414ca70409e34335889063dab88df4167 |
Hashes for flaco-0.6.0rc1-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afb6d259f82f21e70a6468c3a83e87038f2b32983a16c03c54b570328c0bcf26 |
|
MD5 | f64d4279df50c148eefc84bdd78c1286 |
|
BLAKE2b-256 | 3e954a566cbc57182be411c450f5b2f860edb4042f47301619dcbae1d6449a96 |
Hashes for flaco-0.6.0rc1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | deae6179e9d4fb27334bb22bd4b5f98c652c132a71addb40ea0b0ead9e81f13e |
|
MD5 | 42c5a443da7c129dc27f413c9cba45f2 |
|
BLAKE2b-256 | 25fc802073bc9aea7dc0356cec31e670e5b531de2dd403a79554201bbab965bc |
Hashes for flaco-0.6.0rc1-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 343bfa8f9533f46874ab8ef46efe669a1446dc85a1814a41af6133be4de62e2f |
|
MD5 | ed5b0cbc7850da84fe5a503181d75bcb |
|
BLAKE2b-256 | b8c32d8169823543d711ae3300f9c768a9b4f55421aa6393a77b0ab85477fc0c |
Hashes for flaco-0.6.0rc1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d741e31481b0f8ce5a151edea7e50d9c39f73e810a38b5b93dd7b0439c0135ff |
|
MD5 | 160472f578118e7369c476d0c37653e6 |
|
BLAKE2b-256 | 2dcdd2192f464332194f778c30a6313f4a4c3ad7fcb27b250ff6e9c54eaaee75 |
Hashes for flaco-0.6.0rc1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a08b199aadc0e42a3d807aa5df429067209693a091cc773d7f4482c7861537be |
|
MD5 | 506540220515d99310b33b02913c90d3 |
|
BLAKE2b-256 | 914bc10dcd07eb36199093c73da1b075d06c0c0766e0d4f19668b3d4fde61f2b |
Hashes for flaco-0.6.0rc1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 065cefa095175f5f6554265d00ffaeabb01abe4eeb0f608f51c026f6c2ac0bd9 |
|
MD5 | f2a02ee694bb6c3df3587a11670c9da4 |
|
BLAKE2b-256 | 585c80a5af83af2f9737259adc69722432bb0ef9691b3cd90436c589c19e2877 |
Hashes for flaco-0.6.0rc1-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a7542f1024db3b44ac2e9fa985277fb1a80118afc6f1a0e2aa4a9ef27d96bf2 |
|
MD5 | 1d0f1aeff0f2a73fb7fed804c53e7e72 |
|
BLAKE2b-256 | df9a42073cda587d4a93ca4ebeed3f9c194eb21f5babab86d60d5ade5459259c |
Hashes for flaco-0.6.0rc1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b4f058bfdf5c6668eec172ab7bc0039ab4c1e94864ca8ddd2b064dbbeda3902 |
|
MD5 | 200b70ec97179da79048b629008ea8f5 |
|
BLAKE2b-256 | 25d5029c0312d29abe8e17dc6c3e7fa4e7be1d59ce2db07cbad6bb6037dbeb82 |
Hashes for flaco-0.6.0rc1-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fa2ad997f25c3cd1aaaad08b3274ac418a0f0e79bff37f336cd75a57b39c851 |
|
MD5 | 79a8a033a53150857ec63ea98f87ae14 |
|
BLAKE2b-256 | 02114181416409887ac51f9b73cab34b9136db43c9009ba817720316f247bf4f |
Hashes for flaco-0.6.0rc1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18922734930f399942587e4db2e2c9baf552c12308cad455923efe6afcdd8132 |
|
MD5 | b01052952c05d860f5e6d24cbca3fc66 |
|
BLAKE2b-256 | 547734a53b6fcaf229544a5f291f643ceb855fa60d59a23d8b1b9abeffd76397 |
Hashes for flaco-0.6.0rc1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21c0545380da13f0bbdc7bdf502e8adb383edcb940162d5a5be1c3af6c1a9c1f |
|
MD5 | 43f91153cb09e63b1bc3b667a2a43092 |
|
BLAKE2b-256 | 88a51c099962f93f93bc50d4ddc0f6fbd75692b2ecafdf1988cce41989fae097 |
Hashes for flaco-0.6.0rc1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 484e85f559762ef0db5baafc68c7e7e0e5827a39378e06d887a8366431acbc39 |
|
MD5 | 37eadb87cb8075834cfe08ff5360214f |
|
BLAKE2b-256 | fc004fe64b2fe2294d6d59b487242682d38e7d67e9718332044477b06a012020 |
Hashes for flaco-0.6.0rc1-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e68dcf6705a394031ae7b996d3320e491906d53b2360b0799618d731d24446a4 |
|
MD5 | 26cc31b17a443473e39975018e42e31a |
|
BLAKE2b-256 | afc32a31518e5c71a0f5f28b5651b5a7bc99b3696ee5a83db23fe9b4b02a224e |
Hashes for flaco-0.6.0rc1-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48aab82814b7d50304d0ebdc729e2aca01540aca4d45e3fdac3293d0e7b32d76 |
|
MD5 | 3b28895a38d26fd4d7048a79579a0852 |
|
BLAKE2b-256 | 6356091ff5c0aa086862fadb84efa48e48ff4d570f07a8b81b153ad6c6419054 |
Hashes for flaco-0.6.0rc1-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 966ffa8f06d3d750f3edd6b911289c21d8dade36c8641ebeff915b583bcb29bf |
|
MD5 | 806981d5ffc9b31205bcd57717175624 |
|
BLAKE2b-256 | 333d1e587522d9d424360103a3b543f6b63bf5300e01517b510dd7c3811e4163 |
Hashes for flaco-0.6.0rc1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56f663eec6f94dbca124c7afb1cdef6fdcfa3c6f9ed23ab5469d85aa39228a55 |
|
MD5 | cf32a2f080a10c15664e6d5e7f02fc4f |
|
BLAKE2b-256 | 7022038c4422c94ff7343bdb13ae8418f3a19f0037fe52f43362fdfe8a6907e2 |
Hashes for flaco-0.6.0rc1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bc3fa8eea82d756ee8bb5d645c13de716da01c231d996d1b15fc7ef69b5ec43 |
|
MD5 | 015b0f918184f42d92844822335078b9 |
|
BLAKE2b-256 | 6990f5dc32a12246a1229851cb450f70a1db32578775e124ee2f66f41735aff6 |
Hashes for flaco-0.6.0rc1-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de1bb5e4b09d9cd335053c32b69f568551cd6d378e78f4adac6ebfce97022fea |
|
MD5 | 3c589cacb7d9e26abca05bda2bafe57e |
|
BLAKE2b-256 | d97c3b2d61b1bd8ab6d3f68fc627fccccc231ebd709af9d7598eabcbc7d0e686 |