No project description provided
Project description
flaco
The easiest and perhaps most memory efficient way to get PostgreSQL data (more flavors to come?) into Arrow (IPC/Feather) or Parquet files.
If you're trying to load data directly into Pandas then you may find that evan a 'real' 100MB can cause bloat upwards of 1GB. Expanding this can cause significant bottle necks in processing data efficiently.
Since Arrow supports efficient and even larger-than-memory processing, as with dask, duckdb, or others. Just getting data onto disk is sometimes the hardest part; this aims to make that easier.
NOTE: This is still a WIP, and is purpose built for my needs. I intend to generalize it more to be useful towards a wider audience. Issues and pull requests welcome!
Example
from flaco import read_sql_to_file, FileFormat
uri = "postgresql://postgres:postgres@localhost:5432/postgres"
stmt = "select * from my_big_table"
read_sql_to_file(uri, stmt, 'output.data', FileFormat.Parquet)
# Then with pandas...
import pandas as pd
df = pd.read_parquet('output.data')
# pyarrow... (memory mapped file, where potentially larger than memory)
import pyarrow as pa
with pa.memory_map('output.data', 'rb') as source:
table = pa.ipc.open_file(source).read_all() # mmap pyarrow.Table
# DuckDB...
import duckdb
cur = duckdb.connect()
cur.execute("select * from read_parquet('output.data')")
# Or anything else which works with Arrow and/or Parquet files
License
Why did you choose such lax licensing? Could you change to a copy left license, please?
...just kidding, no one would ask that. This is dual licensed under Unlicense or MIT, at your discretion.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for flaco-0.6.0rc3-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25735c20ffcdd0d1ff44f32bba15fddea620b5bb59a5bac9b78df570e436a43f |
|
MD5 | a7b27945fae50e3450a24d8fbbc0e0cd |
|
BLAKE2b-256 | a0333de5c0ee17e2181079cd9e5b571be521fcab4453ff3126a42c1ba1474311 |
Hashes for flaco-0.6.0rc3-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5fea3e8a4450a4a7657cc8451808dde0671b0cfca5c6068e4f7dea1cae44ddae |
|
MD5 | dbb09aa1c7d701e80fdb2fdde7bab47e |
|
BLAKE2b-256 | 5cffc5977f902e3bbf405a612054e0a10e5741375c1acc297ec89644591a78b0 |
Hashes for flaco-0.6.0rc3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1d038bba41f9d6116a51cd0bd293b909225d69be66ca7c49e72ae662a552c02 |
|
MD5 | 89bed8d28676fe4b832d7acfeb9b594d |
|
BLAKE2b-256 | 8063a22dad1e30b57a6d705634bd31a3b89c2d666478118b8c0c24dba79f8dd0 |
Hashes for flaco-0.6.0rc3-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 128a5406a2afe6a1cda05656a6786e3d8fc48e8be5658bf653cfa3f666323641 |
|
MD5 | 6979808cacb0d90680bf9c385760bcd4 |
|
BLAKE2b-256 | 5cfe6cc4b59aa1cd07b14b4eeaf16afa137e786d7d84f953002d8adaa921bc43 |
Hashes for flaco-0.6.0rc3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8eb506a31e5cab0016e8f88cd1933b3f7c6c51b2f8fbba812ca47fab436960dc |
|
MD5 | 2a7d478434e0684115dc9c63b68e4492 |
|
BLAKE2b-256 | 306d21b0fdab6a3a3dc8942f4d1ec468fb62e6defb26bdf3a6a877da1b124078 |
Hashes for flaco-0.6.0rc3-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78b8ad4cd6ec90b2a705a41bf3901584e3e12d9ba7f46a94e5aa169c2890fc9c |
|
MD5 | 3666a444e4d68eb270dfc5212334237a |
|
BLAKE2b-256 | e36f09706be9c7f381a8c5e0c9f7d2c131347049869660f5754dfeedd52a0c2d |
Hashes for flaco-0.6.0rc3-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2022ba1f420f231532c4fd052ba18ceb67fc22ddae9c5158993d06bc86eb60d |
|
MD5 | d01210a6bdb1ba993373ffb9aac8af84 |
|
BLAKE2b-256 | 0863c3d13e056d917faba15240e7713e95f39c81165effe39b40a717cf1d1663 |
Hashes for flaco-0.6.0rc3-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 464cdd493b37efa5067039eae1d81c2561c631e6f35b025a2d789125dbd9cae5 |
|
MD5 | e19221f9933e1bceef87aa7e73e1db6d |
|
BLAKE2b-256 | 9e02d481fe2f5f4c59922f9fd46bbcca8827c7278138f20b57267adfb06a05fe |
Hashes for flaco-0.6.0rc3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ef2716f1ddc2a02b6131365e2a4e3ea0103c76d5f91ae789bc9d1fb8b876bd9 |
|
MD5 | 6489c87dabb59cc768b043b6ee0ee623 |
|
BLAKE2b-256 | 60f78d09bec3f082f0c9a374a55089fc6be59e686832ccde98310ab67b74f0de |
Hashes for flaco-0.6.0rc3-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd5b0dd92b24c506ada046d12f985400eb2d08ccf9720bca8ceb9db36afc44d7 |
|
MD5 | e74297bc9c94ab192931a21d1c034407 |
|
BLAKE2b-256 | 68f397d69b05b191d0a2d6f4dcce31a2f26dc2f8aebbb3a2eff8a60e20cfd6a1 |
Hashes for flaco-0.6.0rc3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 538dc75ad959eb663732f6eada26acf0d067b19bf4968aee88117b9258fa3559 |
|
MD5 | f2b3a7602e6acbb42aaec94d964107c2 |
|
BLAKE2b-256 | 03eeb12f754b3589ebfbe38dff7e93af59341385350202671f5fe88a6c7fb1f8 |
Hashes for flaco-0.6.0rc3-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 826829990fb66639e828dbea1ee83874975f2bd2d84d7550b34ab696e36776a9 |
|
MD5 | bfe0e1ebb9bba557469ce23197c98661 |
|
BLAKE2b-256 | c3e51c31e5133bb7a87ff8069249b2cb566cdd0b6a910a3509a39905f5df0c4e |
Hashes for flaco-0.6.0rc3-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 755e023e9c1e4a15371ae17336778b7639c164edfa787d886c74f5ff1b7c9d7e |
|
MD5 | b3facf8e9ee89ba3384f51612fb663f5 |
|
BLAKE2b-256 | 418a0c21e25268d1ff0a0301f48d58366303ddeee42d451fcffa507d57b6ed0b |
Hashes for flaco-0.6.0rc3-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2eb07567339ac0e166e081cc2cdc356ebe727a1973d8912154ab0c5234f88aae |
|
MD5 | b74fb218e81c5c6e5b1855d9a701fc6a |
|
BLAKE2b-256 | 160434134ace3ab02a69b534e31cc12aab2a692fefc5aae1d40c8bd6c47484d9 |
Hashes for flaco-0.6.0rc3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5fac6f2a6104506e79bc6d66cd3443e4593febd846ff870ca6d08e8d59416f7 |
|
MD5 | 825e6e2282bb1894952f127834b202b5 |
|
BLAKE2b-256 | 84cc6ef093365950d151075ed42f82abc46cc7f008dc69129b9fc5084eaa8787 |
Hashes for flaco-0.6.0rc3-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e8b5bf0a431b266ebb210eb42a26692d53ef2737904edd7e1e3d1df83821621 |
|
MD5 | be13f527de69129457bce85fa1e5a1b6 |
|
BLAKE2b-256 | 89a3ec204ac6cb4f761d26f2c5d46064eb4b201e7d75c45ba038b29e98f63935 |
Hashes for flaco-0.6.0rc3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68df7598c0b0b43ceaa02cd9fe4446de60271964e453d29e24a94a74cdff0aed |
|
MD5 | f28e7f2e36b2886e92532210d23b073e |
|
BLAKE2b-256 | 9d042d2b58fc53b6cb1a38fcd9ef7b48559918aa8d4daa4d2a68d76ffdc363d1 |
Hashes for flaco-0.6.0rc3-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66b19fcef7a9cf5b819bdaf64808179673d8e0cc1c87986200c5a49b6f05e6fd |
|
MD5 | 35511112b0f939864b88c97f0470fae9 |
|
BLAKE2b-256 | d72b129517e926322998ef8580447cc3cc436c9468fb12529b97dc47d92674ad |
Hashes for flaco-0.6.0rc3-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 638ac1c80d1ac991cb7b5a6083d8688e28a028edf843ddaa5b3f284c23e12f1b |
|
MD5 | 0b69c937de8b3d1d592d677309bff564 |
|
BLAKE2b-256 | 0f382bbf781fdbe539849b09c66dd1b30e38b79125612d2fd14c9b473e240089 |
Hashes for flaco-0.6.0rc3-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 889e9de9171a93e4148cc3ab144989d12aa3cde176baa39654d328ff078f9591 |
|
MD5 | 6d8ad675c759908b98226b2d473518f9 |
|
BLAKE2b-256 | 68ddd7be86b63e65b506a8f66da9d0270b956ea58d3719f0e9f7449ea171f492 |
Hashes for flaco-0.6.0rc3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bbaef92b649cd5441c000210d01693eb553119c081347b6c93914af3e2286d0 |
|
MD5 | fefadb274077256b56d2e1d376f6585a |
|
BLAKE2b-256 | c0ce3e353a91fca678c30aa9f540c7a3c38c8f6e3b04e8ee43e9062767dc9ef1 |
Hashes for flaco-0.6.0rc3-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c530d076b3a5d6e293f567a3b7fc5ade4fbf839fdac14aff2d9d835448351b49 |
|
MD5 | 95ba022ff24bd4aaa9bb38672e36e137 |
|
BLAKE2b-256 | bcc78654d83b3ab425052af0a00157b0e1f482eb06fb0d03e3c3b2cf79216f83 |
Hashes for flaco-0.6.0rc3-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eed91e98789a840747e3b8ef632635b03f548ad802f42172d49ddc3e93a7eb56 |
|
MD5 | ca525506bce5e998bc0679560bc0805b |
|
BLAKE2b-256 | 5d371130ed516ba7cb59bbe25549370ccf4ced01b7327616ba9eb8b177b04cd0 |