No project description provided
Project description
flaco
The easiest and perhaps most memory efficient way to get PostgreSQL data (more flavors to come?) into Arrow (IPC/Feather) or Parquet files.
If you're trying to load data directly into Pandas then you may find that evan a 'real' 100MB can cause bloat upwards of 1GB. Expanding this can cause significant bottle necks in processing data efficiently.
Since Arrow supports efficient and even larger-than-memory processing, as with dask, duckdb, or others. Just getting data onto disk is sometimes the hardest part; this aims to make that easier.
NOTE: This is still a WIP, and is purpose built for my needs. I intend to generalize it more to be useful towards a wider audience. Issues and pull requests welcome!
Example
from flaco import read_sql_to_file, FileFormat
uri = "postgresql://postgres:postgres@localhost:5432/postgres"
stmt = "select * from my_big_table"
read_sql_to_file(uri, stmt, 'output.data', FileFormat.Parquet)
# Then with pandas...
import pandas as pd
df = pd.read_parquet('output.data')
# pyarrow... (memory mapped file, where potentially larger than memory)
import pyarrow as pa
with pa.memory_map('output.data', 'rb') as source:
table = pa.ipc.open_file(source).read_all() # mmap pyarrow.Table
# DuckDB...
import duckdb
cur = duckdb.connect()
cur.execute("select * from read_parquet('output.data')")
# Or anything else which works with Arrow and/or Parquet files
License
Why did you choose such lax licensing? Could you change to a copy left license, please?
...just kidding, no one would ask that. This is dual licensed under Unlicense or MIT, at your discretion.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for flaco-0.6.0rc2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7961342ec59958e4d3bda869a7cb8f3f530876571e619d3ab90e262797e3c963 |
|
MD5 | 617e1a4e35cd1f2909086a7bd453af37 |
|
BLAKE2b-256 | 02f4f752441507761fcfaacb80c43bde7aa4ce71d70daddf432afd1c7decea47 |
Hashes for flaco-0.6.0rc2-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66f113c734e1c0b1c6b3f725dbc96c993856c8359f884e1734fecc38d0bd5d1e |
|
MD5 | 529425d4a8c34f5929e6aff1ee175a76 |
|
BLAKE2b-256 | c139a79bb66c9cdaa801b3bd67614cf5b1c2e7bc2a8b64cc8a4f4c35e600b8ff |
Hashes for flaco-0.6.0rc2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85dbc9636224283031e6874e5d27469a2e9c12eb50920eb5facdce49146b7fd4 |
|
MD5 | 94e5925d8dec3f3e63f5ff2bf0205e4e |
|
BLAKE2b-256 | c86bd3ade68555d82a49c4b5078489f89963b1c5d11c19194d6f76564216f251 |
Hashes for flaco-0.6.0rc2-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42201d7c9bd832805248772ae55bf40282800a4dcd136ac6edb33ef428bf209a |
|
MD5 | b644e9d0eac43e0c0ad3e94a92505bcb |
|
BLAKE2b-256 | 651603d0bc6825342e6a716aecf9eaa348e7124eac91511a68d335a592087b29 |
Hashes for flaco-0.6.0rc2-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92ff8a29a6e1bed1b51f4218107c2f911b0c14034958b8938322538e34bacf69 |
|
MD5 | 5592ed4a53d4d7c2ae234c530a02ef84 |
|
BLAKE2b-256 | 469ac0e0cdfc06c4046c59a12d711b3ee75eb77f9c656bf0239366a7ecee7177 |
Hashes for flaco-0.6.0rc2-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01eb0687948da22b607666ac57c99af0d8cd6af36079f9b1a1f0e6f28de81ce6 |
|
MD5 | 56f380302c0421901e056d74bb57c63c |
|
BLAKE2b-256 | c6845875da0d75a0c55cae99b57ff83e247c8c1cbc049971d0748ae433bb23ca |
Hashes for flaco-0.6.0rc2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1bc74ca78a531f9ec1c7db01bfc40ecc8b1ae5f331b74cfe68e236cac86e4d64 |
|
MD5 | cc0935526218f2d25473b1b68bddf89b |
|
BLAKE2b-256 | 9e691370167468b15b153f123e515098ef9e83d35e373ec32544b7d4b9151545 |
Hashes for flaco-0.6.0rc2-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a5984c5ee5962e8bddf5800cc128616cf5e988e01e31ed6366c0a2acc28a7c6 |
|
MD5 | d08a8b2b79e252cc3dd843d3b76df85b |
|
BLAKE2b-256 | aed1747e6792a0d6f81df355406d78223b6c32f9d54e8be13098cec8603afce8 |
Hashes for flaco-0.6.0rc2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbf0d14d77efdb42d83b6b808f0d7213bc04f6dda537939d556ffcd2a1530940 |
|
MD5 | aa06a3aa48e17912a8529cb51e2f5ff0 |
|
BLAKE2b-256 | c80c4ad66205b3781ef3390d7405262c745ec382b6b0c816c3cb478cede59b5e |
Hashes for flaco-0.6.0rc2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c46894ac174f199cb59573cbabfebe4ae8e3a953bd364e7176ae929a76f1c3c9 |
|
MD5 | d2cf3cf94488bab6c705356aaa56a8ee |
|
BLAKE2b-256 | 6e37b597dda375ee8a7db4730ebf872a89ce922c5f7872a612f2fb2ebfa3e01d |
Hashes for flaco-0.6.0rc2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa39c44405aa1a0575b9d4d83e695f2da1075d00fb4e3209f63a4b50d3e89f6c |
|
MD5 | abb6ddcb7e467f339e2082fcad07293e |
|
BLAKE2b-256 | 49787df2bc6b5a50a4baaf75d5946c4c0869eb4286f33d87f9831ac2cae02a2c |
Hashes for flaco-0.6.0rc2-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7bd8038c901bbe0853536ae66652ad3c3a7e17784ab6ad7d46aae35925e90e92 |
|
MD5 | bfa78754b8c96a8cdae95e991a829b75 |
|
BLAKE2b-256 | b9debc4941400d8d567dab458472b58b437441216acb040fd14733e01c9f67f4 |
Hashes for flaco-0.6.0rc2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e5230ec9014ac1bc8916e68ce729e557cb9bad1ad4d79d46fb08aef7d3199d9 |
|
MD5 | 19099ea01ac733908320d065b782ec57 |
|
BLAKE2b-256 | 042de20a8d395f119fd24a900590768ff666aee65cfb67d2b0c8711371d0fa8e |
Hashes for flaco-0.6.0rc2-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f97618858b5cbc3e767aa871d7d6a503d44e07eefd2de200cc70c9c906534115 |
|
MD5 | c5172d3bab374b70903b1ab02c75b734 |
|
BLAKE2b-256 | f5dcc1772fad24ccd3e482fee0d0e399c2aae6c9f7df4aedb0c2f053fe99fe84 |
Hashes for flaco-0.6.0rc2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 745b8cb62c14612ca153edfa8bfe5643bc75270e9348a88374305f723b3758a0 |
|
MD5 | 4aafe3ba1938bc0e491b9e3449b63838 |
|
BLAKE2b-256 | daa4978971b7a16daa0f709f684c932c30318e1ead58377fe6ebf76ef52b7544 |
Hashes for flaco-0.6.0rc2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7dadb64ec1a84d0e8a1192fe7174acd110594573f0e29f281d0af3d6c1267b6d |
|
MD5 | 3ced881632d06771d15a8a99df2a8c50 |
|
BLAKE2b-256 | 7e981eaebb1cdf9388f41e63fb0efc5552e2ed0ae72a52823c7f472255c5ca38 |
Hashes for flaco-0.6.0rc2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c608b824ce0997f4edf9b97bb278f9ae58d8476e1c1d6c9f526b86a26d4a815e |
|
MD5 | 02bf87ef8c702a8a18bba95c9344a540 |
|
BLAKE2b-256 | 30eb8b86a14493cb7dd3147c0d4baeeea14b49d496f3caeb4998ea317c7bb3b5 |
Hashes for flaco-0.6.0rc2-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85c181ec24cc63b649b064f4caf11caf82708e92e53f8507454f37d055dde9c1 |
|
MD5 | 905b449642806308934e43dfb940ffac |
|
BLAKE2b-256 | 191eab5003e4a4d22c138ad3166db04c3ff6a44f9804afbb1ba1c2ba62057699 |
Hashes for flaco-0.6.0rc2-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d8e6b2513628bfb0f14f8c1f4a9756a69e52abc6b917e7acc0a123e5fe2b89b |
|
MD5 | 61cac3c9f6bef04a856cfacd887bc2cb |
|
BLAKE2b-256 | bd48aa120ebebda33dc0e1d3d07f1bacc98a6973b71352ae5148001e16f3ada3 |
Hashes for flaco-0.6.0rc2-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d984242df4e0d95a3f6480329cf60328091a78a791be722bd97f099792b8230 |
|
MD5 | 91e2e941acb67e8cc485c88ebe0e9937 |
|
BLAKE2b-256 | 77fd311e448558f3ef57e27ccea1d1a82b34a1af8b34f7a799d493ee7bb9139e |
Hashes for flaco-0.6.0rc2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90d1e48be6db3350f70c61021ca277b8052bd01a94487a998035934d208234d1 |
|
MD5 | 761fb81945854f6fe5f801d14fc48115 |
|
BLAKE2b-256 | 7fafca067c76347b613d3f96ad6b72fddb8792d05547ca62e594ce28bd99ed6f |
Hashes for flaco-0.6.0rc2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5c30e00334d9d72e2c8830301c02dbda203513fef683e1696d4337f4fec0a50 |
|
MD5 | 6e794e6977dbbd2ce87eaa4259b2ddc9 |
|
BLAKE2b-256 | 1bca1ce93a11ffa64a7bfffbb52cac73736e8ada3b7e5b53bd6e2dc364c0c4ad |
Hashes for flaco-0.6.0rc2-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3008f798cc665ecf8b340970adf2177166bbb573898884a2f112eecf7b000d0f |
|
MD5 | 0a8b72a2eca4f8cf2a8da031691e3b43 |
|
BLAKE2b-256 | 04ca2a84d251163a62f283d0f775843f3959a626177ce773e349eb28e43be43b |