No project description provided
Project description
flaco
The easiest and perhaps most memory efficient way to get PostgreSQL data (more flavors to come?)
into pyarrow.Table
, pandas.DataFrame
or Arrow (IPC/Feather) and Parquet files.
Since Arrow supports efficient and even larger-than-memory processing, as with dask, duckdb, or others. Just getting data onto disk is sometimes the hardest part; this aims to make that easier.
API:
flaco.read_sql_to_file
: Read SQL query into Feather or Parquet file.
flaco.read_sql_to_pyarrow
: Read SQL query into a pyarrow table.
NOTE: This is still a WIP. I intend to generalize it more to be useful towards a wider audience. Issues and pull requests welcome!
Example
Line # Mem usage Increment Occurrences Line Contents
=============================================================
122 147.9 MiB 147.9 MiB 1 @profile
123 def memory_profile():
124 147.9 MiB 0.0 MiB 1 stmt = "select * from test_table"
125
126 # Read SQL to file
127 150.3 MiB 2.4 MiB 1 flaco.read_sql_to_file(DB_URI, stmt, 'result.feather', flaco.FileFormat.Feather)
128 150.3 MiB 0.0 MiB 1 with pa.memory_map('result.feather', 'rb') as source:
129 150.3 MiB 0.0 MiB 1 table1 = pa.ipc.open_file(source).read_all()
130 408.1 MiB 257.8 MiB 1 table1_df1 = table1.to_pandas()
131
132 # Read SQL to pyarrow.Table
133 504.3 MiB 96.2 MiB 1 table2 = flaco.read_sql_to_pyarrow(DB_URI, stmt)
134 644.1 MiB 139.8 MiB 1 table2_df = table2.to_pandas()
135
136 # Pandas
137 648.8 MiB 4.7 MiB 1 engine = create_engine(DB_URI)
138 1335.4 MiB 686.6 MiB 1 _pandas_df = pd.read_sql(stmt, engine)
License
Why did you choose such lax licensing? Could you change to a copy left license, please?
...just kidding, no one would ask that. This is dual licensed under Unlicense or MIT, at your discretion.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for flaco-0.6.0rc4-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c075f13cca843717da2e07afcd58b0fba31c8e0d9ae34bb1005494ef25b8ee1e |
|
MD5 | 14d313b8821d72c010c70e4485cbef8d |
|
BLAKE2b-256 | c451ae65698b87c98dbb3218375dc747f55fb75e41ea0ae14f1fdf1e38498249 |
Hashes for flaco-0.6.0rc4-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 505ae8c9d5621542b538db496576ae8afd32d6e7b41960c95eb9431a2a845091 |
|
MD5 | 463a6fa4b0ce7f345ddcc73481f5c14c |
|
BLAKE2b-256 | 818114c6996fe90aa4912e8fd86e17c7675c20aa062922cdea669d09c206fb25 |
Hashes for flaco-0.6.0rc4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f108d1b3d1a7548305e1fffa5afa81a6301e38623b61b37a028689c0b16710d |
|
MD5 | 7e8395a8a784ff8f2597f37193cb4480 |
|
BLAKE2b-256 | dcf56d5337e3bae1df1d86d87ae2b44176f3fd123f864b966c68a5d6ab8739d0 |
Hashes for flaco-0.6.0rc4-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7139830d32e442ba9ff3557c3159f9111c45727b63c0e56fb46b9221b407e70a |
|
MD5 | f81b7c5c0336ee63a19ef4a73dd38c60 |
|
BLAKE2b-256 | 9bcd17c01ff52e339184157a9ae9fe80563dac0d067650dd2875d9ec5b5dff7d |
Hashes for flaco-0.6.0rc4-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84f5770dab42a761b9ec99359e807b0be0face570b78ecfe48940d66200132ad |
|
MD5 | 3ed95bcf71b1990013a90773d21a818a |
|
BLAKE2b-256 | 01e7fd477f01da705df496ffd8756c6f92ad37de14ebff3eb58c2c8d161e7d7d |
Hashes for flaco-0.6.0rc4-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 068f4a0bf1cf9b15334048dbff0ca211401b07ffd77fc1cda9e7f67d3096b54c |
|
MD5 | b3d1bad19bf481557c36ebee2ee7ca1b |
|
BLAKE2b-256 | 34281b35ed77b1b3bcd7754fd782541db913565edf8abfc8bb86ced381917f39 |
Hashes for flaco-0.6.0rc4-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b74bd9e226a613831f9b54a5f276e0c3ef2994ef62fd9fbe2e620cd5e282deec |
|
MD5 | 7ca1d0bcebef4ca61a89e854a43fa7b1 |
|
BLAKE2b-256 | a51a7efd38aaa8224d383b63390470e55ee331c5af15cc44bd835b0b4a605efb |
Hashes for flaco-0.6.0rc4-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 854d10de2ef7e807521a23e261b0a5d694496f7bdac01c21be154914a850703d |
|
MD5 | 7f2ffec42603ba984f230b9f5b48ac18 |
|
BLAKE2b-256 | 2238bacaa1d61b2a64b85cd853dc5356533adb5e56c9970cab9a7b134dd19c95 |
Hashes for flaco-0.6.0rc4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13bc6e65f758a41d89ff5948348350e8c38a8a1eb86d318464f8d1fef819d40e |
|
MD5 | 2e0f1221898978ccedee37b22cacfb02 |
|
BLAKE2b-256 | 125258511cfdea646f124eeb1c8135d0562daf7bfca7e2d61d9598121c8b8570 |
Hashes for flaco-0.6.0rc4-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f61b52b22ace4a054329cc59c245347fe57b66fff70a82b6f5f72ed6488798a |
|
MD5 | 9e0750a02cee522d64fb20a0a4c0d3da |
|
BLAKE2b-256 | 4b01a3b33adcc5f8e9c4908b7d8e1fd0ca7da428c9549046638a4599ac93ffde |
Hashes for flaco-0.6.0rc4-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b874613a6462a4e0d56ba4f3714d99186e7325727f73977c30102bedfb543914 |
|
MD5 | bdcb7c49a27bac827c4ad6316ad649b3 |
|
BLAKE2b-256 | c826e4bf563087c82e0daa9015b2450bae611db2ead608b1c6a231095c0de497 |
Hashes for flaco-0.6.0rc4-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6effe6dbc74831e4c9360b2a008ba2320bb2745705bbb8c24d623d2f6cc12c84 |
|
MD5 | 4281765d45b430c4062007db63c236d9 |
|
BLAKE2b-256 | e6dc02f1bf23da8f75551ec2a29b07e52f58f8877f258114f3d6d604f8bba05a |
Hashes for flaco-0.6.0rc4-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | deef4f8074387e973a2764c2386c07473ad3775a7ebd03ead4c19d8a530dae02 |
|
MD5 | 3a1f3c53d7680f7d2357070edfbe3e41 |
|
BLAKE2b-256 | 21424fb0c7ae6e90d522b557f494884a1d4e3815660713f4c369d3ea34b320c0 |
Hashes for flaco-0.6.0rc4-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0c5fb2440e07dc3617be7276c25de9b57e4d88c94d46e65c737726635f613fa |
|
MD5 | 0e6c6589e116e72b48f791f80427248f |
|
BLAKE2b-256 | bfc181831dda21e87d8949b16d315dbf5e2fc865c16b86659163115e0a2f1f37 |
Hashes for flaco-0.6.0rc4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 438d832e7fef8bcdc394811c3201226aba17178817d8478c79085f37b25c1575 |
|
MD5 | d51cd1d83b18e2b5f3825bca0f5c7d55 |
|
BLAKE2b-256 | 62f511f7dee72b6e74ea1aa26ba1e6ffd250178aa8c4e6bf28fc72f05301703c |
Hashes for flaco-0.6.0rc4-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a5caa0052f428232850fe0686e843297283b413ccd5137227d6a9cd32e434e2 |
|
MD5 | b0b224638c0f72cc4809ebd8231df7ed |
|
BLAKE2b-256 | 1d415bec994a261e879cb6a7fb2dada16f54d1779808d538e6f0465e44001e38 |
Hashes for flaco-0.6.0rc4-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b74d4131a33252799d66227115a02d28bfda404ee67daa686cf42e72d430a31 |
|
MD5 | 49d8d7eb2b1184ba9f4fd367939a64e5 |
|
BLAKE2b-256 | 87dd425c3b12623df45981b3fe2074b630c38a0911b8ba68885e05f6f440f2fb |
Hashes for flaco-0.6.0rc4-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7e89aad43bb6ab993397bee8d7335356de396c119ac1c4577a0f769129e33e7 |
|
MD5 | 8baa3e41bb022584c77587b0f8c93906 |
|
BLAKE2b-256 | 2173e61e32047136ccaba917b7fbaf19d7f2e1b0d38d687f834a3ecec4a794bb |
Hashes for flaco-0.6.0rc4-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 036fdab70aff581dc52da5cd973e57233b6b949dd86d6a480f1ca5e3a02c4d6e |
|
MD5 | dacc85e1de949028deddca94f6541621 |
|
BLAKE2b-256 | 833594bb7eb0c49d1fc38d1b29a7678e30d8194aa1826bef8427e8d99ce57492 |
Hashes for flaco-0.6.0rc4-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae3202899fb3080351106696705ce156f6c9be9fb42f92836279d9106ff89f42 |
|
MD5 | fada69d21b0d871926b8c11626975625 |
|
BLAKE2b-256 | 6e5b8b42af11bf114d5ed6416cd39e710dddc3f490503230be575fa30d873af1 |
Hashes for flaco-0.6.0rc4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf3aec90071d4072596ced80602eaa517ac4140fe465f9368c22f92a62f4421b |
|
MD5 | 9f1c525c96ff57a7384a9d95a720ea53 |
|
BLAKE2b-256 | 3412de6c115edc3697046f0aa6b3b3db914cb52c03767586929a4c426a064b6d |
Hashes for flaco-0.6.0rc4-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56dedda77880748e4b0bc7ad53e73423c527e024a8a52c6fd3fd7e36d8dd3bd9 |
|
MD5 | 313a3db772ab14e1a3803d68ba124c73 |
|
BLAKE2b-256 | 4346448e5b9346a84f1abb0edb36d309b21576d557dcbec54a611bbeee7dabc9 |
Hashes for flaco-0.6.0rc4-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02c4e7c88f3235cb7f9fec2437256773028a6167d814682ba5db9d8b49f34003 |
|
MD5 | 51169048a31d9fd3d4a845a242272dee |
|
BLAKE2b-256 | 8483bdd674fe5d10cfb236de02bab434fea45bc9271b69fc305b567c8c8590fd |