No project description provided
Project description
flaco
The easiest and perhaps most memory efficient way to get PostgreSQL data (more flavors to come?)
into pyarrow.Table
, pandas.DataFrame
or Arrow (IPC/Feather) and Parquet files.
Since Arrow supports efficient and even larger-than-memory processing, as with dask, duckdb, or others. Just getting data onto disk is sometimes the hardest part; this aims to make that easier.
API:
flaco.read_sql_to_file
: Read SQL query into Feather or Parquet file.
flaco.read_sql_to_pyarrow
: Read SQL query into a pyarrow table.
NOTE: This is still a WIP. I intend to generalize it more to be useful towards a wider audience. Issues and pull requests welcome!
Example
Line # Mem usage Increment Occurrences Line Contents
=============================================================
122 147.9 MiB 147.9 MiB 1 @profile
123 def memory_profile():
124 147.9 MiB 0.0 MiB 1 stmt = "select * from test_table"
125
126 # Read SQL to file
127 150.3 MiB 2.4 MiB 1 flaco.read_sql_to_file(DB_URI, stmt, 'result.feather', flaco.FileFormat.Feather)
128 150.3 MiB 0.0 MiB 1 with pa.memory_map('result.feather', 'rb') as source:
129 150.3 MiB 0.0 MiB 1 table1 = pa.ipc.open_file(source).read_all()
130 408.1 MiB 257.8 MiB 1 table1_df1 = table1.to_pandas()
131
132 # Read SQL to pyarrow.Table
133 504.3 MiB 96.2 MiB 1 table2 = flaco.read_sql_to_pyarrow(DB_URI, stmt)
134 644.1 MiB 139.8 MiB 1 table2_df = table2.to_pandas()
135
136 # Pandas
137 648.8 MiB 4.7 MiB 1 engine = create_engine(DB_URI)
138 1335.4 MiB 686.6 MiB 1 _pandas_df = pd.read_sql(stmt, engine)
License
Why did you choose such lax licensing? Could you change to a copy left license, please?
...just kidding, no one would ask that. This is dual licensed under Unlicense or MIT, at your discretion.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for flaco-0.6.0-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dac9ee51b6b9cab2c4e41571cfe83cb2610a556db83f33e89bfb5ffba358385f |
|
MD5 | fe5e95612e9b1946e209de1139fdb9d5 |
|
BLAKE2b-256 | 14b41b9bdbdeb98e526c72aa40eb6f8eff385f58e26d097be5275823b4c4aec7 |
Hashes for flaco-0.6.0-cp311-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 874881a3c0d79ca0073569c1668e8b5d0f92d3dac6ba87c360791a4615da6d5a |
|
MD5 | 35fa5f70878a6db9c9811d81a925e54e |
|
BLAKE2b-256 | 8d8cdb0ab1f48585a304b59b97841bebddf36801b579292589c46ffd1978919e |
Hashes for flaco-0.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec18ab6c012fe76ca0a402159720e926a16a271c23099ad5ee56c88e172628e0 |
|
MD5 | d4450d54ddcbf7ae6f57be7cc8e866d9 |
|
BLAKE2b-256 | 410a6e19c7d3ead29161a7f81973dfd5548ba293139dbfe20292575653f6c1b2 |
Hashes for flaco-0.6.0-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f64888287f4a62584a0976973ee7e9a2dc2f64a9ca7e6eae64234fcbafb8156f |
|
MD5 | 272bb28bbe48e8a1dedfd9e7ad4d2008 |
|
BLAKE2b-256 | 1fd7dceca57812ccbda263ec8ace266e253a0ba4c33604e08b7f07e081c44288 |
Hashes for flaco-0.6.0-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a79a725405a7f61c009955fc8f8d7f7b64527bf9fb0b406cb3332a5078438d67 |
|
MD5 | ab033cd8c6156663bece272c7dc5c3e1 |
|
BLAKE2b-256 | bcad32f62b1d8698709a499415bafcbaf30343cf9bccc0021a3b962e394ba1c7 |
Hashes for flaco-0.6.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a9dbcd83641d67237c15d51be51e521ec4302782bc0b34621a880bdf8798feb |
|
MD5 | 9319053607f3ffa02f960bb8420e6829 |
|
BLAKE2b-256 | 7550ab9d093b884c81fabdfb9caf3e59500acc7ec79f8b5325dcf097c7163b45 |
Hashes for flaco-0.6.0-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2890386ca5abbd3afec5c11205ca604ec1b9e6f43fcde5b6c6f920b15724cf57 |
|
MD5 | 481bc68c220b7e93dd567b3eda36d88e |
|
BLAKE2b-256 | 92c0e07a8321aeca291bdf290273e936160eb58d729a57f722e7e6d49ebfb0a1 |
Hashes for flaco-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b84bada9dfdc29047740bfe7395b0562c8e22ca8f7582ffff87ce3f9412e444 |
|
MD5 | d0815273bb86ad647b670f4056568bcf |
|
BLAKE2b-256 | 3a6cd7e16c58f0775aff3bc7d28fbb8df8356cd574de8d6bc579009ba6c9b281 |
Hashes for flaco-0.6.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 440a3e455c322fb084e24f4e86b2d242a23289e49a9833521a8b86c627575808 |
|
MD5 | 24c9e4427ffe32cd4f9aef9d1447d8a8 |
|
BLAKE2b-256 | cb9b5b1ee60e490a7ae1147787cc7e6e6d1fca8913b6a3a670b71ed79b09a130 |
Hashes for flaco-0.6.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3984753ab1168a514f66e76b28d4f22bd6eaada6272a62e59e5135081b3eacc |
|
MD5 | 777258a02f33f45c8fbaaf2f20ed2683 |
|
BLAKE2b-256 | 8141ea75861dfe01cfdb0633d8e07bb34aae33b1c9b2ffafe28c9e77719fe5f7 |
Hashes for flaco-0.6.0-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e0963ea257cc47526fcad08c49cf9e770ba30d2e4ae4c42ca4d1ff19c90f87e |
|
MD5 | 3c5a3d03fc15fa4e468816b53222a8d9 |
|
BLAKE2b-256 | 7f4b06af72133911ca7ee984312ab601b3e716d346321cf13c7b4026c2f344ca |
Hashes for flaco-0.6.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d46a5672d31569264be2e9f3452fd1a952c58310bf0516e7eaeb640dc620b35e |
|
MD5 | dbe542d2992b85ec57f47376a286af92 |
|
BLAKE2b-256 | 2ae0435b739bdc66c6eae095887787984515231c21cf496700e3f2bbb57bbb7a |
Hashes for flaco-0.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f682a9a09b6fc276480f5dbe6965843b62c931efbd07f48fbc1f26809ac6ae23 |
|
MD5 | 09c09c7c4820095b500a32f9596ac3cb |
|
BLAKE2b-256 | be351b6319aff938449cb77c2678aa6ccf87952c610a220b51ac992561fda905 |
Hashes for flaco-0.6.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4016a5905a81e5d56719682b2081a7085244174947285fa1d1b306d10b9e4d6e |
|
MD5 | c6f2ef29b09e8fbffc95fc1f86d8c3de |
|
BLAKE2b-256 | 45425d9a724a6defcd6108c20e061c315f95b7f5ad0a07168abfa9a607ed87c8 |
Hashes for flaco-0.6.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4b5b44c91d69de31fd2565b1f8fcd2aff993b99ac27c3e196845ee1a02a1ffbe |
|
MD5 | 6ba8b51cc559f82fbc064703bb84e10d |
|
BLAKE2b-256 | 4b210ec21ea37fee83822185876e16df41843c651603a9bed5a9c46bd26f587c |
Hashes for flaco-0.6.0-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84e81696f02477f389085b531844658276afd5af9217a5daed94e208525d5c2d |
|
MD5 | f2926967e5e0d60898f3bcd96b0da781 |
|
BLAKE2b-256 | b9a07d85836115f4896762ee0d67bc03f5c00c55b5f34adc709a9a787b34e49c |
Hashes for flaco-0.6.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49025f58f4013313b996c6de003e6430941e6f7913b30aa4387607f553f47aa8 |
|
MD5 | 046ec0d19256ceeddf2a46fdfde2f3d7 |
|
BLAKE2b-256 | 32fb887eb81acde3ce0f87cda9373a412c7ed36382deb8b96fb306a45b05210a |
Hashes for flaco-0.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c22aa901e766db1abb5d37028e55125536ab1893aebb6280eca563ba849a621 |
|
MD5 | 95a760e509f16e8dd1a6ee4ea3f7b6f8 |
|
BLAKE2b-256 | a4bffffc7afe93c5f128d2aa90296ec2b35a8b27c5815ec28fbddf129aa3d42b |
Hashes for flaco-0.6.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0cd81c916e923564c431ecc258a62f03ec5922e86e9b1d5e6172dfeb66463e1f |
|
MD5 | 1ae29bfeb048e3af9c4afd31b6188804 |
|
BLAKE2b-256 | 7441606e218aeb4d7f24e351bc6658964e75ad06f471a986bb4ed1995571ec02 |
Hashes for flaco-0.6.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4f9c79a9f1cd386656728826382bfdbadb5fe15834c24115748d3943cc82ca6 |
|
MD5 | 81c7e2d4e6597607a69cf8c1939f5590 |
|
BLAKE2b-256 | 697c217b0ea3a71d7c49dfb749afd79187484b7c75b5e86ab4bc2414dcbba581 |
Hashes for flaco-0.6.0-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9c95c9e9c62aa04ce3fb894864b9aade694a72ffe1e991fd66ba422f6ff285e |
|
MD5 | 02540d44996ee4ab555de1a24d71208d |
|
BLAKE2b-256 | 3afa5f80c8a6ab0b407c73a9232035b81b98962995b1f95559482daf5fd70b9a |
Hashes for flaco-0.6.0-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fc7fd7f3185b4e3aebd9a76745b93ff48ebc3eaee1197f21b00db3bebe61a25 |
|
MD5 | 7438a8475cf5f0811b426787b94b92d5 |
|
BLAKE2b-256 | 13a4336d27f84de83f3c322fb8db520145320f251b581a3d28ea1ade494a75a6 |
Hashes for flaco-0.6.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2408e51f91283dd823bc82168bbeaffa218d4e2fbd3b06f4bb315eaa7116f4c |
|
MD5 | 8f8f7261f64b9450005a8fd6fb846abb |
|
BLAKE2b-256 | c04962704438b1c4dfceddd6fe32e6d1711e70a9c879c89518d2afb6d2738d63 |
Hashes for flaco-0.6.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 19c8df4046354257a0cc06fd7500b3437f645ce859154823c2e160e2b4523843 |
|
MD5 | b7a20b0a46b41e2eb1ebc2dc1c819ec1 |
|
BLAKE2b-256 | 45ec824ed7c363256603b5ce5db93a9fee6b930d46cc8866bc69edb5321bbb35 |
Hashes for flaco-0.6.0-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 116f902c85e904c657d93e171acc869584898a780e1b26e9c58578d99324ba20 |
|
MD5 | 5cd3a8e04263997375be53f72b836987 |
|
BLAKE2b-256 | e11bd24c699ad478473fac7aa9c1b06cf2911f381d53f11d5724311d198e5024 |