Helpers for doing IO with Pandas DataFrames
Project description
df_io
Python helpers for doing IO with Pandas DataFrames
Available methods
read_df
- bzip2/gzip/zstandard compression
- passing parameters to Pandas' readers
- reading from anything, which
smart_open
supports (local files, AWS S3 etc) - most of the available formats, Pandas supports
write_df
This method supports:
- streaming writes
- chunked writes
- bzip2/gzip/zstandard compression
- passing parameters to Pandas' writers
- writing to anything, which
smart_open
supports (local files, AWS S3 etc) - most of the available formats, Pandas supports
Documentation
Examples
Write a Pandas DataFrame (df) to an S3 path in CSV format (the default):
import df_io
df_io.write_df(df, 's3://bucket/dir/mydata.csv')
The same with gzip compression:
df_io.write_df(df, 's3://bucket/dir/mydata.csv.gz')
With zstandard compression using pickle:
df_io.write_df(df, 's3://bucket/dir/mydata.pickle.zstd', fmt='pickle')
Using JSON lines:
df_io.write_df(df, 's3://bucket/dir/mydata.json.gz', fmt='json')
Passing writer parameters:
df_io.write_df(df, 's3://bucket/dir/mydata.json.gz', fmt='json', writer_options={'lines': False})
Chunked write (splitting the df into equally sized parts and creating/writing outputs for them):
df_io.write_df(df, 's3://bucket/dir/mydata.json.gz', fmt='json', chunksize=10000)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
df_io-0.0.12.tar.gz
(6.7 kB
view hashes)
Built Distribution
Close
Hashes for df_io-0.0.12-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2195bc9d1606a763c64c58fe4b48a3a06d9a526e0c25932ba2eb00a73fb1eebe |
|
MD5 | 379757bda3889550571b475a68eaddea |
|
BLAKE2b-256 | 1798300958e6c7c1cbac4997437105a4cd889926dfdcf29af3161ac95a458ba2 |