Streaming operations with pandas.
Project description
pandas_streaming: streaming API over pandas
pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.
from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8") for df in sdf: # process this chunk of data # df is a dataframe print(df)
The module can also stream an existing dataframe.
import pandas df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"), dict(cf=1, cint=1, cstr="1"), dict(cf=3, cint=3, cstr="3")]) from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_df(df) for df in sdf: # process this chunk of data # df is a dataframe print(df)
It contains other helpers to split datasets into train and test with some weird constraints.
Links:
History
current - 2020-08-06 - 0.00Mb
0.0.0 - 2020-08-06 - 0.00Mb
16: Unit tests failing with pandas 1.1.0. (2020-08-06)
15: implements parameter lines, flatten for read_json (2018-11-21)
14: implements fillna (2018-10-29)
13: implement concat for axis=0,1 (2018-10-26)
12: add groupby_streaming (2018-10-26)
11: add method add_column (2018-10-26)
10: plan B to bypass a bug in pandas about read_csv when iterator=True –> closed, pandas has a weird behaviour when names is too small compare to the number of columns (2018-10-26)
9: head is very slow (2018-10-26)
8: fix pandas_streaming for pandas 0.23.1 (2018-07-31)
7: implement read_json (2018-05-17)
6: add pandas_groupby_nan from pyensae (2018-05-17)
5: add random_state parameter to splitting functions (2018-02-04)
2: add method sample, resevoir sampling (2017-11-05)
3: method train_test_split for out-of-memory datasets (2017-10-21)
1: Excited for your project (2017-10-10)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for pandas_streaming-0.2.175-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d66bc4a8565958ba92682cea56a10fa840dba64150d0ba6a4e26ce874661455b |
|
MD5 | d4f14bcd0dedd4c0700ea67bfa1f885a |
|
BLAKE2b-256 | 84410660927512c2d8453d3cbe5a8484ad5da3aaf6bf1da1800341e2e268703e |