Streaming operations with pandas.
Project description
README
pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.
from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8") for df in sdf: # process this chunk of data # df is a dataframe print(df)
The module can also stream an existing dataframe.
import pandas df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"), dict(cf=1, cint=1, cstr="1"), dict(cf=3, cint=3, cstr="3")]) from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_df(df) for df in sdf: # process this chunk of data # df is a dataframe print(df)
Links:
History
current - 2018-05-17 - 0.00Mb
6: add pandas_groupby_nan from pyensae (2018-05-17)
0.1.66 - 2018-02-05 - 0.02Mb
5: add random_state parameter to splitting functions (2018-02-04)
2: add method sample, resevoir sampling (2017-11-05)
3: method train_test_split for out-of-memory datasets (2017-10-21)
1: Excited for your project (2017-10-10)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
Close
Hashes for pandas_streaming-0.1.87-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f54c74252f8613c8ffe28811b15d9214cb2dc5367862e846bfce39247a673a9c |
|
MD5 | d023860b96235060bfae4fa6568d10ff |
|
BLAKE2b-256 | 294ab29c53aad11c6d7607d0423ae6f909296c03da20c7cb862355198ac9852f |