Streaming operations with pandas.
Project description
pandas_streaming: streaming API over pandas
pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.
from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8") for df in sdf: # process this chunk of data # df is a dataframe print(df)
The module can also stream an existing dataframe.
import pandas df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"), dict(cf=1, cint=1, cstr="1"), dict(cf=3, cint=3, cstr="3")]) from pandas_streaming.df import StreamingDataFrame sdf = StreamingDataFrame.read_df(df) for df in sdf: # process this chunk of data # df is a dataframe print(df)
It contains other helpers to split datasets into train and test with some weird constraints.
Links:
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pandas_streaming-0.3.239.tar.gz
(33.2 kB
view hashes)
Built Distribution
Close
Hashes for pandas_streaming-0.3.239-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3fbe9c6b1537e1d76c232608eea7f9f10dec0fe0582310e6b16c00a2ab4c250 |
|
MD5 | c68ceac8eeb10dfacbcffde28fb47e44 |
|
BLAKE2b-256 | c01aba56ac1cf03435a8f37258a8aed8681edeb4bcb715bbaa4d8f2be8b5c652 |