Array (and numpy) API for ONNX
Project description
pandas-streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.
from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")
for df in sdf:
# process this chunk of data
# df is a dataframe
print(df)
The module can also stream an existing dataframe.
import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
dict(cf=1, cint=1, cstr="1"),
dict(cf=3, cint=3, cstr="3")])
from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)
for df in sdf:
# process this chunk of data
# df is a dataframe
print(df)
It contains other helpers to split datasets into train and test with some weird constraints.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pandas_streaming-0.5.1.tar.gz
(34.4 kB
view hashes)
Built Distribution
Close
Hashes for pandas_streaming-0.5.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c5780742c8c6fcf86b7871caa9b4b52a6c10463ed2bd4cebcb6bda9d06c59cc |
|
MD5 | e2e5adbdad3dfb8ba06c6aecb241116f |
|
BLAKE2b-256 | 38bba7e7f01c416200ed03247c688e080d2b50bf697b3819aa368a63ba2c7cc2 |