pandas-streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.
from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")
for df in sdf:
# process this chunk of data
# df is a dataframe
print(df)The module can also stream an existing dataframe.
import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
dict(cf=1, cint=1, cstr="1"),
dict(cf=3, cint=3, cstr="3")])
from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)
for df in sdf:
# process this chunk of data
# df is a dataframe
print(df)It contains other helpers to split datasets into train and test with some weird constraints.
1 maintainer
Metadata
1 maintainer
Release files for pandas-streaming 0.5.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pandas_streaming-0.5.2.tar.gz | 35.6 kB | Details |
Release files / pandas_streaming-0.5.2.tar.gz
| Download URL | pandas_streaming-0.5.2.tar.gz |
|---|---|
| Size | 35.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
18022bed8adcc55db17ed4851be26de3d6fca5888152c674f82df78c5199f9fd
|
|
BLAKE2b-256 checksum How to use checksums |
c755e0096d6755c779a5e9be9f16d2e9b683dac7d2f5d0fbb5934562274ba312
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
