Skip to main content

Streaming operations with pandas.

Project description


Build status Build Status Windows MIT License Requirements Status GitHub Issues Waffle Notebook Coverage

pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")

for df in sdf:
    # process this chunk of data
    # df is a dataframe

The module can also stream an existing dataframe.

import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
                       dict(cf=1, cint=1, cstr="1"),
                       dict(cf=3, cint=3, cstr="3")])

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)

for df in sdf:
    # process this chunk of data
    # df is a dataframe



current - 2018-05-17 - 0.00Mb

  • 6: add pandas_groupby_nan from pyensae (2018-05-17)

0.1.66 - 2018-02-05 - 0.02Mb

  • 5: add random_state parameter to splitting functions (2018-02-04)
  • 2: add method sample, resevoir sampling (2017-11-05)
  • 3: method train_test_split for out-of-memory datasets (2017-10-21)
  • 1: Excited for your project (2017-10-10)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pandas_streaming, version 0.1.87
Filename, size File type Python version Upload date Hashes
Filename, size pandas_streaming-0.1.87-py3-none-any.whl (25.5 kB) File type Wheel Python version 3.6 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page