Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Streaming operations with pandas.

Project description

https://github.com/sdpython/pandas_streaming/blob/master/_doc/sphinxdoc/source/phdoc_static/project_ico.png?raw=true

pandas_streaming: streaming API over pandas

Build status Build Status Windows https://circleci.com/gh/sdpython/pandas_streaming/tree/master.svg?style=svg https://dev.azure.com/xavierdupre3/pandas_streaming/_apis/build/status/sdpython.pandas_streaming https://badge.fury.io/py/pandas_streaming.svg MIT License Requirements Status https://codecov.io/github/sdpython/pandas_streaming/coverage.svg?branch=master GitHub Issues Notebook Coverage https://api.codacy.com/project/badge/Grade/f53b7f4d6a0447aa9ce0c4ad5df659ef Downloads Forks Stars size

pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

The module can also stream an existing dataframe.

import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
                       dict(cf=1, cint=1, cstr="1"),
                       dict(cf=3, cint=3, cstr="3")])

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

It contains other helpers to split datasets into train and test with some weird constraints.

Links:

History

current - 2020-08-06 - 0.00Mb

0.0.0 - 2020-08-06 - 0.00Mb

  • 16: Unit tests failing with pandas 1.1.0. (2020-08-06)
  • 15: implements parameter lines, flatten for read_json (2018-11-21)
  • 14: implements fillna (2018-10-29)
  • 13: implement concat for axis=0,1 (2018-10-26)
  • 12: add groupby_streaming (2018-10-26)
  • 11: add method add_column (2018-10-26)
  • 10: plan B to bypass a bug in pandas about read_csv when iterator=True –> closed, pandas has a weird behaviour when names is too small compare to the number of columns (2018-10-26)
  • 9: head is very slow (2018-10-26)
  • 8: fix pandas_streaming for pandas 0.23.1 (2018-07-31)
  • 7: implement read_json (2018-05-17)
  • 6: add pandas_groupby_nan from pyensae (2018-05-17)
  • 5: add random_state parameter to splitting functions (2018-02-04)
  • 2: add method sample, resevoir sampling (2017-11-05)
  • 3: method train_test_split for out-of-memory datasets (2017-10-21)
  • 1: Excited for your project (2017-10-10)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pandas_streaming, version 0.2.175
Filename, size File type Python version Upload date Hashes
Filename, size pandas_streaming-0.2.175-py3-none-any.whl (33.2 kB) File type Wheel Python version 3.7 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page