Skip to main content

Streaming operations with pandas.

Project description

README

Build status Build Status Windows https://circleci.com/gh/sdpython/pandas_streaming/tree/master.svg?style=svg MIT License Requirements Status https://codecov.io/github/sdpython/pandas_streaming/coverage.svg?branch=master GitHub Issues Waffle Notebook Coverage https://api.codacy.com/project/badge/Grade/f53b7f4d6a0447aa9ce0c4ad5df659ef

pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

The module can also stream an existing dataframe.

import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
                       dict(cf=1, cint=1, cstr="1"),
                       dict(cf=3, cint=3, cstr="3")])

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

Links:

History

current - 2018-05-17 - 0.00Mb

  • 6: add pandas_groupby_nan from pyensae (2018-05-17)

0.1.66 - 2018-02-05 - 0.02Mb

  • 5: add random_state parameter to splitting functions (2018-02-04)
  • 2: add method sample, resevoir sampling (2017-11-05)
  • 3: method train_test_split for out-of-memory datasets (2017-10-21)
  • 1: Excited for your project (2017-10-10)

Project details


Release history Release notifications

This version
History Node

0.1.87

History Node

0.1.66

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pandas_streaming-0.1.87-py3-none-any.whl (25.5 kB) Copy SHA256 hash SHA256 Wheel 3.6 May 17, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page