Skip to main content

luigi monkey patch for large s3 file iteration

Project description

Luigi is a Python (2.7, 3.3, 3.4, 3.5) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

Current Luigi has some issues in s3 file iteration in production. When the file is too large, Amazon s3 would tend to disconnect first. Any application should support retry based on current status to aquire the whole part. This project add this part for luigi s3 file iteration feature.

Getting Started

Run pip install luigi_monkey_patch to install the latest stable version from Pypi. Documentation and any issues are hosted here.

Modified Functions

  • luigi.s3.ReadableS3File.close
  • luigi.s3.ReadableS3File.__iter__

Authors

Luigi was built at Spotify. This patch is done by Zenix Huang.

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
luigi_monkey_patch-0.0.1-py2-none-any.whl (4.6 kB) Copy SHA256 hash SHA256 Wheel 2.7 Jan 12, 2017
luigi_monkey_patch-0.0.1.tar.gz (6.6 kB) Copy SHA256 hash SHA256 Source None Jan 12, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page