Skip to main content

luigi monkey patch for large s3 file iteration

Project description

Luigi is a Python (2.7, 3.3, 3.4, 3.5) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

Current Luigi has some issues in s3 file iteration in production. When the file is too large, Amazon s3 would tend to disconnect first. Any application should support retry based on current status to aquire the whole part. This project add this part for luigi s3 file iteration feature.

Getting Started

Run pip install luigi_monkey_patch to install the latest stable version from Pypi. Documentation and any issues are hosted here.

Modified Functions

  • luigi.s3.ReadableS3File.close

  • luigi.s3.ReadableS3File.__iter__

Authors

Luigi was built at Spotify. This patch is done by Zenix Huang.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

luigi_monkey_patch-0.0.1.tar.gz (6.6 kB view details)

Uploaded Source

Built Distribution

luigi_monkey_patch-0.0.1-py2-none-any.whl (4.6 kB view details)

Uploaded Python 2

File details

Details for the file luigi_monkey_patch-0.0.1.tar.gz.

File metadata

File hashes

Hashes for luigi_monkey_patch-0.0.1.tar.gz
Algorithm Hash digest
SHA256 705c77cc8b7b4281916a3c73bda67c414dc32c4fad0a2f7cb94aa30ea356e255
MD5 b2ee25e16a89b03b2f1a38b933b99440
BLAKE2b-256 a606cd4ac1c16974268035b151dc1dec7f31c801cca0f5bd158f228d3bc527b7

See more details on using hashes here.

File details

Details for the file luigi_monkey_patch-0.0.1-py2-none-any.whl.

File metadata

File hashes

Hashes for luigi_monkey_patch-0.0.1-py2-none-any.whl
Algorithm Hash digest
SHA256 11a02cbfe5da836e1616c1c1aa54f6b18f7d87dc00f2a43f0dcc14ae7a7a6e68
MD5 fa8bb6384dbf9d96a0682fcfcadfaac8
BLAKE2b-256 ea392da8ef86c93666d3d21cf8862ba10d3739bfd9d904e16b505b3e45b3cf8f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page