Skip to main content

A lightweight framework to build and execute data processing pipelines in pyspark (Apache Spark's python API)

Project description

sparklanes is a lightweight data processing framework for Apache Sparkwritten in Python. It was built with the intention to make buildingcomplex spark processing pipelines simpler, by shifting the focustowards writing data processing code without having to spent much timeon the surrounding application architecture. Data processing pipelines, or lanes, are built by stringing togetherencapsulated processor classes, which allows creation of lane definitionswith an arbitrary processor order, where processors can be easilyremoved, added or swapped. Processing pipelines can be defined using lane configuration YAML files,to then be packaged and submitted to spark using a single command.Alternatively, the same can be achieved manually by using the frameworkAPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparklanes-0.2.4.tar.gz (14.7 kB view details)

Uploaded Source

Built Distributions

sparklanes-0.2.4-py3-none-any.whl (18.8 kB view details)

Uploaded Python 3

sparklanes-0.2.4-py2-none-any.whl (18.8 kB view details)

Uploaded Python 2

File details

Details for the file sparklanes-0.2.4.tar.gz.

File metadata

  • Download URL: sparklanes-0.2.4.tar.gz
  • Upload date:
  • Size: 14.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.5

File hashes

Hashes for sparklanes-0.2.4.tar.gz
Algorithm Hash digest
SHA256 06d5909c9c2ce2e1e66d647b04589b2621532eff43502d415996be86937f7c22
MD5 3b5c64eb1808dfed5b6ea698cedc525c
BLAKE2b-256 61bb8c22201fd293a0415b17cf08037f28fb97da0bf6b57e682b44e848b50613

See more details on using hashes here.

File details

Details for the file sparklanes-0.2.4-py3-none-any.whl.

File metadata

  • Download URL: sparklanes-0.2.4-py3-none-any.whl
  • Upload date:
  • Size: 18.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.5

File hashes

Hashes for sparklanes-0.2.4-py3-none-any.whl
Algorithm Hash digest
SHA256 b23e0a319574e37827c9d70ca745972829c05ba82b4ef4e03fbcee4cf248cb28
MD5 3f35c9ec2ac65c55a274f651afe31e53
BLAKE2b-256 16f86f60e8c998ca5a3b0ac86dbcd8ddb0ad1a41a7056611651a8364c8e664e8

See more details on using hashes here.

File details

Details for the file sparklanes-0.2.4-py2-none-any.whl.

File metadata

  • Download URL: sparklanes-0.2.4-py2-none-any.whl
  • Upload date:
  • Size: 18.8 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.5

File hashes

Hashes for sparklanes-0.2.4-py2-none-any.whl
Algorithm Hash digest
SHA256 791aa71df792d8dd4b316a1ad9fab1b8e1327103b86794ab5f97e6a170be6fbc
MD5 f3f103acb8ed70d391bc89b90b2e8202
BLAKE2b-256 bfedd90ea71cf31f719c3d3aee57362ebc72778b33ff30bf57bf0bd1fdeb2dbe

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page