Configurable data pipeline with Pyspark
Project description
Pyspark-config
Pyspark-Config is a Python module for pyspark use with the help of a configuration file, granting access to build distributed data piplines with configurable input and output.
Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
Installation
Dependencies
- Python (>= 3.6)
- Pyspark (>= 2.4.5)
- PyYaml (>= 5.3.1)
- Dataclass (>= 0.0.0)
User installation
A step by step series of examples that tell you how to get a development env running
Say what the step will be
Give the example
And repeat
until finished
End with an example of getting some data out of the system or using it for a little demo
Changelog
See the changelog for a history of notable changes to scikit-learn.
Running the tests
Explain how to run the automated tests for this system
Break down into end to end tests
Explain what these tests test and why
Give an example
And coding style tests
Explain what these tests test and why
Give an example
Deployment
Add additional notes about how to deploy this on a live system
Built With
- Dropwizard - The web framework used
- Maven - Dependency Management
- ROME - Used to generate RSS Feeds
Contributing
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
Versioning
We use SemVer for versioning. For the versions available, see the tags on this repository.
Authors
- Patrizio Guagliardo - Patrizio1301
See also the list of contributors who participated in this project.
License
This project is distributed under the 3-Clause BSD license. - see the LICENSE.md file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pyspark_config-0.0.2.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c64df37db7b49d3791a548a384deb9c42f5bbf9dafb699f1bf0685333f2366b4 |
|
MD5 | 1bf1c0a6b5d48f5f89be35a9b2fc4039 |
|
BLAKE2b-256 | 7ffa9501c37df0a33823d379b570f7e7b706126643d4f5874b4604ed70f76a97 |