toolbox-pyspark 0.10.0

Helper files/functions/classes for generic PySpark processes

toolbox-pyspark

Introduction

The purpose of this package is to provide some helper files/functions/classes for generic PySpark processes.

Key URLs

For reference, these URL's are used:

Type	Source	URL
Git Repo	GitHub	https://github.com/data-science-extensions/toolbox-pyspark
Python Package	PyPI	https://pypi.org/project/toolbox-pyspark
Package Docs	Pages	https://data-science-extensions.com/toolbox-pyspark

Installation

You can install and use this package multiple ways by using pip, pipenv, or poetry.

Using pip:

1. In your terminal, run:

   python3 -m pip install --upgrade pip
   python3 -m pip install toolbox-pyspark
   

2. Or, in your requirements.txt file, add:

   toolbox-pyspark
   

   Then run:

   python3 -m pip install --upgrade pip
   python3 -m pip install --requirement=requirements.txt
   

Using pipenv:

1. Install using environment variables:

   In your Pipfile file, add:

   [[source]]
   url = "https://pypi.org/simple"
   verify_ssl = false
   name = "pypi"
   
   [packages]
   toolbox-pyspark = "*"
   

   Then run:

   python3 -m pip install pipenv
   python3 -m pipenv install --verbose --skip-lock --categories=root index=pypi toolbox-pyspark
   

2. Or, in your requirements.txt file, add:

   toolbox-pyspark
   

   Then run:

   python3 -m run pipenv install --verbose --skip-lock --requirements=requirements.txt
   

3. Or just run this:

   python3 -m pipenv install --verbose --skip-lock toolbox-pyspark
   

Using poetry:

1. In your pyproject.toml file, add:

   [tool.poetry.dependencies]
   toolbox-pyspark = "*"
   

   Then run:

   poetry install
   

2. Or just run this:

   poetry add toolbox-pyspark
   poetry install
   poetry sync
   

Contribution

Contribution is always welcome.

1. First, either fork or branch the main repo.

2. Clone your forked/branched repo.

3. Build your environment:

   1. With pipenv on Windows:

      if (-not (Test-Path .venv)) {mkdir .venv}
      python -m pipenv install --requirements requirements.txt --requirements requirements-dev.txt --skip-lock
      python -m poetry run pre-commit install
      python -m poetry shell
      

   2. With pipenv on Linux:

      mkdir .venv
      python3 -m pipenv install --requirements requirements.txt --requirements requirements-dev.txt --skip-lock
      python3 -m poetry run pre-commit install
      python3 -m poetry shell
      

   3. With poetry on Windows:

      python -m pip install --upgrade pip
      python -m pip install poetry
      python -m poetry init
      python -m poetry add $(cat requirements/root.txt)
      python -m poetry add --group=dev $(cat requirements/dev.txt)
      python -m poetry add --group=test $(cat requirements/test.txt)
      python -m poetry add --group=docs $(cat requirements/docs.txt)
      python -m poetry install
      python -m poetry run pre-commit install
      python -m poetry shell
      

   4. With poetry on Linux:

      python3 -m pip install --upgrade pip
      python3 -m pip install poetry
      python3 -m poetry init
      python3 -m poetry add $(cat requirements/root.txt)
      python3 -m poetry add --group=dev $(cat requirements/dev.txt)
      python3 -m poetry add --group=test $(cat requirements/test.txt)
      python3 -m poetry add --group=docs $(cat requirements/docs.txt)
      python3 -m poetry install
      python3 -m poetry run pre-commit install
      python3 -m poetry shell
      

4. Start contributing.

5. When you're happy with the changes, raise a Pull Request to merge with the main branch again.

Build and Test

To ensure that the package is working as expected, please ensure that:

1. You write your code as per PEP8 requirements.
2. You write a UnitTest for each function/feature you include.
3. The CodeCoverage is 100%.
4. All UnitTests are passing.
5. MyPy is passing 100%.

Testing

• Run them all together

  poetry run make check
  

• Or run them individually:

  • Black

    poetry run make check-black
    

  • PyTests:

    poetry run make ckeck-pytest
    

  • MyPy:

    poetry run make check-mypy