ssb-spark-tools 0.0.9

A collection of data processing Spark functions for the use in Statistics Norway

SSB Spark Tools

A collection of data processing Spark functions for the use in Statistics Norway (SSB)

The SSB Spark Tools Library is a colection of Data processing functions for the use in Data processing in Statistics Norway

Installation

pip install ssb_spark_tools

Development setup

Run make help to see common development commands.

install-build-tools            Install required tools for build/dev
build                          Build dist
test                           Run tests
clean                          Clean all build artifacts
release-validate               Validate that a distribution will render properly on PyPI
release-test                   Release a new version, uploading it to PyPI Test
release                        Release a new version, uploading it to PyPI
bump-version-patch             Bump patch version, e.g. 0.0.1 -> 0.0.2
bump-version-minor             Bump minor version, e.g. 0.0.1 -> 0.1.0

Refer to the Makefile to see details about the different tasks.

Testing

Run tests for all python distributions using

make test

This will require that your dev machine has the required python distributions installed locally. (You can install python distributions using pyenv.)

Releasing

Prerequisites: You will need to register accounts on PyPI and TestPyPI.

Before releasing, make sure you're working on a "new" version number. You can bump the version using the bumpversion tool.

Also, make sure to update release notes.

To release and publish a new version to PyPI:

make release-validate

This will run tests, build distribution packages and perform some rudimentary PyPI compliancy checking.

For a dress rehearsal, you can do a test release to the TestPyPI index. TestPyPI is very useful, as you can try all the steps of publishing a package without any consequences if you mess up. Read more about TestPyPI here.

make release-test

To perform the actual release, run:

make release

You should see the new release appearing here (it might take a couple of minutes for the index to update).

Release History

• 0.0.1
  • Initial version with functions as in use on initiaition

Meta

Statistics Norway – https://github.com/statisticsnorway

Distributed under the MIT license. See LICENSE for more information.

[https://github.com/statisticsnorway/ssb-pseudonymization-py]

Contributing

1. Fork it (https://github.com/statisticsnorway/ssb-pseudonymization-py/fork)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request