A package containing Python utilities for use across NYPL
Project description
PythonUtils
This package contains common Python utility classes and functions.
Classes
- Pushing records to Kinesis
- Setting and retrieving a resource in S3
- Decrypting values with KMS
- Encoding and decoding records using a given Avro schema
- Connecting to and querying a MySQL database
- Connecting to and querying a PostgreSQL database using a connection pool
- Connecting to and querying Redshift
Functions
- Reading a YAML config file and putting the contents in os.environ
- Creating a logger in the appropriate format
- Obfuscating a value using bcrypt
Developing locally
In order to use the local version of the package instead of the global version, use a virtual environment. To set up a virtual environment and install all the necessary dependencies, run:
python3 -m venv testenv
source testenv/bin/activate
pip install --upgrade pip
pip install .
pip install '.[tests]'
deactivate && source testenv/bin/activate
Add any new dependencies required by code in the nypl_py_utils
directory to the dependencies
section of pyproject.toml
. Add dependencies only required by code in the tests
directory to the [project.optional-dependencies]
section.
Troubleshooting
If running main.py
in this virtual environment produces the following error:
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: dlsym(0x7f8620446f40, PQsslInUse): symbol not found
then try running:
pip uninstall psycopg
pip install "psycopg[c]"
Git workflow
This repo uses the Main-QA-Production git workflow.
main
has the latest and greatest commits, qa
has what's in our QA environment, and production
has what's in our production environment.
Ideal Workflow
- Cut a feature branch off of
main
- Commit changes to your feature branch
- File a pull request against
main
and assign a reviewer (who must be an owner)- In order for the PR to be accepted, it must pass all unit tests, have no lint issues, and update the CHANGELOG (or contain the
Skip-Changelog
label in GitHub)
- In order for the PR to be accepted, it must pass all unit tests, have no lint issues, and update the CHANGELOG (or contain the
- After the PR is accepted, merge into
main
- Merge
main
>qa
- Deploy app to QA on GitHub and confirm it works
- Merge
qa
>production
- Deploy app to production on GitHub and confirm it works
Deployment
The utils repo is deployed as a PyPI package here and as a Test PyPI package for QA purposes here. In order to be deployed, the version listed in pyproject.toml
must be updated. To deploy to Test PyPI, create a new release in GitHub and tag it qa-vX.X.X
. The GitHub Actions deploy-qa workflow will then build and publish the package. To deploy to production PyPI, create a release and tag it production-vX.X.X
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nypl_py_utils-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03acf08bbab7a26656ce1e89e810a0ab2e0ab0068c1c3c8e74cf8ed8b1ac0161 |
|
MD5 | 9a4e7f2b1ac4f0179af7149ec4a4a4b0 |
|
BLAKE2b-256 | 910796d3415c9612364a7e08be5b0fdcc9183bfff540c8e96a4d6265884aad90 |