A package containing Python utilities for use across NYPL
Project description
PythonUtils
This package contains common Python utility classes and functions.
Classes
- Pushing records to Kinesis
- Setting and retrieving a resource in S3
- Decrypting values with KMS
- Encoding and decoding records using a given Avro schema
- Connecting to and querying a MySQL database
- Connecting to and querying a PostgreSQL database using a connection pool
- Connecting to and querying Redshift
- Making requests to the Oauth2 authenticated APIS such as NYPL Platform API and Sierra
Functions
- Reading a YAML config file and putting the contents in os.environ
- Creating a logger in the appropriate format
- Obfuscating a value using bcrypt
Developing locally
In order to use the local version of the package instead of the global version, use a virtual environment. To set up a virtual environment and install all the necessary dependencies, run:
python3 -m venv testenv
source testenv/bin/activate
pip install --upgrade pip
pip install .
pip install '.[tests]'
deactivate && source testenv/bin/activate
Add any new dependencies required by code in the nypl_py_utils
directory to the dependencies
section of pyproject.toml
. Add dependencies only required by code in the tests
directory to the [project.optional-dependencies]
section.
Troubleshooting
If running main.py
in this virtual environment produces the following error:
ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: dlsym(0x7f8620446f40, PQsslInUse): symbol not found
then try running:
pip uninstall psycopg
pip install "psycopg[c]"
Git workflow
This repo uses the Main-QA-Production git workflow.
main
has the latest and greatest commits, qa
has what's in our QA environment, and production
has what's in our production environment.
Ideal Workflow
- Cut a feature branch off of
main
- Commit changes to your feature branch
- File a pull request against
main
and assign a reviewer (who must be an owner)- In order for the PR to be accepted, it must pass all unit tests, have no lint issues, and update the CHANGELOG (or contain the
Skip-Changelog
label in GitHub)
- In order for the PR to be accepted, it must pass all unit tests, have no lint issues, and update the CHANGELOG (or contain the
- After the PR is accepted, merge into
main
- Merge
main
>qa
- Deploy app to QA on GitHub and confirm it works
- Merge
qa
>production
- Deploy app to production on GitHub and confirm it works
Deployment
The utils repo is deployed as a PyPI package here and as a Test PyPI package for QA purposes here. In order to be deployed, the version listed in pyproject.toml
must be updated. To deploy to Test PyPI, create a new release in GitHub and tag it qa-vX.X.X
. The GitHub Actions deploy-qa workflow will then build and publish the package. To deploy to production PyPI, create a release and tag it production-vX.X.X
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for nypl_py_utils-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24b1c8976401180d760c72a39130b2242c2ab394c6da3c58ffee4b9dbffaca0a |
|
MD5 | 267dee4796d50a5a4198fbba2e370dce |
|
BLAKE2b-256 | 5f58018c6ee72b97aa7feadcc3a55568ee4d5422040aab16f8ee02293a79db53 |