sat-automations 0.26.5

Contains a collection celery based apps

SAT Automations

This repository contains a collection of Django apps intended to run automation tasks that support the operations of SAT.

Environment Variables

• PEOPLESOFT_PROXY_URL
• CLEARANCE_SERVICE_URL
• AUTH_SERVICE_URL
• CCURE_SERVER
• CCURE_PORT
• CCURE_USERNAME
• CCURE_PASSWORD
• GOLD_DB_USERNAME
• GOLD_DB_PASSWORD
• GOLD_DB_DSN
• FEED_SERVER
• FEED_PORT
• FEED_USERNAME
• FEED_PASSWORD
• SENDGRID_API_KEY
• IMAGE_VOLUME_BASE_PATH
• GRAVITY_FORMS_CONSUMER_KEY
• GRAVITY_FORMS_CONSUMER_SECRET
• GRAVITY_FORMS_BASE_URL

Installation

Usage

Add the automations you want to run in your Django APPS

ACS Journal Automation

An automation that runs hourly to pull in Journal data from the CCURE database. This is now and should always be a read-only operation.

Tasks

Process Access Control Journal Task

This task runs hourly and pulls in journal data from the CCURE database. It will pull in all journal data starting from the last record it finds in the local database. It will then process the journal data up to the time that the process was started.

Demo Automation

This is exactly what it sounds like. It is a demo automation that can be used to test the automation framework.

Disablement Automation

The disablement automation is designed to run daily and disable users in CCURE that are no longer active in the system. If a Student or a Faculty/Staff member is no longer active in the system, they should be disabled in CCURE. The default rules are that when a student is determined inactive they will be disabled in CCURE 180 days from the day they were marked as inactive in Peoplesoft. For Faculty/Staff, they will be disabled in CCURE 30 days from the day they were marked as inactive in Peoplesoft.

Tasks

Populate Tracker Task

This task runs daily. It reconciles the CCURE list of users with the list of users from Peoplesoft. If a user comes back as inactive in Peoplesoft, they will have a disable_on date set in the tracker. This date is used to determine when the user should be disabled in CCURE.

Process Tracker Task

This task runs daily. It will disable users in CCURE that:

1. Are marked as INACTIVE
2. Are not already disabled in CCURE.
3. And have a disable_on date that is less than or equal to the current date.

This task can also be triggered as an Action from the Admin interface. In which case it will process the user that have been selected. It does not do any sanity checks on the users selected, so it is important to only select users that should be disabled.

Send Emails Task

This task runs daily. It sends emails to users that are marked for disablement.

Our current process is to send emails to the user and their supervisor (if they have one).

Emails are sent on the following schedule:

1. The day the user is marked for disablement.
2. 7 days before the user is disabled.

Gold Feed Automation

The gold feed automation is designed to run daily and update the UserFeed table. Future work will add the information directly to CCURE via the API.

Housing Automation

The housing automation is designed to run daily and process housing assignments for students.

Tasks

Housing Populate Tracker Task

This task runs daily and pulls in housing assignments and housing revocations from the ps_nc_his_bed_proc view in Peoplesoft.

Housing Process Move Ins Task

Runs daily in the morning and processes move ins for students.

Housing Process Move Outs Task

Runs daily in the afternoon and processes move outs for students.

Image Update Automation

An automation that runs daily to pull images from the GOLD database. Currently, this automation just stores the images, but future work will include adding/updating the image for a user in CCURE.

Matches are performed based on campus_id.

Tasks

Daily Image Update Task

Checks gold for images added over the past 1 day, and stores the image on an NFS filesystem. Currently this is mounted at /app/images.

All Images Task

Should be run once to populate the image store with all images from the GOLD database. CAUTION: This task will take a long time to run and should only be run once.

Affiliate Automation

This automation tracks details of OneCard requests. Stores data in tables for Sponsor, Affiliate, and CardRequest. Data is gathered and updated on a set schedule.

Tasks

Get Form Entries Task

Gathers all card request forms submitted over a given time interval

Save Card Requests Task

Processes the data from the gathered card request forms and persists the data in Sponsor, Affiliate, and CardRequest tables

Manage Automations

This app is currently used to manage the service accounts in the authentication process for automations. Generally this would be the place to put models and or processes that are required to manage the automations themselves.

API

The api app is used to provide a central location for the API endpoints that are used to interact with the automations.

Development

Setup

Ensure you are in a virtual environment with Python 3.9.6 or higher.

> make setup

Ensure pre-commit is installed.

> pre-commit install

Add dependencies

Updating Requirements

This project uses pip-tools to manage requirements. To update the requirements add your requirement to the pyproject.toml file.

For dependencies required to run the app in production, add them to the pyproject.toml file under the [project] section.

[project]
...
dependencies = [
    "fastapi>=0.95.1, <1.0.0",
    "pyjwt>=2.6.0, <3.0.0",
    "...",
    "<YOUR NEW REQUIREMENT HERE>",
    "...",
]

For developer dependencies required or nice to have for development, add them to the pyproject.toml file under the [project.optional-dependencies] section.

[project.optional-dependencies]
dev = [
    "pytest>=6.2.5, <7.0.0",
    "...",
    "<YOUR NEW DEV REQUIREMENT HERE>",
    "...",
]

When you have added the dependency run:

$> make update-requirements

Create an automation app

To create an automation app execute the following command:

NOTE: Using test_app for example, substitute the app name you desire.

$> python startapp.py test_app

This will create a new app in the sat_automations directory. You will need to edit the apps.py file and add sat_automations to the name.

The below boilerplate should also be in place.

Example:

from logging import DEBUG

from django.apps import AppConfig
from django.conf import settings
from sat.logs import SATLogger

if settings.DEBUG:
    logger = SATLogger(__name__, level=DEBUG)
else:
    logger = SATLogger(__name__)
...
class Automation(AppConfig):
    name = "sat_automations.test_app"

Build and Publish during development

On your branch, you can build and publish by updating the version in pyproject.toml For example, if the version for your feature is say 0.1.0, you would update the version in pyproject.toml to 0.1.0.

Rules:

• Major features like adding a new app or breaking changes to an existing app require a major version change.
• New feature work to an existing app that will not break existing APIs requires a minor version change.
• Bug fixes and other minor changes require a patch version change.

Large work that should be tested on staging before merging to main should be done in a feature branch, and use a release candidate version. For example: 1.23.4rc1. This will allow you to test your work on staging before merging to main.

In django-automation, you would use the just pushed version in it's pyproject.toml file.

CI/CD

When pushing or merging to the main branch, a new build will automatically be generated and published to our PyPI at https://pypi.ehps.ncsu.edu. This will be based on the updated pyproject.toml, so be sure to update it before pushing or merging main. This will also create a new release on Github which can be found at https://github.ncsu.edu/SAT/sat-automations/releases.

Build

If you are testing before merging to main, you can build the package locally with the following command:

> git checkout <your-branch>
> git pull
... update the version in pyproject.toml with release candidate version ...
> flit build

Publish

Currently, this is handled via the CI/CD pipeline, but if you need to do it manually, you can do so with the following command:

As long as your PyPI credentials are set up correctly, you can publish to PyPI with the following command:

> flit publish

Testing

Testing requires a running postgres installation in order to accommodate the triggers and indexes used in the acs_journal app.

Before you run tests locally you will need to make sure it is up and running.

> make run-db
...
> pytest