benchalerts 2024.1.10.1

Automated alerting for conbench

benchalerts

A package to facilitate automated alerting based on Conbench data.

Overview

This package is intended to make the following steps easier in CI. Before these steps, it is assumed that an execution environment has performed a run of benchmarks and submitted the results to Conbench.

• Hit the Conbench API, which uses the lookback z-score method, to understand if there were any:
  • errors
  • regressions (with configuration for how these regressions may be detected)
  • improvements (with configuration for how these improvements may be detected)
• Format and submit a summary of these findings to various places (again, with configuration):
  • GitHub Check on a commit with a Markdown summary
  • Comment on a GitHub pull request

In the future, there will be more places to submit alerts/reports/summaries, and more configuration possible.

Currently, the way to configure these workflows in CI is to create and run a Python script that creates and runs an alerting pipeline, like so:

import os

import benchalerts.pipeline_steps as steps
from benchalerts import AlertPipeline


# Pretend we're running on GitHub Actions. Get some environment variables.
# (they may be different for different CI systems)
commit_hash = os.environ["GITHUB_SHA"]
repo = os.environ["GITHUB_REPOSITORY"]
build_url = (
    os.environ["GITHUB_SERVER_URL"]
    + f"/{repo}/actions/runs/"
    + os.environ["GITHUB_RUN_ID"]
)

# Create a pipeline to update a GitHub Check
pipeline = AlertPipeline(
    steps=[
        steps.GetConbenchZComparisonStep(
            commit_hash=commit_hash,
            baseline_run_type=steps.BaselineRunCandidates.fork_point,
        ),
        steps.GitHubCheckStep(
            commit_hash=commit_hash,
            comparison_step_name="GetConbenchZComparisonStep",
            repo=repo
        ),
    ],
    error_handlers=[
        steps.GitHubCheckErrorHandler(
            commit_hash=commit_hash, repo=repo, build_url=build_url
        )
    ],
)

# Run the pipeline
print(pipeline.run_pipeline())

See the docstrings of each pipeline step for more details on how to configure the workflow, including how to set up the required environment variables.

GitHub App Authentication

The preferred method that benchalerts recommends for authenticating and posting to GitHub is to use a machine user called a GitHub App. Using an App will allow you to post using a "bot" entity without taking up a seat in your organization, and will allow you to use the extra features of the Checks API. These features give much more context when analyzing benchmark runs.

Each Conbench server must create its own GitHub App for security reasons. To do so, follow these instructions.

Creating a GitHub App to work with benchalerts

1. Go to the official GitHub instructions for creating an App.
   • If you are an admin of your GitHub organization, follow the instructions for "a GitHub App owned by an organization." This method is preferred because the org will own the app instead of a user, who may not be part of the org in the future. (This will not affect the identity of the bot that posts to GitHub, just the ownership of the App.)
   • If not, you can follow the instructions for "a GitHub App owned by a personal account." You will send an installation request to org admins after creating the app. You can always transfer the ownership of the app to an org later.
2. For the App Name, use conbench-<your org>.
3. For the Homepage URL, use the link to your Conbench server.
4. Ignore the Callback URL and Setup URL.
5. Uncheck the "Active" box under Webhook. Since this App will not be an active service, we don't need GitHub to push webhook events to the App.
6. For full use of this package, the App requires the following permissions:
   • Repository > Checks > Read and Write
   • Repository > Pull requests > Read and Write
7. After creating the App, save the App ID for later.
8. For the App's photo, use this one.
9. In the App Settings, scroll down to Private Keys and generate a private key. This will download a file to your computer. Treat the contents of this file like a password.
10. IMPORTANT: After creation, go to https://github.com/apps/<YOUR_APP_NAME>/installations/new to install the new App on the repos you'd like it to be able to post to. You must be a member of the organization to install the App on. If you are not an admin, an email request will be sent to org admins, which must be approved.

Running benchalerts as the GitHub App you created

All that's necessary to use benchalerts workflows that post to GitHub as your App is to set the following environment variables:

• GITHUB_APP_ID - the App ID from above
• GITHUB_APP_PRIVATE_KEY - the contents of the private key file from above. This is a multiline file, so ensure you quote the contents correctly if necessary.

Since benchalerts is usually used in CI, it's recommended to set these two environment variables in your CI pipeline as secret env vars. Most CI systems have a mechanism for doing this. For security reasons, do not check these values into version control.

License information

Copyright (c) 2022, Voltron Data.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.