aind-codeocean-pipeline-monitor
Package for starting a pipeline, waiting for it to finish, and optionally capturing the results as a data asset.
Installation
The repo can be install from PyPI. To pip install all of the necessary dependencies to run the pipeline monitor, run:
pip install aind-codeocean-pipeline-monitor[full]
To install only the minimum dependencies required for model validation, run:
pip install aind-codeocean-pipeline-monitor
To install the package for development, clone this repo and run
pip install -e .[dev]
Usage
- Define job using PipelineMonitorJobSettings class.
- Define a CodeOcean client.
- Construct a PipelineMonitorJob with these settings.
- Run the job with the run_job method.
import os
from codeocean import CodeOcean
from codeocean.computation import DataAssetsRunParam, RunParams
from urllib3.util import Retry
from aind_codeocean_pipeline_monitor.job import PipelineMonitorJob
from aind_codeocean_pipeline_monitor.models import (
CaptureSettings,
PipelineMonitorSettings,
)
domain = os.getenv("CODEOCEAN_DOMAIN")
token = os.getenv("CODEOCEAN_TOKEN")
# Recommend adding retry strategy to requests session
retry = Retry(
total=5,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET", "POST"],
)
client = CodeOcean(domain=domain, token=token, retries=retry)
# Please consult Code Ocean docs for info about RunParams and DataAssetParams
settings = PipelineMonitorSettings(
run_params=RunParams(
capsule_id="<your capsule id>",
data_assets=[
DataAssetsRunParam(
id="<your input data asset id>",
mount="<your input data mount>",
)
],
),
capture_settings=CaptureSettings(
tags=["derived"]
), # 'tags' is the only required field
)
job = PipelineMonitorJob(job_settings=settings, client=client)
job.run_job()
Contributing
Linters and testing
There are several libraries used to run linters, check documentation, and run tests.
- Please test your changes using the coverage library, which will run the tests and log a coverage report:
coverage run -m unittest discover && coverage report
- Use interrogate to check that modules, methods, etc. have been documented thoroughly:
interrogate .
- Use flake8 to check that code is up to standards (no unused imports, etc.):
flake8 .
- Use black to automatically format the code into PEP standards:
black .
- Use isort to automatically sort import statements:
isort .
Pull requests
For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:
<type>(<scope>): <short summary>
where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:
- build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
- ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
- docs: Documentation only changes
- feat: A new feature
- fix: A bugfix
- perf: A code change that improves performance
- refactor: A code change that neither fixes a bug nor adds a feature
- test: Adding missing tests or correcting existing tests
Semantic Release
The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):
| Commit message | Release type |
|---|---|
fix(pencil): stop graphite breaking when too much pressure applied |
|
feat(pencil): add 'graphiteWidth' option |
|
perf(pencil): remove graphiteWidth optionBREAKING CHANGE: The graphiteWidth option has been removed.The default graphite width of 10mm is always used for performance reasons. |
(Note that the BREAKING CHANGE: token must be in the footer of the commit) |
Documentation
To generate the rst files source files for documentation, run
sphinx-apidoc -o docs/source/ src
Then to create the documentation HTML files, run
sphinx-build -b html docs/source/ docs/build/html
More info on sphinx installation can be found here.
Metadata
Release files for aind-codeocean-pipeline-monitor 0.11.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aind_codeocean_pipeline_monitor-0.11.2.tar.gz | 18.5 kB | Without provenance | Details for aind_codeocean_pipeline_monitor-0.11.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aind_codeocean_pipeline_monitor-0.11.2-py3-none-any.whl | Python 3 | none | any | Details for aind_codeocean_pipeline_monitor-0.11.2-py3-none-any.whl |
Total release size: 29.7 kB
Release files / aind_codeocean_pipeline_monitor-0.11.2.tar.gz
| Download URL | aind_codeocean_pipeline_monitor-0.11.2.tar.gz |
|---|---|
| Size | 18.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5e4c04cbcddfedc78ee56861f7eeaa651a2a645c54176a416a3d79af2a914d16
|
|
BLAKE2b-256 checksum How to use checksums |
aaa2dd8a6211e188f2ef465089d75d648c19624daa062166d50d3dd769ea85b5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / aind_codeocean_pipeline_monitor-0.11.2-py3-none-any.whl
| Download URL | aind_codeocean_pipeline_monitor-0.11.2-py3-none-any.whl |
|---|---|
| Size | 11.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fe8d7e7200503294430d57874d6ff4e1a0cab37fed2d10f705a687ab29d9e82a
|
|
BLAKE2b-256 checksum How to use checksums |
41ac49a512c8637d5a808da1d5738db8eed68607d665d77eefb5e49943121fa1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for aind-codeocean-pipeline-monitor 0.11.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.