artifact-secretary 0.0.0

Agentic discovery of compiled HPC applications: characterize an artifact's arch, linkage, capability, and provenance so a scheduler knows where it can run.

artifact-secretary

This library serves as a discovery tool to look into opaque objects (like containers) to figure out what is needed to run, build, etc. Our use case is for scheduling. We want to find the best spot for said opaque container.

Details

Container Discovery

For each image in a catalog:

1. Start the container running.
2. Copy our inspection code (Python) into it.
3. An agent drives the inspection (find binary, read ELF and linkage, how compiled).
4. Writes results into an artifact.

Install

Use the devcontainer (.devcontainer/).

python3 -m pip install -e .
python3 -m pip install -e .[aws]

# Claude
export ANTHROPIC_API_KEY=sk-ant-...

# AWS
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=us-east-1

Test connection:

aws sts get-caller-identity
aws bedrock list-inference-profiles --region "$AWS_DEFAULT_REGION" \
  --query 'inferenceProfileSummaries[].inferenceProfileId' --output text

python3 -c "import boto3; print(boto3.client('sts').get_caller_identity()['Arn'])"
python3 -c "import boto3,os; c=boto3.client('bedrock', region_name=os.environ.get('AWS_DEFAULT_REGION','us-east-1')); print([p['inferenceProfileId'] for p in c.list_inference_profiles()['inferenceProfileSummaries']])"

And test:

python3 - <<'PY'
from strands import Agent
from strands.models import BedrockModel
m = BedrockModel(model_id="us.anthropic.claude-sonnet-5",  # paste a real one from step 3
                 region_name="us-east-1")
print(Agent(model=m)("say ok"))
PY

Usage

artifact-secretary profile --backend aws \
  --model us.anthropic.claude-sonnet-5 \
  --catalog ghcr.io/converged-computing/metric-lammps-cpu:zen4-reax \
  --out lookup.json

By default it deletes any image it had to pull so your disk doesn't fill up. Images you already had are left alone. Use --keep-images to keep everything.

The inspection library works on its own if you just want the facts and don't want an agent or a key:

from secretary import Target, Inspector, derive_capability
insp = Inspector(Target("/opt/lammps"))
elf = insp.inspect_elf("/build/lmp")
cap = derive_capability(elf["needed"], elf["rpath"])

Output

A lookup keyed by digest. Facts only — no cluster, no jobspec. Multiple artifacts means multiple builds in one image (a LAMMPS image with both a CPU and a CUDA build, say).

{
  "version": "artifact-lookup/v1",
  "entries": {
    "reg/lammps@sha256:...": {
      "reproduce": {"reference": "reg/lammps:efa", "pulled_by_us": true},
      "artifacts": [
        {"application": "lammps", "binary": "/opt/lammps/build/lmp", "arch": "arm64",
         "needed": ["libcudart.so.12","libfabric.so.1","libefa.so.1","libmpi.so.40"],
         "capability": {"accelerator": "cuda", "fabric_efa": true, "mpi": "openmpi"},
         "provenance": {"build_system": "cmake", "compiler": "nvcc_wrapper",
                        "flags": ["Kokkos_ENABLE_CUDA=ON"]}}
      ]
    }
  }
}

Turning this into a fleetq jobspec is a separate job for a separate tool that reads the lookup. It's not this tool's business.

How the framework part works

A task says three things: what to ask the user, which tools the agent gets, and what a valid result looks like. The framework handles the rest — running the setup conversation, saving the manifest so runs are repeatable, and running the agent with those tools.

Tools are either read (agent uses them freely) or action (agent has to get the user to confirm before it runs). Profiling is all read tools. A future submit-a-job task would use an action tool for the actual submit, and the manifest lookup from this tool would be its input.

License

HPCIC DevTools is distributed under the terms of the MIT license. All new contributions must be made under this license.

See LICENSE, COPYRIGHT, and NOTICE for details.

SPDX-License-Identifier: (MIT)

LLNL-CODE- 842614