Skip to main content

assay-pipeline

CI PyPI Python Licence: MIT Dashboard

Trusted assay data, and an AI agent you can attack. Public bioactivity data (EGFR IC50 by default, any target from config) from ChEMBL and PubChem, normalised into one versioned table with provenance on every row and quality problems flagged, never deleted. An AI agent answers questions about that table through a SQL guard and read-only access. Every query goes into a tamper-evident audit log, and the agent is attacked with cases mapped to the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS.

Security dashboard · Compound explorer · PyPI · Quickstart · Console guide · Threat model

The security dashboard: 0 of 248 attack trials got through, with the 95% upper bound per category and agent The compound explorer: 41,463 activity rows with filters, flags and a link to each source record
Security dashboard: did any attack get through, what stopped it, what it cost Compound explorer: every published row, filterable, each linked to its source record
The console replaying a recorded attack: the guard blocks a query on the restricted table and writes audit lines The console's live mode: pick an agent and pay with your own API key, behind the passcode
Console, replay: a recorded attack through the real guard, no API call Console, live: your own question, on the console's key or your own

A 22-second walkthrough: the security dashboard, what stopped each attack, agent behaviour, the comparison with the previous run, the audit trail, the compound explorer, the console replaying a blocked query and the release gate

Walkthrough of the committed outputs (shot list).

What it does

  • A trusted snapshot. Ingests ChEMBL and PubChem into a content-addressed cache, normalises structures with RDKit and units to nM and pIC50, and flags quality problems (QC1 to QC5) without deleting a row. Each snapshot is deterministic Parquet with a hashed manifest, a data card and a licence file. No API keys needed.
  • An agent behind a guard. Claude or GPT answers questions through one SQL tool. Every query is parsed and checked against an allow-list of approved views and limits, runs read-only in an isolated process, and is written to a hash-chained audit log with published anchors.
  • An attack suite with honest numbers. 60 attack cases across the OWASP LLM categories plus 30 normal questions, k trials per case, a scripted attacker with no model in the loop, a guard-off ablation, two judges checked against hand labels, and 95% upper bounds instead of "0%".

Results

Run main-2026-11-18 on snapshot 2026-11-15.1, from docs/results/main-2026-11-18/bundle.json (dashboard).

Snapshot 41,463 EGFR rows (chembl 26,553, pubchem 14,910); flagged, never deleted: QC1 1,470, QC2 4,414, QC3 174, QC5 21,761
Attack trials that succeeded 0 of 248: 240 agent trials (k=3 per case, public cases only) and 8 poisoning cases met by the pipeline itself
Claude Sonnet 5.5 0 of 120; 95% upper bound 2.5% per trial
GPT-6.1 Sol 0 of 120; 95% upper bound 2.5% per trial
What stopped them model 231, guard 9, quality_rule 8 (most attacks were refused by the model before any query ran)
Guard off (ablation) 0 of 80 succeeded: stopped by model 79, limit 1
Scripted attacker (no model) 60 of 60 cases stopped: guard 45, quality_rule 10, limit 5
Normal questions (public split) Claude Sonnet 5.5: query matches gold 70%, answer states the gold value 100%; GPT-6.1 Sol: query matches gold 93%, answer states the gold value 93%
SQL shown in the answer is what ran (B1, all questions) Claude Sonnet 5.5 100%; GPT-6.1 Sol 98%
Judges agree with each other on 78 of 78; against 24 hand labels, Opus 24 and Sol 24
Audit log 997 lines, 394 anchors, verifies
Spend $3.49 for the run, the guard-off ablation and the judges

What these numbers do and don't show: reading the dashboard. The short version: the models refused almost every attack themselves, so the guard is tested mainly by the scripted attacker, which sends the attack SQL with no model in the loop. Held-out attack cases stay private so the result can't be tuned to them.

Get started

Python 3.11 to 3.15 on Linux, macOS or Windows. Building a snapshot needs no API keys.

uv tool install assay-pipeline      # or: pip install assay-pipeline
assay doctor                        # checks Python, keys (set or not, never shown), network, local state
assay ingest chembl                 # ChEMBL activities for the target, into ./cache
assay ingest pubchem                # PubChem bioassays, ChEMBL depositions dropped
assay snapshot                      # normalise, flag QC1-QC5, write snapshots/<date>.<n>/

Then, with a provider key, put an agent in front of it and attack it:

assay build-db snapshots/<id>                         # agent.duckdb: approved views + a synthetic restricted table
assay ask "How many compounds have an exact IC50?" --db agent.duckdb
assay attack --db agent.duckdb --k 3 --dry-run        # the cost estimate; no call is made
assay attack --db agent.duckdb --k 3 --max-cost 10    # refused if estimated over $10, stopped if spend reaches it
assay report --run runs/<id> --snapshot <snapshot-id> # the dashboard

The quickstart walks through each step, configuring a target points the pipeline at another protein, and controlling cost covers estimates and caps.

The console

A Streamlit page where anyone can attack the agent and watch the guard and the audit log answer.

  • Replay (the default) plays a recorded attack through the real guard and a fresh audit log, with no API call. Two scripted samples show the guard blocking queries that the models themselves refused to send.
  • Live takes your own question, behind a passcode. Pay with the console's key, which draws on a small daily budget, or with your own Anthropic or OpenAI key: the passcode is still required, your key goes only to your provider for that one request, and it is never stored, logged or shown.
  • Every query's guard decision sits beside the audit line it wrote.

Run it locally against an agent database you built:

uv sync --extra console
ASSAY_CONSOLE_DB=agent.duckdb ASSAY_CONSOLE_PASSCODE=pick-one uv run streamlit run apps/console/streamlit_app.py

Settings, the own-key rules and deployment are in the console guide.

How it fits together

flowchart LR
  subgraph build["Pipeline (no model, no keys)"]
    C[ChEMBL API] --> I[ingest<br/>raw cache, hashed]
    P[PubChem API] --> I
    I --> N[normalise<br/>RDKit parent, nM, pIC50]
    N --> Q[flag QC1-QC5<br/>never delete]
    Q --> S[(snapshot<br/>parquet + manifest<br/>+ data card)]
  end
  S --> D[(agent.duckdb<br/>approved views +<br/>synthetic restricted table)]
  subgraph agent["Agent boundary"]
    U[question or attack] --> A[agent<br/>Claude or GPT]
    A -->|SQL| G{SQL guard<br/>parse, allow-list,<br/>limits}
    G -->|allowed| R[read-only<br/>isolated process]
    R --> D
    G --> L[(audit log<br/>hash chain + anchors)]
    R --> L
  end
  L --> E[scoring<br/>EvalKit + judges]
  E --> B[results bundle] --> H[dashboard]
  S --> X[compound explorer]

Rebuild

Re-render the published dashboard from its committed bundle and check it is byte-identical:

git clone https://github.com/biplob819/assay-pipeline && cd assay-pipeline && uv sync
uv run assay report --bundle "docs/results/$(cat docs/results/LATEST)" --html /tmp/index.html
cmp /tmp/index.html docs/report/index.html && uv run assay doctor --offline

Build your own snapshot with no API keys: assay ingest chembl, assay ingest pubchem, assay snapshot. See the quickstart.

Documentation

Source github.com/biplob819/assay-pipeline
Package pypi.org/project/assay-pipeline · releases
Live pages security dashboard · compound explorer
Build status CI · release workflow
Get help issues · report a vulnerability privately · contributing
Data sources ChEMBL (EMBL-EBI) · PubChem (NCBI)
Frameworks OWASP Top 10 for LLM Applications 2025 · MITRE ATLAS
Built with EvalKit (scoring and the build harness) · RDKit · DuckDB · sqlglot · Streamlit

Privacy

No telemetry. The tools send data only to ChEMBL and PubChem (when building a snapshot, and when assay doctor checks they can be reached) and to the model providers whose keys you set (when running agents and judges). If you host the console, its live mode keeps a daily spend counter for the console's own keys in the Postgres database you configure; a visitor's own key goes only to that visitor's provider and is never stored or logged. Streamlit, which serves the console, has its usage statistics turned off, but it can look up the server's public IP when it checks a request's origin. Keys are read from your environment only and are never printed or logged. The console keeps visitor text in memory only. See SECURITY.md.

Licence and attribution

Code: MIT (LICENSE). Snapshots and the data embedded in the dashboard and explorer are derived from ChEMBL (EMBL-EBI) and PubChem (NCBI; rows from BindingDB only) and are licensed CC BY-SA 3.0; see the data licence for the full attribution. Scoring reuses EvalKit. The restricted table the attacks aim at, embargoed_results, is synthetic.

Contributions are welcome: see CONTRIBUTING.md and the code of conduct.

Metadata

Release files for assay-pipeline 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for assay-pipeline 0.1.2
File Size Uploaded Provenance Actions
assay_pipeline-0.1.2.tar.gz 185.3 kB With provenance Details for assay_pipeline-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for assay-pipeline 0.1.2
File Interpreter ABI Platform Actions
assay_pipeline-0.1.2-py3-none-any.whl Python 3 none any Details for assay_pipeline-0.1.2-py3-none-any.whl

Total release size: 393.1 kB

Release files / assay_pipeline-0.1.2.tar.gz

Download URL assay_pipeline-0.1.2.tar.gz
Size 185.3 kB
Tags Source
SHA-256 checksum
How to use checksums
46e1add9e5c658b6863f88cb19b3d2b7c8797c6b47ff79c28fa15481440191e4
BLAKE2b-256 checksum
How to use checksums
2ae44fc97d9c32d673a64b8fcf3b276e5ebe396c3bdbcb8252152c3125764dea
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / assay_pipeline-0.1.2-py3-none-any.whl

Download URL assay_pipeline-0.1.2-py3-none-any.whl
Size 207.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dbbc977d61854e56fe0af402ab53ed06d5bb1d668f4bb68431c4a52a1ab09b35
BLAKE2b-256 checksum
How to use checksums
00fcbe784ee3894e0e668c415e95999bacfb4c3633fe6fb8835effac391e9596
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for assay-pipeline 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for assay_pipeline-0.1.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page