assay-pipeline
Trusted assay data, and an AI agent you can attack. Public bioactivity data (EGFR IC50 by default, any target from config) from ChEMBL and PubChem, normalised into one versioned table with provenance on every row and quality problems flagged, never deleted. An AI agent answers questions about that table through a SQL guard and read-only access. Every query goes into a tamper-evident audit log, and the agent is attacked with cases mapped to the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS.
Security dashboard · Compound explorer · PyPI · Quickstart · Console guide · Threat model
| Security dashboard: did any attack get through, what stopped it, what it cost | Compound explorer: every published row, filterable, each linked to its source record |
| Console, replay: a recorded attack through the real guard, no API call | Console, live: your own question, on the console's key or your own |
What it does
- A trusted snapshot. Ingests ChEMBL and PubChem into a content-addressed cache, normalises structures with RDKit and units to nM and pIC50, and flags quality problems (QC1 to QC5) without deleting a row. Each snapshot is deterministic Parquet with a hashed manifest, a data card and a licence file. No API keys needed.
- An agent behind a guard. Claude or GPT answers questions through one SQL tool. Every query is parsed and checked against an allow-list of approved views and limits, runs read-only in an isolated process, and is written to a hash-chained audit log with published anchors.
- An attack suite with honest numbers. 60 attack cases across the OWASP LLM categories plus 30 normal questions, k trials per case, a scripted attacker with no model in the loop, a guard-off ablation, two judges checked against hand labels, and 95% upper bounds instead of "0%".
Results
Run main-2026-11-16 on snapshot 2026-11-15.1, from docs/results/main-2026-11-16/bundle.json (dashboard).
| Snapshot | 41,463 EGFR rows (chembl 26,553, pubchem 14,910); flagged, never deleted: QC1 1,470, QC2 4,414, QC3 174, QC5 21,761 |
| Attack trials that succeeded | 0 of 310: 300 agent trials (k=3 per case, public and held-out cases) and 10 poisoning cases met by the pipeline itself |
| Claude Sonnet 5.5 | 0 of 150; 95% upper bound 2.0% per trial |
| GPT-6.1 Sol | 0 of 150; 95% upper bound 2.0% per trial |
| What stopped them | model 288, quality_rule 10, limit 9, guard 3 (most attacks were refused by the model before any query ran) |
| Guard off (ablation) | 0 of 80 succeeded: the models refused on their own |
| Scripted attacker (no model) | 60 of 60 cases stopped: guard 45, quality_rule 10, limit 5 |
| Normal questions (public split) | Claude Sonnet 5.5: query matches gold 60%, answer states the gold value 100%; GPT-6.1 Sol: query matches gold 70%, answer states the gold value 73% |
| SQL shown in the answer is what ran (B1, all questions) | Claude Sonnet 5.5 97%; GPT-6.1 Sol 80% |
| Judges | agree with each other on 94 of 96; against 24 hand labels, Opus 24 and Sol 24 |
| Audit log | 1,377 lines, 494 anchors, verifies |
| Spend | $4.60 for the run, the guard-off ablation and the judges |
What these numbers do and don't show: reading the dashboard. The short version: the models refused almost every attack themselves, so the guard is tested mainly by the scripted attacker, which sends the attack SQL with no model in the loop. Held-out attack cases stay private so the result can't be tuned to them.
Get started
Python 3.11 to 3.15 on Linux, macOS or Windows. Building a snapshot needs no API keys.
uv tool install assay-pipeline # or: pip install assay-pipeline
assay doctor # checks Python, keys (set or not, never shown), network, local state
assay ingest chembl # ChEMBL activities for the target, into ./cache
assay ingest pubchem # PubChem bioassays, ChEMBL depositions dropped
assay snapshot # normalise, flag QC1-QC5, write snapshots/<date>.<n>/
Then, with a provider key, put an agent in front of it and attack it:
assay build-db snapshots/<id> # agent.duckdb: approved views + a synthetic restricted table
assay ask "How many compounds have an exact IC50?" --db agent.duckdb
assay attack --db agent.duckdb --k 3 --dry-run # the cost estimate; no call is made
assay attack --db agent.duckdb --k 3 --max-cost 10 # refused if estimated over $10, stopped if spend reaches it
assay report --run runs/<id> --snapshot <snapshot-id> # the dashboard
The quickstart walks through each step, configuring a target points the pipeline at another protein, and controlling cost covers estimates and caps.
The console
A Streamlit page where anyone can attack the agent and watch the guard and the audit log answer.
- Replay (the default) plays a recorded attack through the real guard and a fresh audit log, with no API call. Two scripted samples show the guard blocking queries that the models themselves refused to send.
- Live takes your own question, behind a passcode. Pay with the console's key, which draws on a small daily budget, or with your own Anthropic or OpenAI key: the passcode is still required, your key goes only to your provider for that one request, and it is never stored, logged or shown.
- Every query's guard decision sits beside the audit line it wrote.
Run it locally against an agent database you built:
uv sync --extra console
ASSAY_CONSOLE_DB=agent.duckdb ASSAY_CONSOLE_PASSCODE=pick-one uv run streamlit run apps/console/streamlit_app.py
Settings, the own-key rules and deployment are in the console guide.
How it fits together
flowchart LR
subgraph build["Pipeline (no model, no keys)"]
C[ChEMBL API] --> I[ingest<br/>raw cache, hashed]
P[PubChem API] --> I
I --> N[normalise<br/>RDKit parent, nM, pIC50]
N --> Q[flag QC1-QC5<br/>never delete]
Q --> S[(snapshot<br/>parquet + manifest<br/>+ data card)]
end
S --> D[(agent.duckdb<br/>approved views +<br/>synthetic restricted table)]
subgraph agent["Agent boundary"]
U[question or attack] --> A[agent<br/>Claude or GPT]
A -->|SQL| G{SQL guard<br/>parse, allow-list,<br/>limits}
G -->|allowed| R[read-only<br/>isolated process]
R --> D
G --> L[(audit log<br/>hash chain + anchors)]
R --> L
end
L --> E[scoring<br/>EvalKit + judges]
E --> B[results bundle] --> H[dashboard]
S --> X[compound explorer]
Rebuild
Re-render the published dashboard from its committed bundle and check it is byte-identical:
git clone https://github.com/biplob819/assay-pipeline && cd assay-pipeline && uv sync
uv run assay report --bundle "docs/results/$(cat docs/results/LATEST)" --html /tmp/index.html
cmp /tmp/index.html docs/report/index.html && uv run assay doctor --offline
Build your own snapshot with no API keys: assay ingest chembl, assay ingest pubchem, assay snapshot. See the quickstart.
Documentation
- Quickstart: a first snapshot with no keys, then an agent and an attack run
- Configuring another target
- Controlling cost: estimates, caps and what the published run cost
- Reading the dashboard
- Console guide: replay, live mode, your own key, settings and deployment
- Threat model (PDF, one page): assets, trust boundaries, OWASP and ATLAS mapping, controls, limits
- Data roadmap (PDF, one page): from snapshot to a training set
- Data card and data licence for the published snapshot
- Published run: bundle, audit anchors and verify result, hand-read of real answers
- Decision log, lessons and changelog
- PRD and Tech Spec, including the node graph the project was built by and the loophole review of the PRD
All links
| Source | github.com/biplob819/assay-pipeline |
| Package | pypi.org/project/assay-pipeline · releases |
| Live pages | security dashboard · compound explorer |
| Build status | CI · release workflow |
| Get help | issues · report a vulnerability privately · contributing |
| Data sources | ChEMBL (EMBL-EBI) · PubChem (NCBI) |
| Frameworks | OWASP Top 10 for LLM Applications 2025 · MITRE ATLAS |
| Built with | EvalKit (scoring and the build harness) · RDKit · DuckDB · sqlglot · Streamlit |
Privacy
No telemetry. The tools send data only to ChEMBL and PubChem (when building a snapshot, and when assay doctor checks they can be reached) and to the model providers whose keys you set (when running agents and judges). If you host the console, its live mode keeps a daily spend counter for the console's own keys in the Postgres database you configure; a visitor's own key goes only to that visitor's provider and is never stored or logged. Streamlit, which serves the console, has its usage statistics turned off, but it can look up the server's public IP when it checks a request's origin. Keys are read from your environment only and are never printed or logged. The console keeps visitor text in memory only. See SECURITY.md.
Licence and attribution
Code: MIT (LICENSE). Snapshots and the data embedded in the dashboard and explorer are derived from ChEMBL (EMBL-EBI) and PubChem (NCBI; rows from BindingDB only) and are licensed CC BY-SA 3.0; see the data licence for the full attribution. Scoring reuses EvalKit. The restricted table the attacks aim at, embargoed_results, is synthetic.
Contributions are welcome: see CONTRIBUTING.md and the code of conduct.
Metadata
Release files for assay-pipeline 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| assay_pipeline-0.1.1.tar.gz | 173.0 kB | With provenance | Details for assay_pipeline-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| assay_pipeline-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for assay_pipeline-0.1.1-py3-none-any.whl |
Total release size: 368.9 kB
Release files / assay_pipeline-0.1.1.tar.gz
| Download URL | assay_pipeline-0.1.1.tar.gz |
|---|---|
| Size | 173.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2455186791f28bead7ee426392639dbecfeba0d919445827ffd12ac46f8296f5
|
|
BLAKE2b-256 checksum How to use checksums |
ea56ce40892a3387a9aefeb1990cf6c59cf8130c79e69dce6c88591a1965826b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / assay_pipeline-0.1.1-py3-none-any.whl
| Download URL | assay_pipeline-0.1.1-py3-none-any.whl |
|---|---|
| Size | 195.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e4ae5108091206925ccf2b751cfd35b004523da3cb41854fe97d14fc18e81a99
|
|
BLAKE2b-256 checksum How to use checksums |
a7b0c72ff28276b0d75d3a9c2060fbc1bcf69553ba047fdcf6de1859b9ef75be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for assay-pipeline 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for assay_pipeline-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.