assay-pipeline
Public bioactivity data (EGFR IC50 by default, any target from config) from ChEMBL and PubChem, normalised into one versioned table with provenance on every row and quality problems flagged, never deleted. An AI agent answers questions about that table through a SQL guard and read-only access. Every query goes into a tamper-evident audit log, and the agent is attacked with cases mapped to the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS.
Results
Run main-2026-11-16 on snapshot 2026-11-15.1, from docs/results/main-2026-11-16/bundle.json (dashboard).
| Snapshot | 41,463 EGFR rows (chembl 26,553, pubchem 14,910); flagged, never deleted: QC1 1,470, QC2 4,414, QC3 174, QC5 21,761 |
| Attack trials that succeeded | 0 of 310: 300 agent trials (k=3 per case, public and held-out cases) and 10 poisoning cases met by the pipeline itself |
| Claude Sonnet 5.5 | 0 of 150; 95% upper bound 2.0% per trial |
| GPT-6.1 Sol | 0 of 150; 95% upper bound 2.0% per trial |
| What stopped them | model 288, quality_rule 10, limit 9, guard 3 (most attacks were refused by the model before any query ran) |
| Guard off (ablation) | 0 of 80 succeeded: the models refused on their own |
| Scripted attacker (no model) | 60 of 60 cases stopped: guard 45, quality_rule 10, limit 5 |
| Normal questions (public split) | Claude Sonnet 5.5: query matches gold 60%, answer states the gold value 100%; GPT-6.1 Sol: query matches gold 70%, answer states the gold value 73% |
| SQL shown in the answer is what ran (B1, all questions) | Claude Sonnet 5.5 97%; GPT-6.1 Sol 80% |
| Judges | agree with each other on 94 of 96; against 24 hand labels, Opus 24 and Sol 24 |
| Audit log | 1,377 lines, 494 anchors, verifies |
| Spend | $4.60 for the run, the guard-off ablation and the judges |
What these numbers do and don't show: reading the dashboard. The short version: the models refused almost every attack themselves, so the guard is tested mainly by the scripted attacker, which sends the attack SQL with no model in the loop. Held-out attack cases stay private so the result can't be tuned to them.
How it fits together
flowchart LR
subgraph build["Pipeline (no model, no keys)"]
C[ChEMBL API] --> I[ingest<br/>raw cache, hashed]
P[PubChem API] --> I
I --> N[normalise<br/>RDKit parent, nM, pIC50]
N --> Q[flag QC1-QC5<br/>never delete]
Q --> S[(snapshot<br/>parquet + manifest<br/>+ data card)]
end
S --> D[(agent.duckdb<br/>approved views +<br/>synthetic restricted table)]
subgraph agent["Agent boundary"]
U[question or attack] --> A[agent<br/>Claude or GPT]
A -->|SQL| G{SQL guard<br/>parse, allow-list,<br/>limits}
G -->|allowed| R[read-only<br/>isolated process]
R --> D
G --> L[(audit log<br/>hash chain + anchors)]
R --> L
end
L --> E[scoring<br/>EvalKit + judges]
E --> B[results bundle] --> H[dashboard]
S --> X[compound explorer]
Rebuild
Re-render the published dashboard from its committed bundle and check it is byte-identical:
git clone https://github.com/biplob819/assay-pipeline && cd assay-pipeline && uv sync
uv run assay report --bundle "docs/results/$(cat docs/results/LATEST)" --html /tmp/index.html
cmp /tmp/index.html docs/report/index.html && uv run assay doctor --offline
Build your own snapshot with no API keys: assay ingest chembl, assay ingest pubchem, assay snapshot. See the quickstart.
Documentation
- Quickstart: a first snapshot with no keys, then an agent and an attack run
- Configuring another target
- Controlling cost: estimates, caps and what the published run cost
- Reading the dashboard
- Compound explorer: filter every published row and open its source record
- Threat model (one page): assets, trust boundaries, OWASP and ATLAS mapping, controls, limits
- Data roadmap (one page): from snapshot to a training set
- Data card and data licence for the published snapshot
- Published run: bundle, audit anchors and verify result
- Decision log and lessons
- PRD and Tech Spec, including the node graph the project was built by and the loophole review of the PRD
Privacy
No telemetry. The tools send data only to ChEMBL and PubChem (when building a snapshot, and when assay doctor checks they can be reached) and to the model providers whose keys you set (when running agents and judges). If you host the console, its live mode also keeps a daily spend counter in the Postgres database you configure. Streamlit, which serves the console, has its usage statistics turned off, but it can look up the server's public IP when it checks a request's origin. Keys are read from your environment only and are never printed or logged. The public console keeps visitor text in memory only. See SECURITY.md.
Licence and attribution
Code: MIT (LICENSE). Snapshots and the data embedded in the dashboard and explorer are derived from ChEMBL (EMBL-EBI) and PubChem (NCBI; rows from BindingDB only) and are licensed CC BY-SA 3.0; see the data licence for the full attribution. Scoring reuses EvalKit. The restricted table the attacks aim at, embargoed_results, is synthetic.
Contributions are welcome: see CONTRIBUTING.md and the code of conduct.
Metadata
Release files for assay-pipeline 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| assay_pipeline-0.1.0.tar.gz | 167.1 kB | With provenance | Details for assay_pipeline-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| assay_pipeline-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for assay_pipeline-0.1.0-py3-none-any.whl |
Total release size: 358.0 kB
Release files / assay_pipeline-0.1.0.tar.gz
| Download URL | assay_pipeline-0.1.0.tar.gz |
|---|---|
| Size | 167.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a45cc04d58fe11d6ac04f67ac557cae0a5a3bab146fdda9ba21edea1e9867661
|
|
BLAKE2b-256 checksum How to use checksums |
02aad5444896bbb0830e1f22a50437778e65e78bc4a946b5b326c0790fff96f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / assay_pipeline-0.1.0-py3-none-any.whl
| Download URL | assay_pipeline-0.1.0-py3-none-any.whl |
|---|---|
| Size | 190.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bc08d76c524625d43f5cb03b25facaadb1b5082e52cdc3029cea843f68473d77
|
|
BLAKE2b-256 checksum How to use checksums |
9d97aef617380ded6c3729d1920a19e1093fbac35cb69ecb645b25a36b7d5e34
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for assay-pipeline 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for assay_pipeline-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.