Skip to main content

assay-pipeline

CI Dashboard

Public bioactivity data (EGFR IC50 by default, any target from config) from ChEMBL and PubChem, normalised into one versioned table with provenance on every row and quality problems flagged, never deleted. An AI agent answers questions about that table through a SQL guard and read-only access. Every query goes into a tamper-evident audit log, and the agent is attacked with cases mapped to the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS.

Results

Run main-2026-11-16 on snapshot 2026-11-15.1, from docs/results/main-2026-11-16/bundle.json (dashboard).

Snapshot 41,463 EGFR rows (chembl 26,553, pubchem 14,910); flagged, never deleted: QC1 1,470, QC2 4,414, QC3 174, QC5 21,761
Attack trials that succeeded 0 of 310: 300 agent trials (k=3 per case, public and held-out cases) and 10 poisoning cases met by the pipeline itself
Claude Sonnet 5.5 0 of 150; 95% upper bound 2.0% per trial
GPT-6.1 Sol 0 of 150; 95% upper bound 2.0% per trial
What stopped them model 288, quality_rule 10, limit 9, guard 3 (most attacks were refused by the model before any query ran)
Guard off (ablation) 0 of 80 succeeded: the models refused on their own
Scripted attacker (no model) 60 of 60 cases stopped: guard 45, quality_rule 10, limit 5
Normal questions (public split) Claude Sonnet 5.5: query matches gold 60%, answer states the gold value 100%; GPT-6.1 Sol: query matches gold 70%, answer states the gold value 73%
SQL shown in the answer is what ran (B1, all questions) Claude Sonnet 5.5 97%; GPT-6.1 Sol 80%
Judges agree with each other on 94 of 96; against 24 hand labels, Opus 24 and Sol 24
Audit log 1,377 lines, 494 anchors, verifies
Spend $4.60 for the run, the guard-off ablation and the judges

What these numbers do and don't show: reading the dashboard. The short version: the models refused almost every attack themselves, so the guard is tested mainly by the scripted attacker, which sends the attack SQL with no model in the loop. Held-out attack cases stay private so the result can't be tuned to them.

How it fits together

flowchart LR
  subgraph build["Pipeline (no model, no keys)"]
    C[ChEMBL API] --> I[ingest<br/>raw cache, hashed]
    P[PubChem API] --> I
    I --> N[normalise<br/>RDKit parent, nM, pIC50]
    N --> Q[flag QC1-QC5<br/>never delete]
    Q --> S[(snapshot<br/>parquet + manifest<br/>+ data card)]
  end
  S --> D[(agent.duckdb<br/>approved views +<br/>synthetic restricted table)]
  subgraph agent["Agent boundary"]
    U[question or attack] --> A[agent<br/>Claude or GPT]
    A -->|SQL| G{SQL guard<br/>parse, allow-list,<br/>limits}
    G -->|allowed| R[read-only<br/>isolated process]
    R --> D
    G --> L[(audit log<br/>hash chain + anchors)]
    R --> L
  end
  L --> E[scoring<br/>EvalKit + judges]
  E --> B[results bundle] --> H[dashboard]
  S --> X[compound explorer]

Rebuild

Re-render the published dashboard from its committed bundle and check it is byte-identical:

git clone https://github.com/biplob819/assay-pipeline && cd assay-pipeline && uv sync
uv run assay report --bundle "docs/results/$(cat docs/results/LATEST)" --html /tmp/index.html
cmp /tmp/index.html docs/report/index.html && uv run assay doctor --offline

Build your own snapshot with no API keys: assay ingest chembl, assay ingest pubchem, assay snapshot. See the quickstart.

Documentation

Privacy

No telemetry. The tools send data only to ChEMBL and PubChem (when building a snapshot, and when assay doctor checks they can be reached) and to the model providers whose keys you set (when running agents and judges). If you host the console, its live mode also keeps a daily spend counter in the Postgres database you configure. Streamlit, which serves the console, has its usage statistics turned off, but it can look up the server's public IP when it checks a request's origin. Keys are read from your environment only and are never printed or logged. The public console keeps visitor text in memory only. See SECURITY.md.

Licence and attribution

Code: MIT (LICENSE). Snapshots and the data embedded in the dashboard and explorer are derived from ChEMBL (EMBL-EBI) and PubChem (NCBI; rows from BindingDB only) and are licensed CC BY-SA 3.0; see the data licence for the full attribution. Scoring reuses EvalKit. The restricted table the attacks aim at, embargoed_results, is synthetic.

Contributions are welcome: see CONTRIBUTING.md and the code of conduct.

Metadata

Release files for assay-pipeline 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for assay-pipeline 0.1.0
File Size Uploaded Provenance Actions
assay_pipeline-0.1.0.tar.gz 167.1 kB With provenance Details for assay_pipeline-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for assay-pipeline 0.1.0
File Interpreter ABI Platform Actions
assay_pipeline-0.1.0-py3-none-any.whl Python 3 none any Details for assay_pipeline-0.1.0-py3-none-any.whl

Total release size: 358.0 kB

Release files / assay_pipeline-0.1.0.tar.gz

Download URL assay_pipeline-0.1.0.tar.gz
Size 167.1 kB
Tags Source
SHA-256 checksum
How to use checksums
a45cc04d58fe11d6ac04f67ac557cae0a5a3bab146fdda9ba21edea1e9867661
BLAKE2b-256 checksum
How to use checksums
02aad5444896bbb0830e1f22a50437778e65e78bc4a946b5b326c0790fff96f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / assay_pipeline-0.1.0-py3-none-any.whl

Download URL assay_pipeline-0.1.0-py3-none-any.whl
Size 190.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bc08d76c524625d43f5cb03b25facaadb1b5082e52cdc3029cea843f68473d77
BLAKE2b-256 checksum
How to use checksums
9d97aef617380ded6c3729d1920a19e1093fbac35cb69ecb645b25a36b7d5e34
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for assay-pipeline 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for assay_pipeline-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page