foldmetrics: unified confidence metrics & interface contacts for structure-prediction models

Testing
Package
Meta

🧬 What is it?

foldmetrics ingests the raw output folders of the mainstream structure predictors — AlphaFold2 / AlphaFold-Multimer, ColabFold, AlphaFold3, Boltz, Chai-1, Protenix, OpenDDE, HelixFold3, SeedFold — and computes a consistent set of quality metrics for monomers and complexes (protein, nucleic acid and small-molecule ligands), for a single model or whole batches:

Beyond scores it extracts confident interface contacts (with ready-to-open PyMOL/ChimeraX sessions) and renders publication-oriented figures.

Point it at whatever the engine wrote — its own output tree, untouched. To produce those predictions in the first place, foldrunner enumerates the pairs, computes each MSA once and drives all of these engines from one panel; its output is what foldmetrics reads here.

📦 Installation

pip install foldmetrics

With DockQ support (reference-based scoring):

pip install "foldmetrics[dockq]"

Development install:

git clone https://github.com/ChiaChunL/foldmetrics.git 
cd foldmetrics
pip install -e ".[dev]"

⚡ Quickstart

CLI (foldmetrics, short alias fmx):

Every command below runs as-is from a clone, against the bundled real examples — swap examples/data for your own prediction folders or files:

# score everything (tools are auto-detected and can be mixed)
fmx score examples/data -o metrics.tsv --interfaces interfaces.tsv --plot plots/

# one metric only — every metric name is also a subcommand
fmx ipsae examples/data
fmx score examples/data --metrics ipsae,pdockq2,lis

# campaign view: aggregate seeds/samples per target and tool
fmx score preds/ --by-target summary_by_target.tsv

# confident interface contacts: table + figure + PyMOL/ChimeraX sessions
fmx contacts examples/data/af3_server -o contacts.tsv --plot plots/

# DockQ: the AlphaFold2 model against the AlphaFold-Server model as reference
fmx dockq examples/data/af2_multimer --ref examples/data/af3_server/fold_barnase_barstar_s318_model_0.cif

# what would be scored?
fmx detect examples/data

Python:

import foldmetrics as fmx

df = fmx.evaluate("examples/data")  # one row per model
print(df[["model", "tool", "iptm", "ipsae", "pdockq2"]].round(3))
#                                model       tool   iptm  ipsae  pdockq2
#           model_1_multimer_v3_pred_0 alphafold2  0.937  0.897    0.952
#                    mpro_nirmatrelvir alphafold3  0.970  0.836      NaN
#    fold_barnase_barstar_s318_model_0 alphafold3  0.930  0.890    0.944
#              barnase_barstar_model_0      boltz  0.959  0.935    0.939

dfi = fmx.evaluate_interfaces("examples/data")  # one row per chain pair
agg = fmx.aggregate_by_target(df)  # one row per target+tool over all models
pred = fmx.load_predictions("examples/data")[0]
contacts = fmx.find_contacts(pred, dist_cutoff=8.0, pae_cutoff=12.0)

More recipes live in examples/ — real example predictions are included there, so every command runs as-is straight after cloning.

🛠️ Command-line reference

paths accepts anything: a directory (scanned recursively; different tools can be mixed freely), one specific model file, or several paths at once.

fmx <command> --help prints the complete option list for any command. Shell tab-completion is available via pip install "foldmetrics[completion]" followed by activate-global-python-argcomplete.

Screening many seeds and samples

Large campaigns produce many models per complex. --by-target collapses them into one row per target and tool — model count, mean/std/max ipTM and ipSAE, best pDockQ2, and the name of the best model (by ranking_score, else ipSAE, else ipTM):

         target       tool  n_models                         best_model  iptm_mean  iptm_std  ipsae_mean  ipsae_max
barnase_barstar alphafold3        16 barnase_barstar_seed-1030_sample-2      0.930     0.000       0.888      0.893

The standard deviations show how stable the prediction is across seeds — a low mean with high spread is a very different situation from a consistently low one. The same table is available in Python as fmx.aggregate_by_target(df).

The target is inferred from the job directory, so tools that were given different job names for the same complex (1brs here, 1brs_barnase_barstar there) land in different rows. --target-pattern reduces the names to the part that identifies the complex, which puts every method back on one row:

fmx score runs/ --target-pattern '^(\w{4})' --by-target by_target.tsv

The first capturing group becomes the new target; names the pattern does not match are left alone. In Python: fmx.apply_target_pattern(df, pattern).

🖼️ Visualization

--plot DIR (on score and every metric subcommand) or the plot subcommand renders figures that adapt automatically to the shape of the batch:

The structure panel renders via headless PyMOL when installed (--renderer pymol|trace overrides; FOLDMETRICS_PYMOL sets the path). plot also takes --format png|pdf|svg and --dpi.

Single model (AlphaFold3, SARS-CoV-2 Mpro + nirmatrelvir):

Targets × methods comparison (real batch: 10 complexes × 4 tools):

Batch overview (one target, AlphaFold2 + AlphaFold3 models):

🤝 Confident interface contacts

A contact is an inter-chain residue (or ligand-atom) pair within --dist-cutoff 8 Å whose PAE is below --pae-cutoff 12 Å in both directions (negative disables the PAE filter).

fmx contacts examples/data/af3_server -o contacts.tsv --plot plots/

-o writes the contact table; --plot DIR adds, per model:

• *_contacts.png — the figure below
• *_contacts.pse / *_contacts.cxs — PyMOL / ChimeraX sessions: double-click to open the styled interface scene, with if_A / if_B / hotspots / interface selections ready (--no-sessions skips)
• *_contacts.pml / *_contacts.cxc — the same scene as plain scripts, always written

🎯 DockQ against a reference

When an experimental (or otherwise trusted) structure exists, fmx dockq computes the actual interface accuracy via the official DockQ implementation (install with pip install "foldmetrics[dockq]"):

fmx dockq preds/ --ref 1brs.pdb -o dockq.tsv
fmx dockq preds/ --ref native.cif --mapping A:A,B:D   # explicit chain pairing
fmx dockq preds/ --ref homodimer.cif --best-mapping   # search all assignments
fmx dockq preds/ --ref complex.cif --small-molecule   # score ligand poses too

Reports DockQ, fnat, iRMSD, LRMSD, the CAPRI-style class and the chain mapping used, per interface. Chains are matched by name when both structures share names, otherwise by order — mmCIF label vs auth chain ids differ between tools, so check the mapping column. Override explicitly with --mapping MODEL:REF,..., or let --best-mapping try every assignment and keep the best total DockQ (recommended for homo-multimers; refused above 5 chains). Additional switches: --no-align (skip sequence alignment when residue numbering already matches), --low-memory (huge complexes), --capri-peptide (protein–peptide criteria).

📊 How to read the scores

pDockQ2 has two established readings

Zhu 2023 fixes pDockQ2's fitted sigmoid and its PAE term, but not which atom defines a contact, which atom's pLDDT to read, or whether the partner chain contributes to the pLDDT average. Implementations diverge:

The gap is ~0.005 on real AlphaFold3 output, because CA and CB contacts describe different interfaces (33 vs 53 pairs on barnase–barstar). It only matters for engines with per-atom pLDDT: pDockQ2 was fitted on AlphaFold2, where pLDDT is constant within a residue and all readings coincide. Pass variant= to foldmetrics.metrics.pdockq2_asym to choose; pdockq2_ab/pdockq2_ba are the per-chain values the paper defines, and the aggregate pdockq2 is their maximum, which is our convention — the paper defines no interface-level aggregate.

Single scores can mislead — pDockQ ignores PAE and can stay deceptively high on confidently-folded but wrongly-docked chains, which pDockQ2/ipSAE expose. Read them together (that is rather the point of this package).

🧰 Supported tools and files

Native per-tool extras (e.g. Boltz complex_iplddt/ligand_iptm, AF3 chain_pair_pae_min, Chai clash flags) are preserved on Prediction.extras and chain-pair ipTM is surfaced as iptm_native in the interface table.

Forks are a real problem here: OpenDDE writes Protenix's exact layout, file names and confidence keys, and SeedFold writes Boltz's. They are separated by whatever provenance the output actually carries — the mmCIF data block for the Protenix family, the absence of Boltz's own .npz files for SeedFold — never by guessing from the directory name.

✅ Validation

• Numerical parity with the ipsae.py reference implementation (Dunbrack Lab) verified digit-for-digit on real AlphaFold3 server output: ipSAE (both directions and d0chn variant), pDockQ, pDockQ2 and LIS all match to 6 decimal places at the default cutoffs (10/10).
• Batch-tested on 720+ real predictions across AlphaFold2-Multimer, AlphaFold3 (server + local), Boltz-2, Chai-1, ColabFold, Protenix, OpenDDE and SeedFold — including protein–small-molecule complexes, homodimers, monomers and negative controls — with zero parse errors; known binders score ipSAE 0.9+, decoy pairs < 0.1, monomers report NA. On one panel of three two-chain complexes, OpenDDE and Protenix given the same MSAs agree to within 0.02 on pTM, ipTM, ipSAE, pDockQ2 and pLDDT — two independent parses of two engines' own output.
• Layout-tested on native, untouched output directories from every supported engine, including multi-seed/multi-sample campaigns where the seed directory sits above the job directory; each engine's duplicate copies of the same model (AF2 ranked_*, AF3's top-level model, HelixFold3's -rank<N> directories) are counted once. HelixFold3 is so far validated at this level only: its file layout and key names come from a real run, but that run was made without MSAs, so its confidence values were not in a regime worth checking.
• Cross-implementation agreement: on real ColabFold 1.6 output our ipSAE/pDockQ/pDockQ2 reproduce ColabFold's own embedded values to ~1e-5 (bounded only by the 2-decimal PAE rounding in its JSON) — a CI regression test enforces this parity on every commit.

📋 What each metric needs

The structure file is always required (it defines chains and tokens); the table shows which additional inputs each metric consumes. When an input is missing the metric is NA and a note lands in the warnings column — nothing crashes.

📁 Outputs and paths

• Summary table → stdout; -o FILE writes it. The extension picks the format: .tsv (default), .csv, .json; missing values are NA.
• --interfaces FILE → the per chain-pair table (same formats).
• --plot DIR → figures as described under Visualization; contacts --plot adds <model>_contacts.png + <model>_contacts.pml. Model names are sanitized ([^\w.-] → _) for use as filenames.
• plot -o DIR defaults to ./foldmetrics_plots/.
• Exit codes: 0 success, 1 nothing recognized/found, 2 bad arguments or missing optional dependency.

💡 Conventions worth knowing

• Tokens. Standard residues are one token; ligands and modified residues are one token per heavy atom (AF3-style), so token-level PAE matrices line up across tools. pLDDT is stored on the 0–100 scale everywhere (Boltz 0–1 values are rescaled).
• Complex-level interface metrics are the best interface. For >2 chains, ipsae/pdockq/pdockq2/lis in the summary table are the maximum over chain pairs; use --interfaces for the full breakdown.
• Ligand interfaces. pdockq/pdockq2/iplddt are defined for polymer–polymer interfaces only. For chain pairs involving a ligand chain, ipsae/lis are computed over ligand atom tokens (experimental) and marked ipsae_mode = "tokens" in the interface table.
• ranking_score is comparable across tools. It is each tool's own ranking number, kept on 0–1. Some AlphaFold builds write ranking_confidence as a percentage (27.7 for 0.8·ipTM + 0.2·pTM = 0.277); that is rescaled, with the value as written preserved in Prediction.extras["alphafold2_ranking_confidence_raw"]. An AF2 monomer run is the exception the tool itself defines: there ranking_confidence is the mean pLDDT, and it stays on the pLDDT scale.
• The target is the job directory, not the complex. Seed and sample directories are stripped, so all models of one job aggregate together, but two tools given different job names for the same complex stay separate until you pass --target-pattern.
• Missing data degrades gracefully. No PAE → PAE-based metrics are NaN and a note lands in the warnings column; nothing crashes.
• Directionality. PAE is asymmetric, so pdockq2/ipsae/lis have two directional values; the interface table reports both (*_ab, *_ba) plus the aggregate used everywhere else (max for ipSAE/pDockQ2, mean for LIS, matching the reference implementations).

📚 References

• Jumper J et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021). — pLDDT / PAE and their confidence bands
• Bryant P, Pozzati G, Elofsson A. Improved prediction of protein-protein interactions using AlphaFold2. Nat Commun 13, 1265 (2022). — pDockQ
• Zhu W, Shenoy A, Kundrotas P, Elofsson A. Evaluation of AlphaFold-Multimer prediction on multi-chain protein complexes. Bioinformatics 39, btad424 (2023). — pDockQ2
• Dunbrack RL. ipSAE: scoring pairwise interactions in AlphaFold models. bioRxiv (2025). — ipSAE
• Kim AR et al. Enhanced protein-protein interaction discovery via AlphaFold-Multimer. bioRxiv (2024). — LIS
• Basu S, Wallner B. DockQ: A quality measure for protein-protein docking models. PLoS ONE 11, e0161879 (2016); Mirabello C, Wallner B. DockQ v2. Bioinformatics (2024). — DockQ

📄 License

BSD 3-Clause