polycomb

Automatic atlas design and curation utilities for building homeobox atlases from public datasets. The repository provides a Python library (polycomb), agent skills that guide each workflow step, and scripts that automate repetitive operations.

Install

Install the package and agent skills:

pip install polycomb
curl -sSL https://raw.githubusercontent.com/epiblastai/homeobox/refs/heads/main/packages/polycomb/install.sh | bash

The install script copies skills from skills/ into ~/.agents/skills/ and links them for Claude at ~/.claude/skills/. Each skill documents the procedures and scripts for one workflow stage.

Reference cache setup and maintenance

Polycomb resolvers use a shared LanceDB reference cache for fast local lookup of genes, proteins, ontology terms, molecules, guide RNAs, organisms, and cell lines. The cache is optional: resolvers are expected to return structured reports when the cache is missing or empty, and several resolvers can fall back to online services such as Ensembl/gget, OLS, PubChem, ChEMBL, and BLAT where appropriate. A populated cache is still recommended for speed, reproducibility, and offline runs.

By default, Polycomb looks for:

~/.cache/polycomb/reference_db

A prefilled cache is available for download from Hugging Face at epiblastai/polycomb-lancedb. It ships with a few ontologies, PubChem, and Ensembl, plus guide RNAs pre-resolved using bowtie2. Download it and point setup at the resulting path to use it as your reference cache.

You can initialize an empty cache with the correct table schemas:

polycomb setup

To initialize a cache at a specific path:

polycomb setup --db-path /path/to/reference_db

Every setup run writes ~/.polycomb/config.json, making the given --db-path (or the default path) the reference DB used by future resolver calls.

For object-store-backed LanceDB caches, pass storage options as JSON or from a file:

polycomb setup --db-path s3://bucket/polycomb/reference_db --storage-options-json '{"region": "us-east-1"}'

polycomb setup --db-path s3://bucket/polycomb/reference_db --storage-options-file storage-options.json

Use --force-tables only when you deliberately want to recreate existing reference tables as empty tables:

polycomb setup --db-path /path/to/reference_db --force-tables

After populating or updating reference tables, create/update indexes and compact the cache:

polycomb optimize-cache

Useful maintenance variants:

polycomb optimize-cache --db-path /path/to/reference_db
polycomb optimize-cache --dry-run
polycomb optimize-cache --tables ontology_terms genomic_feature_aliases
polycomb optimize-cache --skip-indexes
polycomb optimize-cache --skip-optimize

optimize-cache skips missing tables, missing columns, and empty tables. It creates full-text indexes for text-heavy lookup columns, scalar indexes for exact-match columns, and runs LanceDB table optimization unless disabled.

Workflow overview

An polycomb run turns a public dataset (GEO, SRA, SCP, …) into a finalized homeobox atlas. The stages are sequential; each skill assumes the previous ones have completed.

flowchart LR
  A[atlas-designer] --> B[create-data-package]
  B --> C[prepare-package-for-resolution]
  C --> D{multimodal?}
  D -->|yes| E[multimodal-alignment]
  D -->|no| F[schema-harmonization]
  E --> F
  F --> G[finalize-tables]
  G --> H[write-ingestion-script]

Steps 1 and 2 are technically independent — neither requires the other's output — but in practice you define the atlas schema before downloading and organizing data, so that feature spaces, registry tables, and metadata fields are settled before staging begins.

Scripts by workflow stage

Paths are relative to the repository root. When a skill is installed, its scripts live under that skill's directory (e.g. ~/.agents/skills/create-data-package/scripts/).

1. atlas-designer

Produces a schema YAML (the homeobox schema IR), which codegens into a schema.py. A reference example IR lives at skills/atlas-designer/references/multimodal_perturbation_atlas_schema.yaml.

2. create-data-package

Organize downloaded files into a collection data package using the Collection API (polycomb.collection). Scripts assist with common public-database sources.

See skills/create-data-package/references/geo_instructions.md for GEO-specific guidance and skills/create-data-package/references/figshare_instructions.md for figshare.

3. prepare-package-for-resolution

Stage raw tables into LanceDB under each dataset's lance_db/ (and collection-level lance_db/ for shared registries). Staged columns are not yet schema-aligned.

4. multimodal-alignment

Run only when a dataset has two or more feature-space obs tables (CITE-seq, Multiome, …).

5. schema-harmonization

Harmonize every staged table to its target schema class. Mutations go through the audited CurationApplicator — never edit Lance directly.

Most harmonization work is custom curation transactions planned per table; the script accelerates the common single-column resolution passes.

6. finalize-tables

Run after harmonization has finished on every table in the collection. The main entrypoint resolves registry-key dependencies in DAG order.

7. write-ingestion-script

Ingestion scripts are collection-specific — they declare one Loader per feature space and call polycomb.ingestion.ingest_collection. An example for a finalized GSE264667 collection:

On-disk layout after finalization

collection_root/
  collection.json                # manifest: datasets, uids, DATA file tags
  lance_db/                      # collection-level registry tables
  <dataset>/
    lance_db/
      <ObsClass>                 # finalized obs (joined for multimodal)
      <ObsClass>_<feature_space> # per-fs uid artifact for DATA row alignment
      <DatasetClass>             # one row per feature space; zarr_group set
      <RegistryClass>            # feature registry / var table
    <DATA files>                 # raw matrices, unchanged

Ingestion reads this layout, streams DATA into zarr, and fills pointer fields and summary aggregates deferred until write time.