biofilter-loki 3.0.0

Biofilter v3 – Legacy Edition, modernized with Poetry and APSW

🧬 Biofilter-LOKI 3.0.0

Biofilter-LOKI 3.0.0 is a lightweight, command-line–driven knowledge base builder designed to support BioBin and other legacy Biofilter workflows.
This version preserves the traditional LOKI architecture, while modernizing the codebase and deployment for current HPC environments.

---

🎯 Purpose & Design Goals

Biofilter-LOKI 3.0.0 was built to:

• Maintain full compatibility with BioBin
• Preserve the classic LOKI data model
• Provide a simple CLI-based workflow
• Support HPC module deployments
• Enable rapid database builds for analysis pipelines

---

🧠 What Is LOKI?

LOKI (Library Of Knowledge Integration) is the knowledge ingestion engine behind Biofilter.
It builds a SQLite knowledge database by integrating multiple biological data sources, such as:

• SNP ↔ Gene
• Gene ↔ Pathway
• Gene ↔ Ontology
• Identifier mappings across databases

---

🏗️ Architecture Overview

External Sources
        │
        ▼
┌──────────────┐
│  loki-build  │  ← CLI entry point
└──────────────┘
        │
        ▼
┌─────────────────────┐
│ SQLite Knowledge DB │
│    (LOKI schema)    │
└─────────────────────┘
        │
        ▼
┌──────────────┐
│   Biofilter  │  ← CLI entry point
└──────────────┘

Key characteristics:

• SQLite backend
• Immutable batch loads
• No entity-level curation
• Optimized for downstream queries

---

📦 Included Data Sources

Depending on build options, Biofilter-LOKI can ingest:

• dbSNP
• Entrez Gene
• Gene Ontology (GO)
• Pathways (KEGG / Reactome, if enabled)
• Chain files (genome build liftover)
• Identifier mappings

The available sources depend on how the package was built and deployed.

---

🚀 Installation

Option 1 — HPC Module (Recommended)

module load biofilter/3.0.0

Verify installation:

loki-build --version

---

Option 2 — Python Environment

pip install biofilter==3.0.0

or using Conda:

conda install -c conda-forge biofilter

---

🛠️ Building a Knowledge Database

Basic example:

loki-build \
  --knowledge loki.db \
  --load dbsnp entrez go

Update existing database:

loki-build \
  --knowledge loki.db \
  --update

Build from an archive:

loki-build \
  --from-archive loki_sources.tar.gz \
  --knowledge loki.db

---

🔍 Common CLI Options

Option	Description
--knowledge	Output SQLite database
--load	Load specific sources
--update	Update existing DB
--from-archive	Load from source archive
--to-archive	Save source archive
--no-optimize	Skip DB optimization
--verbose	Verbose logging

Run loki-build --help for full details.

---

🧪 Integration with BioBin

Biofilter-LOKI 3.0.0 is the reference backend for:

• BioBin 2.x
• Existing LOKI-based analysis pipelines
• Legacy workflows used in ADSP, ECHO, and related projects

Example:

biobin \
  --settings biobin.conf \
  --knowledge loki.db

---

❌ Known Limitations

• No entity-level conflict resolution
• No incremental curation
• No Parquet or OLAP support
• No variant-level functional annotations (e.g. VEP)
• Schema is not extensible without breaking compatibility

These limitations are intentional, to preserve stability.

---

🔮 Future Direction

All future innovation is happening in:

👉 Biofilter3R

Key differences:

Biofilter-LOKI	Biofilter3R
SQLite	PostgreSQL / hybrid
Immutable batches	Master entities
CLI only	CLI + Python
BioBin-focused	Multi-domain
Legacy schema	Modern relational

---

📚 Documentation

• BioBin documentation
• Internal Ritchie Lab Confluence pages
• Historical Biofilter publications

---

🧑‍🔬 Maintainers

Developed and maintained by the Ritchie Lab University of Pennsylvania

---

📜 License

Distributed under the original Biofilter license. See LICENSE file for details.

---

Development Documentation in:

https://ritchielab.github.io/biofilter/