bicep 0.1.0

Bi-directional Cross-Attention Enrichment Preprocessor — a PEFT adapter for HuggingFace transformers.

BiCEP — Bi-directional Cross-Attention Enrichment Preprocessor

BiCEP is a PEFT adapter that enriches prompt embeddings with a small trainable bidirectional transformer before they reach a frozen causal LLM. Unlike LoRA (which modifies the LLM's weights) or prompt tuning (which prepends virtual tokens), BiCEP transforms the input embedding sequence in place — same length, no tokens added or removed — then feeds the result to the frozen model.

It behaves like a native PEFT adapter: get_peft_model, save_pretrained, from_pretrained, push_to_hub, and generate all work without special-casing.

⚠️ Experimental. BiCEP is an architectural hypothesis whose efficacy has not yet been empirically validated. The API is usable and stable, but there is no published evidence yet that the enricher improves downstream performance. I plan to train and release checkpoints to gather empirical data in the near-to-mid future — and anyone is more than welcome to do so independently in the meantime. Until then, treat results as unproven and expect the project to stay on 0.x.

Install

pip install bicep        # also needs torch>=2.1, transformers>=5.0,<6, peft>=0.19.1,<0.20

Configuration — three fields

Field	Default	Meaning
base_model_name_or_path	required	HF model id / local path of the frozen base LLM
num_blocks	3	Number of (bidirectional self-attention + FFN) enrichment blocks
d_head	64	Attention head dimension. Must divide the base model's hidden_size

d_model, n_heads, and vocab_size are derived automatically from the base model — you never set them.

Train

import bicep                                  # registers the BICEP PEFT type on import
from bicep import BiCEPConfig, BiCEPDataCollator
from peft import get_peft_model
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          Trainer, TrainingArguments)

name = "meta-llama/Llama-3.1-8B"
base = AutoModelForCausalLM.from_pretrained(name)
tok  = AutoTokenizer.from_pretrained(name)

model = get_peft_model(base, BiCEPConfig(base_model_name_or_path=name))
collator = BiCEPDataCollator(tokenizer=tok)   # MANDATORY — see below

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="out",
        remove_unused_columns=False,          # REQUIRED with BiCEPDataCollator (see note)
    ),
    train_dataset=dataset,                     # see "Data formats"
    data_collator=collator,
)
trainer.train()
model.push_to_hub("you/bicep-llama3")

Only the adapter is trained — all base-model weights stay frozen, and save_pretrained / push_to_hub store only the adapter weights.

Publish with a model card

BiCEP is a third-party PEFT method, so anyone loading your checkpoint must import bicep first (see below). Ship a model card that says so — render_model_card produces a Hub-ready README.md (with library_name: peft front matter):

from pathlib import Path
from bicep import render_model_card

save_dir = "out/bicep-llama3"
model.save_pretrained(save_dir)
card = render_model_card(model.peft_config["default"], repo_id="you/bicep-llama3")
Path(save_dir, "README.md").write_text(card, encoding="utf-8")
model.push_to_hub("you/bicep-llama3")          # README.md rides along

Load & generate

import bicep
from peft import PeftModel
from transformers import AutoModelForCausalLM

base  = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base, "you/bicep-llama3")
model.generate(input_ids=...)                  # no BiCEP-specific arguments

The collator is mandatory — causal integrity

BiCEP's enricher is bidirectional, so a prompt token's enriched embedding can encode later prompt tokens. To keep this from leaking into the loss, two things must hold, and BiCEPDataCollator enforces both:

1. Loss masking — prompt token positions get labels = -100, so loss is computed on response tokens only.
2. Prompt-only enrichment — the collator emits a bicep_prompt_mask that tells the model to enrich prompt positions only; response/generated tokens pass through as raw embeddings.

Using BiCEPDataCollator is not optional. Skipping it (or hand-rolling labels) silently breaks the causal structure and produces a model that cannot generalize.

Data formats

BiCEPDataCollator accepts either:

{"prompt": "Translate to French: Hello", "response": "Bonjour"}
{"text": "The cat sat on the mat ...", "prompt_length": 42}   # prompt_length in tokens

A record with no response tokens (empty response, or prompt_length ≥ the token count) raises ValueError.

Why remove_unused_columns=False?

Trainer drops dataset columns not in the model's forward signature before collation — which would remove the collator's raw prompt/response/text inputs. Set TrainingArguments(remove_unused_columns=False) so the collator receives them.

Scope (v1)

• Single-GPU training. Multi-GPU (DataParallel/FSDP/DeepSpeed) is out of scope.
• You load and manage the base model (dtype, quantization, device); BiCEP does not.
• Composing BiCEP with another PEFT adapter simultaneously is undefined.

Development

python -m venv .venv && . .venv/Scripts/activate    # or source .venv/bin/activate
pip install -e ".[dev]"
pytest                                               # CPU-only, uses tiny HF test models

License

BiCEP is licensed under the GNU General Public License v3.0 or later (GPL-3.0-or-later). See LICENSE for the full text.