OpenForage

Python library first run

Create a settings file, check the installed version and status, then register with an invite code supplied by the operator:

openforage init
openforage --version
openforage status --json
openforage register

Python callers can check the installed runtime with openforage.runtime_info().

OpenForage is a research and investment platform. Contributors submit Python search templates through the openforage SDK (software development kit). One Rust core runs each template on the contributor's machine and on the server, so both sides agree by construction. This repository holds the production system, rewritten in Rust.

What it does

The platform carries one pipeline. Every value in the pipeline is a T×N matrix: one row per time step and one column per instrument. Data moves in one order:

raw data -> cleaned raw data -> aggregation -> features -> signals -> strategies -> allocations -> book -> execution

A contributor writes a search template as a Python generator function and passes it to openforage.search(...). The template runs inside the contributor's own Python process. The numeric work runs in one Rust simulator, so a template that scores well on a laptop scores the same on the server. The non-negotiable P1, one implementation, makes that true: the client simulation and the server re-simulation call one code path, never two.

The platform then admits, evaluates, distributes and executes results:

• the signal roles accept submissions, validate them, run tamper checks and backtest them;
• the evaluation roles run admission batteries over candidate functions, features, aggregations, rotations, filtrations and reductions;
• the distribution roles publish the artifacts that agents consume;
• the contributor roles capture consent, ingest stats, materialise derivatives and read back earnings;
• the keeper runs the on-chain side;
• the web roles serve the landing page, the application, the documentation and the godeye cockpit.

Four non-negotiables carry through the whole rewrite:

Architecture

The code follows a strict shape, and rule 03 owns it. Three terms matter.

• A domain is one business area, such as market data, signals or contributors. Its code lives in crates/<domain>/<layer>/.
• A layer is one of six ordered stages inside a domain: types, config, repo, service, runtime, ui. A layer may depend only on earlier layers of its own domain. A domain may depend on another domain only through that domain's types crate.
• A provider is the single entry point for one cross-cutting concern, such as a database, an object store or a clock. The port is a trait crate; each backing is its own crate. A domain calls the port. It never names the software development kit behind the port.

flowchart TB
  T[types: data and contracts] --> C[config: typed settings]
  C --> R[repo: persistence and SQL]
  R --> S[service: business logic]
  S --> RT[runtime: composition root]
  RT --> U[ui: routes and rendering]
  S -.-> P[provider port]
  RT -.-> I[provider implementation]
  I --> P

Figure 1. A domain runs forward through six layers, and every cross-cutting call enters through a provider port that the runtime binds to one implementation.

Five patterns cover almost every need across domains, in this order: a data contract in the producer's types; a merge of two domains; a provider; a process boundary between roles; or a bound domain port. Anything else needs a founder decision, recorded as a DEC-nnn row and cited in an edges row of the root manifest.

Three zones sit outside the domain grid. Binaries live in apps/ and only wire domains together, with no business rules. Tools live in tools/ and may use kernels, ports, other tools and any domain's types. The telemetry capture core has no first-party dependencies, so every crate may emit typed events through it.

Repository map

This map lists every top-level entry of the tree at the head.

Key root files:

The ofg binary and roles

Every deployment carries one binary, ofg. A subcommand selects what runs. The tree has three kinds of subcommand:

• a role: a long-running service that serves or consumes in production;
• a job: a bounded command started by a timer or a batch system;
• a tool: a command a human or an agent runs by hand.

A host runs only the roles listed in /etc/ofg/roles.allow. A role outside the list refuses to start.

A role call flows from the command line to the domain code and out through a provider:

flowchart LR
  CMD[ofg role subcommand] --> APP[apps/ofg composition root]
  APP --> RT[domain runtime]
  RT --> SVC[domain service]
  SVC --> REPO[domain repo]
  SVC -.-> PORT[provider port]
  RT -.-> IMPL[provider implementation]
  IMPL --> PORT
  REPO -.-> PORT

Figure 2. A role command enters the composition root, which builds the domain runtime and binds the provider implementations that the service and repo layers use through ports.

Version control and the daemon

Agents never run git or jj. A daemon (a long-running local helper process), ofgd, owns the repository store and listens on a Unix socket. Agents reach it only through ofg vcs. The daemon exposes six verbs: workspace open, checkpoint, read, merge, export and events.

Four terms carry the flow:

• a workspace is a copy-on-write checkout that the daemon opens for one agent on one task;
• a checkpoint is a daemon-made commit on that agent's bookmark (a named commit pointer inside the daemon);
• the integration bookmark, integration/<project>, is the project's landing line;
• export pushes that line to main.

The rules of the flow:

1. The daemon opens a workspace as a snapshot of the integration checkout. The snapshot is cheap, so every agent gets its own.
2. The agent edits and checkpoints there. A checkpoint costs about a millisecond, and nothing waits on it. Writers never block writers: a collision is found after the acknowledgement and is stored as data.
3. A merge-arbiter (the subagent class that resolves a collision) lands each finished bookmark on integration/<project>. The merge runs the full gate table on an unconflicted result; a failing result does not land.
4. Export runs the same gate table on the integration line, then pushes main as a fast-forward. Export is the only path to main. An operator runs it, never the main session.
5. Several people and several machines share the one trunk. Each machine runs its own daemon. A worker daemon ships its bookmarks straight to the project's landing daemon by direct bundle sync, and only the merge host's pull-request merge moves main.

flowchart LR
  W[workspace per agent] --> CP[checkpoint on the agent bookmark]
  CP --> LG[merge lands on integration]
  LG --> EX[gated export pushes main]
  CP -.-> CN[collision stored as data]
  CN -.-> LG
  LG -.-> SYNC[bookmarks reach the landing daemon by bundle sync]

Figure 3. Work moves from one agent workspace to a checkpoint, lands on the integration line through a merge-arbiter, and reaches main only through a gated export.

The gate

A gate is one mechanical check over the tree. ofg gates runs the whole table. The table is the sole merge condition: the daemon refuses merge-to-integration and export while any row reports a finding. No reviewer reads code before a merge, and no approval exists beyond the gate.

The block #6 table, by row name:

Several further families run in the same aggregate:

• architecture rows, ARCH-001 to ARCH-024, enforce rule 03: layer order, provider edges, process rules and file size;
• sibling rows cover the database placement map, era literals, telemetry catalog, determinism, web tokens, build profiles and the environment doctor;
• the prose row checks human-facing text against ISO 24495-1 plain language, and the secrets row blocks credential-shaped strings.

The standard has no tests and no comments. Verification is a real command on real data, and its output is the evidence. The only suppression form is a scoped #[expect(<lint>, reason = "…")]; the gate counts every suppression, and a verifier reads the count before the code.

Two modes exist. Changed mode measures the files a packet touched. Full mode measures the whole tree and runs at every merge and export. Every finding prints one line with the file, the line, what failed, the fix and the owning rule. Fix the code, never the row.

The agent harness and the swarm

The repository is built for agent work. Three layers carry it:

• Rules state invariants. The numbered files under rules/ are short, indexed in the boot digest, and bind every session.
• Skills hold procedure. A skill loads when its trigger fires, so detail stays out of context until an agent needs it.
• Hooks run one binary, ofg hook, for all three harnesses (Claude Code, Codex and OpenCode). Session start injects the field-guide indexes, the environment verdict and unread reports. A pre-tool hook blocks source edits from the main session and blocks git and jj.

One manifest, harness/manifest.toml, generates every harness file: the boot digest, the projections and the tool configuration. A check command fails on drift, so the generated views never become a second source of truth.

The work runs as a swarm. A middle manager is a session that spawns and coordinates subagents; it never edits code itself. Eight subagent classes come from one roster:

Each unit of work is a packet: one task handed to one subagent. The packet uses an eight-field format: objective, scope allowlist, acceptance lines, rules digest, interfaces, local context, verification command and handoff fields. A packet names one endpoint and one small bound of work, so a subagent cannot wander. An independent verifier re-checks every merged unit.

The local orchestrator never does leaf work. It plans, mints packets, dispatches every ready packet at once and coordinates landings. A landing round is a batch of finished work that one arbiter lands on the integration line after review:

flowchart TB
  LO[local orchestrator] --> M1[middle manager lane]
  LO --> M2[middle manager lane]
  M1 --> A1[implementer subagents]
  M1 --> A2[verifier subagents]
  M2 --> A3[implementer subagents]
  M2 --> A4[verifier subagents]
  A1 --> LR[landing rounds]
  A2 --> LR
  A3 --> LR
  A4 --> LR
  LR --> HEAD[integration head]

Figure 4. The local orchestrator runs middle-manager lanes, each lane spawns implementer and verifier subagents, and landing rounds carry finished work to the integration head.

Knowledge lives in two field-guide levels, repository and project. Every finish note goes to an inbox (the fieldguide/_inbox directory, where each subagent writes one note at close-out). A consolidator folds those notes into the guides at each wave close-out and once an hour. A question that research cannot settle goes to the founder through the harness's question tool, and the answer becomes a DEC-nnn row in the ledger. A hard gate (an action that needs the founder's explicit approval) uses a bound native approval receipt for that exact action.

Build, run and prove

First, check the machine. Run ofg doctor. It reads one required-tool table, checks every floor and prints one line when green. When a tool is missing and the agent has permission, ofg doctor --fix installs it and writes one note. Without permission, the command prints one block naming each missing item and its install command.

Build. The root Cargo.toml defines three profiles: dev for edit loops, research for optimised runs, and release for deploys. A heavy command enters a shared host slice through ofg build run --, which caps memory and never multiplies the cap. The committed .cargo/config.toml sends every compiler call through sccache, a shared compiler cache, and caps jobs at 8. A plain cargo build works for a workstation check.

Run. Start a role with its subcommand, for example ofg platform, ofg signal-api or ofg web --profile landing. The journey ledger defines runtime configuration refusals (JRN-PLAT-156); rule 24 governs the contract. ofg config check --role <role> checks the required set for one role. Tools run the same way: ofg parity …, ofg journey run …, ofg db migrate ….

Prove. The repository holds no tests, by design. Proof takes four forms:

1. The gate. ofg gates lint on the merged tree, in full mode, is the merge condition.
2. Parity. The Rust system must equal the old Python system on three canonical real-data subsets (hltop100, hltop20 and market 41017). Floating outputs match within atol + rtol·|old| with both tolerances at 1e-12; wire formats, manifests and codecs match byte for byte. Rust output is also bit-identical across runs, threads and machines.
3. Journey runs. A journey run is a browser-driven check of one user journey against a test stack (a per-project copy of the production services on virtual machines). The journey ledger defines browser run evidence (JRN-REL-067); rule 24 governs the contract.
4. The clean room. The SDK ships as fifteen wheels (cp311, cp312 and cp313 across five platforms), built with maturin, the tool that builds a Python wheel from a Rust crate. The clean room installs a candidate wheel on a throwaway machine without a checkout or build context. The journey ledger defines the wheel campaign (JRN-PARITY-053); rule 24 governs its scope.

Evidence is telemetry, one run log per verification under projects/<slug>/runs/, and the project's summary.html. Nothing else counts, and a command that cannot run says so.

Failures, retries and waits

Every failure belongs to one of five classes. The library names the class in the message, the status row and the event. One renderer writes that text, so the CLI, this file and the library agree on the words.

The library fault budget is 10 attempts. The tenth consecutive library fault stops that one activity. Its queued work stays on disk, and the supervisor retries it after a 3600 s cool-down.

The ladder counts consecutive failures for each activity and host. A wait sleeps in chunks of at most 5 s, checks for a stop between chunks and records a heartbeat on each chunk.

Each wait carries plus or minus 20 percent jitter, and a wait already at the 3600 s cap carries none. A graceful stop finishes within stop_wait_seconds (30).

A server can state how long to wait. A usable server-stated wait is honoured exactly, without jitter, and capped at 3600 s (one hour). A stated wait that the library cannot use as a number follows the ladder above. A success from the same server ends a stated wait early, once. After that, each stated wait of the activity is honoured in full until its own request succeeds. A success from another server never ends a stated wait. A 403 ban is the one exception to the cap and to the early end: the activity waits until the ban's stated end, however long, and sends no request before it.

Every error, status row and failure event carries the same six facts in this order. One message from the renderer looks like this:

server gave no usable answer; class: server_error; retry: without limit on the ladder; next: 2025-08-12T12:00:10.000Z; server: none; fix: nothing: the library recovers by itself

• what failed;
• class: one of server_error, server_verdict, library_fault, local_environment or local_verdict, or none for a status row that holds work without a failure: an idle search or an upload the settings opt out of;
• retry: the plan, one of:
  • without limit on the ladder;
  • attempt N of 10;
  • stopped after 10 attempts, retried hourly by the supervisor;
  • paused: <local cause>, resumes by itself without limit;
  • stopped at once: <auth reason>;
  • dropped after 10 attempts;
  • none: rejected;
  • waits on the sync role without limit on the ladder, never spending the library budget;
  • for a direct call that stops before or around its durable write, its outcome: NOT saved; input payload retained; retry the same call after the SQLite lock is released; submission id <id> (or no submission ID was allocated before store open when the call supplied no submission id), NOT saved; the same call can be repeated unchanged, the same call can be repeated: it keeps its idempotency key, so a submission saved before Ctrl-C is stored once or saved and pending: the worker's journal replay writes it to the store <store> once the store answers, or the next call on this data directory does;
• next: the UTC time of the next attempt, or none;
• server: the server's error code and stated wait when present, else none;
• fix: what you can do, or nothing: the library recovers by itself.

Run openforage help recovery to print this policy. The CLI routes that command to the same renderer, so the help, this file and the library keep one voice.

Further reading

Sources

Every claim above comes from one of these files, read at the head:

• migration/spec/00-overview.md (product scope, pipeline context, operating model)
• migration/spec/07-services-and-deployment.md (role, job and tool vocabulary, role table)
• migration/source/00-commission.md (the founder's commission)
• projects/migration/spec.md (the migration record)
• rules/01-source-of-truth.md, rules/02-non-negotiables.md (P1, P2, P3, P4)
• rules/03-architecture.md (domains, layers, providers, zones, patterns)
• rules/04-gates.md (the block #6 table, modes and fix discipline)
• rules/08-orchestrator.md (the swarm, packets, classes, founder questions)
• rules/09-harness.md (the manifest, projections, hooks, the roster)
• rules/10-vcs.md (the daemon, six verbs, bookmarks, landing, export, transport)
• rules/11-fieldguide.md (the two guide levels, the inbox, consolidation)
• rules/12-environment-and-build.md (profiles, sccache, job cap, capped scopes; ofg doctor and the required-tool table)
• rules/13-aesthetic.md (plain language and human-facing text)
• rules/16-clean-room.md (clean room, fifteen wheels, journeys J1 to J14)
• rules/18-investment-process.md (the T×N pipeline and the domain map)
• harness/manifest.toml, harness/roster.toml, harness/fleet/README.md (generated projections, classes, fleet loop)
• skills/plain-language/SKILL.md, skills/visualize/SKILL.md (writing and proof-page procedure)
• skills/contributor-pipeline/SKILL.md (the contributor chain)
• apps/ofg/src/cli/command.rs (the live subcommand tree)
• The top-level tree at the head, listed with ls, for the repository map.