Skip to main content

okfquery

SQL over an OKF v0.2 bundle, via DuckDB. okfquery loads a published bundle into an in-memory DuckDB connection and hands the connection back — no daemon, no cache, and no committed artifact except the navigation index.md you opt into with okfquery index. It depends on no kbforge code at runtime, so it reads any OKF v0.2 bundle, kbforge-built or not.

Install and use

uv pip install kbforge-okfquery   # or: pip install kbforge-okfquery
okfquery query "select count(*) from concepts" --bundle path/to/bundle
okfquery shell --bundle path/to/bundle

query runs one SQL statement and prints the result (--format table|json|csv). shell writes a temporary .duckdb file, opens the duckdb CLI on it, and deletes the file when the shell exits.

index: a front door for agents

okfquery index --bundle path/to/bundle             # writes path/to/bundle/index.md
okfquery index --bundle . --group-by owner         # sections by a facet, not by type
okfquery index --bundle . --check                  # writes nothing; exit 1 if stale

An agent reading a bundle cold starts at the root index.md (OKF §8, progressive disclosure): one line per concept, * [Title](path) - description, grouped under one heading per type (or per --group-by facet; concepts without it go last). It then opens only the concepts it needs. Without an index, its first step is listing concepts/*/overview.md and opening files to learn what they are.

  • Derived, never edited. The file starts with a generated-by marker; a hand-written index.md is refused unless you pass --force.
  • Deterministic. No timestamps, total ordering: rerunning on an unchanged bundle rewrites nothing, so CI commits only when a listing changed.
  • No frontmatter, not even the optional root okf_version: a reserved name that opens a fence is a concept, and the index would count itself.

Run it after merge, not per kbforge run. Each kbforge system publishes on its own sync branch; a whole-bundle file on several branches would collide. Generate it on the bundle repo's default branch instead. The sync branches never touch it, so they inherit the latest one.

Both jobs share three properties, each learned from running them on a real forge:

  • They wait for the first concept. A bundle repo is created before anything merges, and okfquery index rejects a bundle with no concepts/ on purpose (that is how a wrong --bundle shows). The jobs skip until one exists.
  • They index the branch as it is now. A cold start merges chunks in quick succession; a job that indexed its own commit and pushed would be rejected when a later merge had already moved the branch. They fetch the tip, index it, and retry the push on a race, one job at a time.
  • The bot's commit starts nothing: GITHUB_TOKEN pushes do not trigger workflows, and GitLab honours [skip ci].

GitHub Actions:

on: { push: { branches: [main] } }
permissions: { contents: write }
concurrency: { group: okfquery-index }   # one run at a time; the newest queued run wins
jobs:
  index:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: |
          git config user.name okfquery && git config user.email okfquery@users.noreply.github.com
          # Index the branch as it is NOW, not the commit that started the run: a
          # merge that lands meanwhile would make this push stale. Retry on a race.
          for attempt in 1 2 3; do
            git fetch -q origin "$GITHUB_REF_NAME" && git reset -q --hard FETCH_HEAD
            # Nothing to index until the first concept merges.
            [ -d concepts ] || { echo "no concepts/ yet; nothing to index"; exit 0; }
            uvx --from kbforge-okfquery okfquery index --bundle .
            git add index.md && git diff --cached --quiet && exit 0
            git commit -qm "chore: regenerate index.md"
            git push -q origin "HEAD:$GITHUB_REF_NAME" && exit 0
            echo "push raced another change; retrying on the new tip"
          done
          exit 1

GitLab CI. The script is one block scalar on purpose: a list item such as - git commit -m "chore: ..." contains : , which YAML reads as a mapping, and GitLab rejects the whole file. The push uses the job's own token, so no personal token is stored anywhere; enable it once under Settings → CI/CD → Job token permissions → Allow Git push requests to the repository (GitLab 17.2+), or via the API with ci_push_repository_for_job_token_allowed=true:

index:
  image: ghcr.io/astral-sh/uv:python3.12-bookworm-slim
  rules:
    # Default branch only, and only once a concept has merged: before the first
    # merge there is nothing to index, and `okfquery index` rejects a bundle
    # with no concepts/ directory on purpose (it is how a wrong --bundle shows).
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      exists: ['concepts/**/*.md']
  resource_group: okfquery-index   # one index job at a time
  script:
    - |
      apt-get update -qq && apt-get install -yqq git
      remote="https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
      # Index the branch as it is NOW, not the commit that started the job: a
      # merge that lands meanwhile would make this push stale. Retry on a race.
      for attempt in 1 2 3; do
        git fetch -q "$remote" "$CI_COMMIT_BRANCH" && git reset -q --hard FETCH_HEAD
        uvx --from kbforge-okfquery okfquery index --bundle .
        git add index.md && git diff --cached --quiet && exit 0
        git -c user.name=okfquery -c user.email=okfquery@example.invalid commit -qm "chore: regenerate index.md [skip ci]"
        git push -q "$remote" "HEAD:${CI_COMMIT_BRANCH}" && exit 0
        echo "push raced another change; retrying on the new tip"
      done
      exit 1

On an older GitLab, push with a project access token instead: store it as a masked CI variable and use oauth2:${YOUR_TOKEN_VARIABLE} in place of gitlab-ci-token:${CI_JOB_TOKEN}. Both jobs, exactly as printed here, were run against a real forge: a cold start with no concepts, then two changes seconds apart, with every run green and one index listing both.

If the default branch is protected against bot pushes, run okfquery index --check in the default-branch pipeline instead, and when it fails regenerate index.md in a small follow-up change of its own. Do not put the check on merge requests: kbforge's sync branches never touch index.md, so every request that adds or retitles a concept would fail it, and fixing that on the sync branches makes several systems edit one whole-bundle file, the collision this job exists to avoid.

okfquery related concepts/gaps/redundant-supply/overview.md --bundle .
okfquery related concepts/reports/q3/overview.md --by tags,scenario --limit 5

Three sections, each line in index's * [Title](path) - description format:

  • Links to: the concept's links. A target not in the bundle is listed by its path.
  • Linked from: concepts whose links name it. No single file shows these.
  • Shares: concepts sharing at least one value of the --by facets (default tags), most shared values first, then by path, each naming what it shares ((shares tags: 800v, sic)). Concepts already listed above are left out, and --limit (default 10) caps this section only.

An empty section prints (none), so "nothing" reads differently from "not checked". These relations are derived when asked, not stored: OKF §3.1 leaves tag views to the consumer, and a derived view cannot go stale. A path with a leading / (OKF §6.1's bundle-absolute form) or ./ is accepted, in the argument and in links.

Schema

$ okfquery schema
CREATE TABLE concepts (
    path          VARCHAR NOT NULL,
    type          VARCHAR,
    title         VARCHAR,
    description   VARCHAR,
    generated_by  VARCHAR,
    -- TIMESTAMPTZ, never TIMESTAMP. Not because a naive column would corrupt
    -- the instant on this package's load path -- `load` binds an aware Python
    -- datetime through executemany, so the instant survives a naive column
    -- fine. What a naive column loses is the type: values come back with no
    -- tzinfo, so every comparison against now() (itself TIMESTAMPTZ) needs a
    -- cast, and the column asserts UTC by convention with nothing recording
    -- that it does. §4.4 law 4 exists to force an *aware* stamp; a column that
    -- cannot hold one throws away exactly the property the law buys.
    generated_at  TIMESTAMPTZ,
    facets        JSON,
    body          VARCHAR
);

CREATE TABLE sources (
    path          VARCHAR NOT NULL,
    -- A real column. kbforge's synthesize.assemble puts the OWNING anchor first
    -- and grounding anchors after it, and that ordering is the only thing
    -- telling owner from ground -- OKF has no field for it. Discarding ordinal
    -- during unnest would destroy the signal.
    ordinal       INTEGER NOT NULL,
    id            VARCHAR,
    resource      VARCHAR,
    content_hash  VARCHAR
);

CREATE TABLE links (
    path          VARCHAR NOT NULL,
    target        VARCHAR NOT NULL
);

CREATE TABLE problems (
    path          VARCHAR NOT NULL,
    kind          VARCHAR NOT NULL,
    detail        VARCHAR NOT NULL
);

path is the bundle-relative concept path and the join key throughout. sources.ordinal = 0 is the owning source for a kbforge-produced bundle (the convention synthesize.assemble writes) — on a foreign bundle it is merely source order, with no error to tell you that. facets is one JSON column, open by construction, so facets->>'owner' reaches whatever a connector attached. problems is additive: a file that failed to parse still gets a concepts row with NULLs for what could not be read, plus one problems row per distinct failure — nothing a broken concept says gets silently dropped from an audit.

Queries, run against the messy test fixture

All four queries below were run as shown against packages/okfquery/tests/fixtures/messy — 9 .md files under concepts/, 7 of them concepts (concepts/index.md and concepts/log.md are reserved and fenceless, so they are skipped; concepts/claims/index.md is reserved-named but carries frontmatter, so it counts).

What is built on a given upstream document:

$ okfquery query "
SELECT c.path, s.ordinal
FROM concepts c JOIN sources s USING (path)
WHERE s.resource = 'https://wiki/ok';
" --bundle packages/okfquery/tests/fixtures/messy
┌─────────────────────────┬─────────┐
│          path           │ ordinal │
│         varchar         │  int32  │
├─────────────────────────┼─────────┤
│ concepts/ok/overview.md │       0 │
└─────────────────────────┴─────────┘

Staleness by owning system (ordinal = 0 picks the owning source, not a grounding one — see the ordinal note above):

$ okfquery query "
SELECT split_part(s.id, ':', 1) AS system,
       count(*) AS concepts,
       min(c.generated_at) AS oldest
FROM concepts c JOIN sources s USING (path)
WHERE s.ordinal = 0
GROUP BY 1 ORDER BY oldest;
" --bundle packages/okfquery/tests/fixtures/messy
┌─────────┬──────────┬──────────────────────────┐
│ system  │ concepts │          oldest          │
│ varchar │  int64   │ timestamp with time zone │
├─────────┼──────────┼──────────────────────────┤
│ wiki    │        3 │ 2026-08-23 00:00:00+00   │
└─────────┴──────────┴──────────────────────────┘

Only one system shows up on this fixture: most of the messy bundle's files exist to exercise a problems kind and never got far enough to have a sources entry at all, so they don't join into this query — which is correct, not a bug in the query.

Orphans — nothing links to them:

$ okfquery query "
SELECT c.path FROM concepts c
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.target = c.path);
" --bundle packages/okfquery/tests/fixtures/messy
┌───────────────────────────────────┐
│               path                │
│              varchar              │
├───────────────────────────────────┤
│ concepts/badyaml/overview.md      │
│ concepts/claims/index.md          │
│ concepts/incomplete/overview.md   │
│ concepts/nofence/overview.md      │
│ concepts/notmapping/overview.md   │
│ concepts/unterminated/overview.md │
└───────────────────────────────────┘

Everything that failed to parse (--format csv here, since the default table renderer truncates long detail strings):

$ okfquery query "SELECT path, kind, detail FROM problems ORDER BY path, kind;" \
    --bundle packages/okfquery/tests/fixtures/messy --format csv
path,kind,detail
concepts/badyaml/overview.md,invalid-yaml,frontmatter is not valid YAML: ParserError
concepts/incomplete/overview.md,bad-links,'links' entry 3 is not a string
concepts/incomplete/overview.md,bad-sources,'sources' entry 0 has no 'resource' (§5.1 requires one)
concepts/incomplete/overview.md,bad-timestamp,'generated.at' 'not-a-date' is not an ISO-8601 datetime
concepts/incomplete/overview.md,missing-required,"missing required OKF keys: title, description"
concepts/nofence/overview.md,no-frontmatter,file does not open a '---' frontmatter fence
concepts/notmapping/overview.md,frontmatter-not-mapping,"frontmatter parses to list, not a mapping"
concepts/unterminated/overview.md,unterminated-frontmatter,frontmatter fence is opened but never closed by a closing '---'

The mirror, and why both obvious joins are wrong

--mirror <path> attaches a mirror view over a kbforge mirror directory (read_json_auto('<mirror>/*.json')) — the same CanonicalDocument slots mirror.slot_key writes, anchor struct included. It answers a different question than the bundle alone can: not "what does this concept say" but "does it still say what its source currently does."

Both of the joins you'd reach for first are lossy, in different ways:

  • sources.id = mirror.doc_id holds for every shipped connector, but synthesize._source_entry builds id from anchor.system/anchor.native_id while doc_id is the mirror's own field — nothing enforces the two agree for a third-party connector. Where they diverge, this join produces a silent false miss, not an error.
  • sources.content_hash = mirror.anchor.content_hash as an equi-join is not the sound alternative either. kbforge.pipeline.run calls commit(mirror_path, docs) right after kbforge_publish, and publishing only opens a review request — kbforge never merges. Between publish and merge the mirror already holds the new hash while the bundle still carries the previous run's, so an equi-join on hash returns zero rows for every concept with a review request open — exactly the ones an audit most wants to see.

The correct form: join on id with a LEFT JOIN, not an inner join, then compare hashes instead of joining on them — an inner join silently drops every source with no mirror row at all, which is exactly the "never published" case the query needs to report. A match means current; a mismatch means an update is sitting in review; a source with no mirror row was never published. Run for real against the messy fixture (concepts/ok/overview.md's two sources — an owning wiki:ok and a grounding notes:ok — plus two other concepts whose sources have no mirror row at all) with a synthetic two-document mirror covering only wiki:ok and notes:ok, one hash left matching and one changed to simulate an update stuck in review:

$ okfquery query "
SELECT s.path,
       s.id,
       s.content_hash AS bundle_hash,
       m.anchor.content_hash AS mirror_hash,
       CASE WHEN m.doc_id IS NULL THEN 'never published'
            WHEN s.content_hash = m.anchor.content_hash THEN 'current'
            ELSE 'update in review' END AS status
FROM sources s LEFT JOIN mirror m ON s.id = m.doc_id
ORDER BY s.path, s.id;
" --bundle packages/okfquery/tests/fixtures/messy --mirror /path/to/mirror
┌─────────────────────────────────┬─────────────────┬─────────────┬─────────────┬──────────────────┐
│               path              │       id        │ bundle_hash │ mirror_hash │      status      │
│             varchar             │     varchar     │   varchar   │   varchar   │     varchar      │
├─────────────────────────────────┼─────────────────┼─────────────┼─────────────┼──────────────────┤
│ concepts/claims/index.md        │ wiki:claims     │ h-claims    │ NULL        │ never published  │
│ concepts/incomplete/overview.md │ wiki:incomplete │ NULL        │ NULL        │ never published  │
│ concepts/ok/overview.md         │ notes:ok        │ h-ground    │ h-ground-v2 │ update in review │
│ concepts/ok/overview.md         │ wiki:ok         │ h-own       │ h-own       │ current          │
└─────────────────────────────────┴─────────────────┴─────────────┴─────────────┴──────────────────┘

That does not fix the id gap above: a connector whose anchor disagrees with its doc_id still misses this join silently. There is no workaround for that here — only the honest statement of it.

Not built

  • No full-text search out of the box. body is a plain column; INSTALL fts and an index over it is a documented recipe, not a feature.
  • No remote bundles. --bundle is a local checkout path; reading one over HTTP wants a caching story this package deliberately avoids.
  • No MCP server. load() is already small enough to wrap in one query tool and a schema resource; nothing here does that yet.

Metadata

Release files for kbforge-okfquery 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kbforge-okfquery 0.3.0
File Size Uploaded Provenance Actions
kbforge_okfquery-0.3.0.tar.gz 31.4 kB With provenance Details for kbforge_okfquery-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kbforge-okfquery 0.3.0
File Interpreter ABI Platform Actions
kbforge_okfquery-0.3.0-py3-none-any.whl Python 3 none any Details for kbforge_okfquery-0.3.0-py3-none-any.whl

Total release size: 54.1 kB

Release files / kbforge_okfquery-0.3.0.tar.gz

Download URL kbforge_okfquery-0.3.0.tar.gz
Size 31.4 kB
Tags Source
SHA-256 checksum
How to use checksums
2ee8a85b552f0d8cdb300925c46d79ecc1645cd9733497bca9fd2f1454e84b11
BLAKE2b-256 checksum
How to use checksums
665e6f07728d55d5f57490e0eae31d174c0926f87c7964527e698e80e051150f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.

Transparency log

Release files / kbforge_okfquery-0.3.0-py3-none-any.whl

Download URL kbforge_okfquery-0.3.0-py3-none-any.whl
Size 22.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ce2f0095b5206a57775339fc0df0a9f062c334b81d587667f596f7d7e3d2c293
BLAKE2b-256 checksum
How to use checksums
33792b797887749141130bc6f3ee7d64086ca2a22ffc5229a925ed06cdc0fd17
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.

Transparency log

Security for kbforge-okfquery 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for kbforge_okfquery-0.3.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page