cortexdb-connectors

Managed data connectors for CortexDB. Run them yourself, or let CortexDB host them for you on the Starter tier.

Each connector pulls from a third-party system (Slack, GitHub, Jira, …) and writes every event into CortexDB as a v1 experience.

Plan availability (audit BLK-3)

The free self-hosted path is always available; the managed dashboard path is what the pricing page calls "Starter+". The badge below appears on each connector's setup page for the same reason.

Install

# core only — no third-party SDK dependencies
pip install cortexdb-connectors

# with one source's SDK
pip install 'cortexdb-connectors[slack]'

# everything
pip install 'cortexdb-connectors[all]'

Requires Python 3.10+.

Auth

The connectors talk to the CortexDB v1 API. Two things you need:

• A PASETO bearer token (Authorization: Bearer …)
• An actor id that matches the token's sub claim (X-Cortex-Actor: …)

The fastest way to get both is to install cortexdb-cli and run cortexdb init — anonymous signup, no email or card, 7-day free-tier token. The connectors read ~/.cortexdb/state.json automatically.

Or set env vars:

export CORTEXDB_URL=https://api-v1.cortexdb.ai
export CORTEXDB_API_KEY=v4.public...
export CORTEXDB_ACTOR=user:u_019e...
export CORTEXDB_TENANT_ID=acme                            # optional assertion; verified via whoami
export CORTEXDB_SCOPE_TEMPLATE='org:acme/source:{source}'   # optional
export CORTEXDB_SOURCE_INSTANCE_ID='acme-slack-workspace'   # see below

{source} (and {scope}, {namespace}, {actor}, {entity.<type>}) get filled in per event. ({tenant} is still accepted as a legacy alias of {scope}.)

Self-hosted server with a static CORTEX_API_KEY: that key has no sub claim, and the server runs a request without X-Cortex-Actor as user:local. When CORTEXDB_ACTOR is unset, cortexdb-sync sync|watch and the Docker worker ask GET /v1/auth/whoami (without the header) and adopt user:local only when the server confirms a static key (0.2.22+). With a PASETO/JWT token the actor stays explicit and must equal the token's sub. When no actor can be determined the sync stops before fetching anything, with one error naming CORTEXDB_ACTOR, and the cursor is left untouched.

CORTEXDB_SOURCE_INSTANCE_ID is an optional stable, non-secret account/site identifier. Set it (or --source-instance-id) whenever two accounts of the same provider write to the same CortexDB endpoint, actor, and scope. Its digest partitions canonical source ids and the lifecycle ledger; never put an access token or other credential in it. The empty default preserves existing single-account identities. Connector-specific YAML may set source_instance_id under that connector as well.

Before it opens durable lifecycle state, the connector resolves the billing tenant from GET /v1/auth/whoami. CORTEXDB_TENANT_ID (or --cortex-tenant-id) is an optional expected value and must match that authenticated response; it never bypasses verification. Tenant identity is part of the opaque ledger namespace, so credentials for a different tenant cannot reuse completed source rows.

Run

# one shot
cortexdb-sync sync slack

# poll loop, 60s between cycles
cortexdb-sync watch slack --interval 60

# resume from a specific point
cortexdb-sync sync github --since 2026-05-01T00:00:00Z

# see what's available
cortexdb-sync list

# check resolved CortexDB creds
cortexdb-sync auth

# cursor state across every connector
cortexdb-sync status

# move a tenant's items onto containers, once (see "Container mode")
cortexdb-sync migrate-containers slack --dry-run

# give up an incomplete migration (its unmoved items stay listed)
cortexdb-sync migrate-containers slack --abandon

Available connectors

All 18 connectors are 🟢 Free to self-host and 🔒 Starter+ ($29/mo) to use via the managed dashboard.

slack            SLACK_BOT_TOKEN
github           GITHUB_TOKEN, GITHUB_REPOS
gitlab           GITLAB_TOKEN
jira             JIRA_URL, JIRA_EMAIL, JIRA_API_TOKEN   (incl. crash-safe keyed attachments + multimodal extraction, 0.2.10+)
freshdesk        FRESHDESK_DOMAIN, FRESHDESK_API_KEY    (incl. crash-safe keyed attachments + multimodal extraction, 0.2.20+)
tldv             TLDV_API_KEY
linear           LINEAR_API_KEY
confluence       CONFLUENCE_URL, CONFLUENCE_EMAIL, CONFLUENCE_API_TOKEN
notion           NOTION_TOKEN
pagerduty        PAGERDUTY_API_KEY
discord          DISCORD_BOT_TOKEN
teams            TEAMS_TENANT_ID, TEAMS_CLIENT_ID, TEAMS_CLIENT_SECRET
google-workspace GW_SERVICE_ACCOUNT_KEY, GW_DELEGATED_USER
salesforce       SF_INSTANCE_URL, SF_CLIENT_ID, SF_CLIENT_SECRET, SF_USERNAME, SF_PASSWORD
hubspot          HUBSPOT_TOKEN
zendesk          ZENDESK_SUBDOMAIN, ZENDESK_EMAIL, ZENDESK_TOKEN
intercom         INTERCOM_TOKEN
servicenow       SNOW_INSTANCE, SNOW_USERNAME, SNOW_PASSWORD

cortexdb-sync list prints the same required variables plus each connector's optional ones.

GitHub

export GITHUB_TOKEN=ghp_...                  # PAT or GitHub App installation token
export GITHUB_REPOS=acme/api,acme/web        # required: owner/repo, comma-separated (no discovery)
export GITHUB_EVENTS=pull_request,issues     # optional; default: all six kinds below
export GITHUB_BACKFILL_DAYS=30               # optional; first-sync history, 0 disables
cortexdb-sync sync github

• What is stored (0.2.22+): PRs as PR #n [state]: title plus the full description; issues with their full body; issue comments, review bodies and inline review comments (file, line and diff hunk) in full; pushes with the newest 20 commit messages and the total count. Nothing is truncated — text over 1 MiB is split losslessly into source-bound fragments.
• GITHUB_EVENTS kinds: push, pull_request, issues, issue_comment, pull_request_review, pull_request_review_comment. Raw Events API names (PullRequestEvent, PullRequestReviewCommentEvent, …) are accepted too, case-insensitively.
• Sources: PRs, issues, issue comments and review comments are read from the REST list endpoints on every sync (the first sync reaches back GITHUB_BACKFILL_DAYS, default 30, as does the first sync of a repository added to GITHUB_REPOS later); the repository Events feed, which GitHub caps at 300 events / 30 days, supplies pushes and reviews. Each record is keyed by the GitHub object — the same identity the webhook receiver uses — so a PR re-read by the poller or delivered by a webhook updates one record instead of adding another.
• Persist ~/.cortexdb between runs: it holds the cursor (with the work a run deferred — pushes and PR file lists / review listings held when the API budget ran low, and a new repository's backfill items that failed to ingest — retried on the next sync) and the lifecycle ledger that retires superseded versions (or point CORTEXDB_STATE_DIR at a volume for the ledger, and --state-file / the worker's CORTEX_SYNC_STATE_PATH for the cursor). A throwaway docker run --rm loses both.

Per-connector setup pages live at https://cortexdb.ai/docs/connectors/ (e.g. Slack). The "Add Connector" button on those pages is the managed (Starter+) path — the CLI on this page is the free path.

For deep self-hosted operation — auth lifetimes, cursor state, retries, OAuth setup for Salesforce/Teams/Google Workspace — see SELF_HOSTING.md in this directory.

YAML config

If env vars get unwieldy, drop a cortexdb-connectors.yaml next to your sync invocation:

slack:
  slack_bot_token: xoxb-...
  source_instance_id: acme-slack-workspace
  channels: [C01ABCDEF, C02GHIJKL]

github:
  github_token: ghp_...
  repos: [acme/api, acme/web]
  events: [pull_request, issue_comment]

Programmatic use

Skip the CLI entirely:

import asyncio
from cortexdb_connectors.slack import SlackConnector

connector = SlackConnector(
    cortex_url="https://api-v1.cortexdb.ai",
    cortex_api_key="v4.public...",
    scope="org:acme/source:slack",
    slack_bot_token="xoxb-...",
    channels=["C01ABC", "C02DEF"],
)
# actor + a richer scope_template are set after construction:
connector.bind(
    actor="user:u_019e...",
    scope_template="org:acme/source:slack/channel:{entity.channel}",
    source_instance_id="acme-slack-workspace",
    cortex_tenant_id="acme",  # optional assertion, verified with whoami
)

result = asyncio.run(connector.sync())
print(result.episodes_ingested, "ingested,", len(result.errors), "errors")

Webhooks

The cortexdb-connectors[webhooks] extra ships a Starlette-based webhook receiver for sources that push (Slack Events API, GitHub Apps, Jira webhooks, …) instead of polling. See cortexdb_connectors/webhooks.py.

Durable lifecycle ledger

Every canonical text or blob source is recorded in a local SQLite ledger before a sync cursor advances or a webhook is acknowledged. It stores only opaque source/version/idempotency digests, random upload tokens, and CortexDB event/blob ids—never source text or file bytes—and, in container mode, the scope each item's events live at, for an item a container migration has not moved yet its scopes, its raw external id when it failed before it normalized, and the last error, and for a copy left under another tenant root its scope and event ids. The default is ~/.cortexdb/connector_event_ledger.sqlite3 (or under CORTEXDB_STATE_DIR); override it with CORTEX_CONNECTOR_EVENT_LEDGER_PATH. The authenticated billing tenant is included in its opaque namespace and an optional configured tenant assertion must match /v1/auth/whoami.

Keep this database on durable local storage and back it up with SQLite's online backup API (or stop writers and copy the database together with its -wal and -shm files). A replica needs the same ledger to suppress replay beyond the server idempotency window and to erase every historical source version, so run one active connector writer per source namespace unless all writers share a correctly coordinated ledger. Same-source generation changes use bounded, crash-released OS advisory-lock stripes beside the SQLite file; network filesystems are supported only when they provide equivalent locking semantics. Accepted replacements retire older generations through exact derived-then-raw erasure before the sync cursor advances; a failed cleanup remains durable for retry. Rows are removed after confirmed source erasure or supersession; monitor disk usage as part of connector operations.

New binary attachments use only CortexDB's versioned keyed-blob protocol. The ledger commits a random, non-secret upload token before any blob network request, then preflights and uploads the bytes through /v1/blobs/idempotent. A timeout or restart reuses that same token and reconciles the authoritative blob ID, so it cannot create a second physical object. Deletion tombstones the token through the matching keyed endpoint before removing the local mapping. A missing, disabled, malformed, or older server implementation is a hard sync failure; connectors never fall back to an unkeyed upload. Existing blob-ID-only ledger rows from releases before keyed uploads remain deletable through the legacy ID route as a migration path, but no new rows are created that way. Binary payloads over the 32 MiB source ceiling, malformed descriptors, and invalid envelope metadata are rejected before whoami, ledger allocation, or upload. Text-shaped attachments are strictly decoded and losslessly fragmented at UTF-8 boundaries instead of taking the binary path.

Container mode (source permission units)

For a tenant that is an org tree, set CORTEXDB_CONTAINER_ROOT to its root (org:acme) and the Slack, Teams, Discord, Google Drive, Confluence and Notion connectors write each item into the scope of its source permission unit instead of the scope template, so who reads it follows the source's own ACL:

• a channel, folder, space or page tree is a container, org:acme/source:<system>/container:<key>; an item shared differently from its folder or page tree goes into a container keyed by the fingerprint of its ACL;
• a public unit is read through org:acme/set:<system>-members (the source's full members), a private one through its own set, and a DM or group DM is a private group org:acme/group:dm-<system>-<key> read by its participants;
• a source account reaches a roster only once the tenant directory links it to a person (POST /v1/access/accounts/resolve); one it does not is counted, and a unit whose ACL the connector could not read is a container nobody reads.

The directory links accounts per system, so link each person's account under the system its connector names:

The connector's actor (CORTEXDB_ACTOR, a service: principal) owns every record it writes, and must be one of the tenant root companion's access_sources (or an Owner of the root) with Reader at the root for GET /v1/access/limits. Each sync registers what changed through change sets, the narrowing one first; moves an item whose sharing changed (POST /v1/access/rehome; an event the server erased stays erased: it is counted, and never captured again); leaves out the containers past the source's containers_per_source limit (their items are not ingested, and are counted); and, when it changed something or the source has no watermark yet, proposes the source's watermark. The records the connector registered are kept with the cursor; a first sync, or one whose saved state was lost, reads them back from the registry (GET /v1/access/records) before it writes, so it never creates a record that exists, and a set an operator filled (set:<system>-apps, set:<system>-services) keeps its entries. cortexdb-sync status shows each source's counts. Each connector's module docstring says which ACLs its source API exposes and what it does when one is missing.

Items written before container mode. Within a tenant, a sync never captures an item fresh while a copy of it is still live at its template scope: such a copy would keep a private channel's content readable in the shared template scope, the leak containers exist to close. So every container-mode ingest of an item with a unit (a sync, a sync's retry of a listed item, a migration, or a direct ingest_episode_events or ingest_episode call) first checks whether the ledger still records the item under its template scope's namespace, whether or not a migration listed it, and if so moves it as migrate-containers does: POST /v1/access/rehome to its home and its ledger rows re-keyed, so the ingest finds it recorded. Sealed or expired events are erased at the template scope and captured again at the home, absent ones stay erased, and a DM message is captured at its group, then its template copy erased. A copy under another tenant root (a template such as user:default outside the container root) cannot move, since no rehome crosses tenants: the item is captured at its home and the copy is recorded, with its source id, its scope and the event ids the ledger holds there, for that tenant's operator to erase. cortexdb-sync status lists these records until the copy is erased through the connector or an operator clears them with cortexdb-sync migrate-containers <connector> --clear-other-tenant-copies [SOURCE_ID ...]. Clearing makes the ledger forget the copy and erases nothing at the server.

Raising containers_per_source. Items of a container left out at the cap are not ingested, and the cursor moves past them as past any other item, so they do not hold every later sync back. After the limit is raised, backfill them: cortexdb-sync sync <connector> --since <time of the oldest left-out item>. The source's watermark says how many containers it left out.

Webhooks. A Slack delivery carries no channel ACL, so the Slack webhook receiver writes to the scope template, and in container mode it would put private channels where the template's readers see them. With a container root and SLACK_SIGNING_SECRET set, cortexdb-sync serve exits with an error instead of starting, and a Slack handler bound to a root answers every delivery 503. Run the poll connector (sync or watch) for Slack instead. The GitHub, Jira, Freshdesk and tl;dv receivers serve sources that do not write containers, and run as before.

Moving a tenant onto containers

A tenant whose connectors wrote to the scope template moves onto containers once, with its owner, as a breaking change for its readers:

1. Make the tenant an org tree (its root registered with kind org, PUT /v1/scopes/tree). Walking views at the root, at sources and at containers then answer 422 VIEW_NOT_ALLOWED_AT_SCOPE, so move the tenant's clients to mounts or extra_scopes first.
2. Set CORTEXDB_CONTAINER_ROOT and preview the move: cortexdb-sync migrate-containers <connector> --dry-run. It reads the source, the limits and the registry, prints per home the items and events that would move, and writes nothing.
3. Before the first container-mode sync, run cortexdb-sync migrate-containers <connector> (with the --scope the syncs use, and --since as far back as the first sync when the connector's own backfill is shorter: it learns each item's unit from the source). It registers the units' records exactly as a sync does, then moves each item's events, the ids its ledger holds, to its home with POST /v1/access/rehome in batches of at most 1,024: the events keep their ids and derived rows, and nothing is captured again. The syncs that follow find every moved item recorded and post nothing; the first proposes the source's watermark.

What it does not move: an item whose events the server skips as sealed or expired is erased at the template scope and captured again at its home; a DM message whose author the directory resolves is captured again at its group, since its observed actor must be that person for a person's erasure to reach it; an absent event (already erased) is reported and never captured again; the items of a container left out at the cap stay where they are, listed (raise the limit: the next sync that reads them moves them, or run it again); items with no unit keep the template. A copy at an item's home beside its template copy (written by an earlier connector version's container-mode sync, or a DM message whose template copy could not be erased) makes the template copy a duplicate, which it erases. A template scope under another tenant root cannot be moved from: the run names each such copy (source id, scope, the event ids the ledger holds there) and records it as a sync does, and the next sync captures the item at its home. The rehome needs the connector's actor to be an Owner, with policy.administer.scope, of the template scope the items are at now as well as of the containers. A rerun moves nothing (the server answers unchanged).

The migration moves everything at once; a sync moves only the items it reads. An item the migration did not read (outside its window) is still moved, not captured beside its copy, by the first sync that reads it.

A run marks the sync state before it writes anything, and clears the mark only when it completes: every rehome answered, every moved item's ledger rows moved with it. While the mark is there, sync, watch and the worker refuse to start (an item whose rehome failed is still recorded at the template scope, and a sync would capture it a second time at its home), naming the command that finishes the run; cortexdb-sync status lists it. Run cortexdb-sync migrate-containers <connector> again: it resumes the window the interrupted run covered and clears the mark.

An item that fails on its own (it could not be read or planned, captured again, captured at its DM group, or its template copy could not be erased) does not hold the syncs: the run reports it under "failed" (its source id, or its raw external id when it failed before it normalized, its template scope and home, the stage and the error), exits 1, and keeps it listed in the ledger; cortexdb-sync status shows how many are listed and the first of them. The next sync that reads a listed item moves it the way the migration would have (rehome and re-keyed ledger rows, sealed or expired events erased and captured again at the home, absent ones kept erased; a template copy beside a copy at the home erased; a DM message captured at its group, then its template copy erased), never by a fresh capture beside its template copy. Once it has moved it leaves the list; if the move fails again it stays listed with the new error and is that sync's failure, holding the cursor as any failed item does. Listed items a sync does not read stay listed for a later sync or a rerun, and a rerun clears what it settles.

To give up an incomplete run instead, cortexdb-sync migrate-containers <connector> --abandon clears its mark so the syncs run again. It reads only the ledger and the sync state (nothing is fetched from the source or sent to CortexDB), logs how many items the run leaves at their template scopes and their ids, and keeps them listed, so status shows them and the next sync that reads each one moves it. Until then, an item of a private unit left at a pooled template scope stays readable there. Without an incomplete run it says so and changes nothing, and it does not combine with --dry-run.

License

Apache-2.0