pocketsync

Sync Pocket voice-recorder recordings, transcripts (STT) and summaries from the Pocket cloud into MongoDB (with vector search) and optional local files - a durable landing zone that culture-rules workflows fan out to anywhere.

What it does

Pocket records audio and produces transcripts, summaries and action items in its own cloud. pocketsync copies that data into a store you own:

• MongoDB (one document per recording, with enrichment and vectors), and
• local disk (an optional directory tree plus the events.jsonl outbox).

That is the whole job. pocketsync does not send the data anywhere else. culture-rules workflows pick it up from the landing zone and route it onward (Notion, the Culture mesh, email, eidetic memory, …). The landing-zone layout and the event stream below are the stable contract they build against.

Non-goals: re-running speech-to-text (Pocket's transcript is the source of truth), routing data downstream, and any UI.

Status

Built (0.11.0); live deploy pending. Anything marked (planned) below is not on disk yet. The build brief is issue #1. The operator deploy and acceptance run are issue #3.

Pocket API access uses the official API. Pocket has an official public API at public.heypocketai.com/api/v1/public/*, authenticated with a static Authorization: Bearer pk_... key (Settings → Developer → API Keys), plus signed personal webhooks. The earlier reverse-engineered app.heypocket.com route with an hourly Firebase ID token is superseded and not used: Pocket's Terms of Service bar automated access "for unauthorized purposes". See the spec in docs/specs/2026-10-14-pocket-to-mongodb-near-realtime.md.

Audiences

• Operator (whoever runs pocketsync): deploys the Mongo stack, sets the secrets below, runs pocketsync serve (sync and enrichment), and reads pocketsync status. Start with the operations guide.
• Consumers (culture-rules workflows and Culture mesh agents): read the stores and react to the events. The recordings document, sync_state and event sections below are the public contract. Treat changes to them like API changes. Consumers never call Pocket and never need the Pocket or webhook secrets.

CLI

Write verbs are dry-run by default and commit only with --apply. Every verb takes --json.

Secrets come from the environment, and only from the environment. If a runtime verb (auth, list, sync, watch, serve, status, reembed, doctor) finds one unset, and the operator's grant store has a secret of exactly that name, for example POCKETSYNC_MONGO_URI, pocketsync re-runs itself under grant run --inject. So a bare pocketsync status works on the serving host even though the secrets are hidden. Set POCKETSYNC_NO_GRANT=1 to turn this off.

Sync is incremental and idempotent. A re-run with no remote changes does nothing. A content hash detects when Pocket re-processes a transcript or summary, and that recording is re-synced.

Catch-up

Catch-up is built in, so nothing has to be run by hand after downtime or an upgrade. Every reconcile pass (serve, watch and sync) picks one of three modes:

• Incremental (normal): lists from the last good pass's watermark minus a 2-day overlap. The watermark only advances after a clean pass, so downtime of any length is covered by the first pass after restart.
• Initial: there is no watermark yet (a fresh install). The pass backfills everything since POCKETSYNC_BACKFILL_SINCE.
• Upgrade: the stored landing_schema (in sync_state) is older than the running code's. The pass re-lands every recording since POCKETSYNC_BACKFILL_SINCE once, even if Pocket did not change it.

A failed catch-up pass is retried by the next pass. Webhooks Pocket could not deliver while the receiver was down are covered by the same passes, and webhooks that were acknowledged but not handled are replayed from the durable inbox. Audio is downloaded once per recording; if it is missing (--no-audio, or a failed download), a later pass with audio fetches it.

Data contract

Two stores hold the data. Both are written only by pocketsync; consumers read.

Mongo (primary)

Database pocketsync (default) on the MongoDB Atlas Local stack in deploy/mongo/. Collections:

The recordings document

Fields come from pocketsync/sinks/mongo.py. Timestamps are ISO-8601 UTC strings.

Behavior consumers can rely on:

• Pocket's raw payload and fetch time are not stored.
• A content change rewrites the content group, sets enrichment.status to pending, resets attempts to 0 and removes every derived enrichment field, so a stale vector never describes new content.
• Enrichment is written compare-and-set on content_hash: a result computed from superseded content is dropped.
• Enrichment heals itself after a gateway outage.
  • A lane that is down or loading (404, 429, 5xx, unreachable) keeps the recording pending; it is retried with backoff.
  • Any other failure marks it error, with attempts and next_attempt_at, and it is retried automatically from 1 minute up to every 6 hours.
  • After 10 failures it stays error. pocketsync status counts enrichment_retrying and enrichment_failed, and doctor warns.
• Deleted and merged recordings are tombstoned or marked, never purged. Filter on deleted_in_pocket and merged_into when querying.
• Mongo stores file references only, never audio bytes.

Example, with synthetic values and truncated vectors:

{
  "_id": "rec_0001",
  "title": "Weekly sync",
  "description": "",
  "recording_at": "2026-01-15T09:30:00Z",
  "duration": 1260.5,
  "language": "en",
  "state": "completed",
  "folder_id": null,
  "tags": ["work"],
  "speakers": ["Speaker 1", "Speaker 2"],
  "segments": [
    {"start": 0.0, "end": 4.2, "text": "Let's get started.",
     "speaker": "s1", "speaker_name": "Speaker 1"}
  ],
  "summary": "- Agreed to ship on Friday.",
  "action_items": [{"text": "Send the release notes", "done": false, "owner": null}],
  "source": {"created_at": "2026-01-15T09:30:05Z", "updated_at": "2026-01-15T10:02:11Z"},
  "content_hash": "<64 hex chars>",
  "files": {
    "transcript_json": {"rel_path": "2026/01/rec_0001/transcript.json", "size": 812,
                        "sha256": "<64 hex chars>"},
    "transcript_md": null, "summary_md": null, "audio": null, "meta": null
  },
  "landed_at": "2026-01-15T10:03:00Z",
  "updated_at": "2026-01-15T10:03:00Z",
  "deleted_in_pocket": false,
  "deleted_at": null,
  "merged_into": null,
  "merged_at": null,
  "enrichment": {
    "status": "done",
    "attempts": 0,
    "model": "example-embed-model",
    "dims": 768,
    "labels": ["meeting"],
    "filters": {"has_action_items": 0.82, "sensitive": 0.07},
    "taxonomy_version": "1",
    "summary_vector": [0.012, -0.034, "…768 floats"],
    "chunks": [
      {"index": 0, "start": 0.0, "end": 60.0, "text": "Let's get started. …",
       "vector": [0.021, 0.007, "…768 floats"]}
    ],
    "enriched_at": "2026-01-15T10:03:20Z",
    "error": null
  }
}

The enrichment worker writes filters from the lobes decider as {filter_name: confidence}, a probability from 0 to 1. The default taxonomy asks two filters:

• has_action_items: the recording contains action items for the operator.
• sensitive: the recording is private (health, money, relationships, credentials) and should not leave the operator's own stores.

A filter the decider did not answer is absent, not 0. Treat an absent filter as unknown. In particular, route a recording to a shared destination only when sensitive is present and low.

Vector search

pocketsync/vindex.py creates two Atlas vectorSearch indexes on recordings, both cosine similarity, 768 dimensions by default:

A model/dimension guard refuses to mix vector spaces. Both indexes were verified against a real atlas-local 8.0.34 (tests/integration/). On the nestedRoot chunk index, recording-level filters such as enrichment.labels must go in $vectorSearch's parentFilter, not filter; under filter they silently match nothing. Regular indexes cover enrichment.status, deleted_in_pocket and recording_at.

The sync_state collection

A single document, {_id: "state"}, from pocketsync/state.py:

webhook_dedupe makes webhook handling idempotent: inserting an existing key fails on _id, which reports the delivery as a duplicate. Each entry is also a durable inbox record (status pending, done or failed, plus attempts) until the event is handled.

Local files (optional sink)

Same content as a directory tree, per pocketsync/sinks/local.py:

<root>/
  events.jsonl                     # the event outbox (below)
  events.jsonl.published           # publisher progress (below)
  <YYYY>/<MM>/<recording-id>/
    meta.json                      # manifest, written LAST
    transcript.json                # {id, segments[]}
    transcript.md
    summary.md
    audio.<ext>                    # only when audio is requested

<root> is POCKETSYNC_FILES_ROOT. Every file is written atomically and meta.json last, so a reader that sees meta.json never sees a half-written recording. meta.json carries id, title, created_at, updated_at, recording_at, duration, language, content_hash, synced_at, deleted_in_pocket, deleted_at, merged_into, merged_at, files (refs as above, without meta) and audio_absent. audio_absent: true means Pocket reported it has no audio file for the recording (404 recording audio file not found). The reconciler then stops requesting audio until Pocket changes the recording. Any other audio failure is recorded in sync_state.errors and retried on the next pass.

Events contract

When a recording is new or changed, and its enrichment is done, pocketsync appends one event to <files root>/events.jsonl. This is the handoff to culture-rules and the mesh. Per pocketsync/events.py, one JSON object per line:

{
  "event_id": "00000000-0000-4000-8000-000000000001",
  "type": "pocketsync.recording.enriched",
  "recording_id": "rec_0001",
  "content_hash": "<64 hex chars>",
  "change": "created",
  "sinks": {
    "mongo": {"collection": "recordings", "doc_id": "rec_0001"},
    "files": {"meta": "2026/01/rec_0001/meta.json"}
  },
  "enrichment": {"labels": ["meeting"], "taxonomy_version": "1", "model": "example-embed-model"},
  "emitted_at": "2026-01-15T10:03:21Z"
}

Guarantees:

• A given (recording_id, content_hash) is appended at most once.
• The file is append-only. Each append is one write plus fsync. A crash can leave a torn final line, which readers ignore and the next append heals.
• Single writer: run one sync writer per files root.
• Optional MQTT publish through events-cli (pip install 'pocketsync[events]'): the outbox is drained at QoS 1 on topic events/pocketsync/recording/enriched, with the event as the envelope data and event_id as the envelope id. Progress is the byte offset in the events.jsonl.published sidecar. Delivery is at-least-once, so consumers must dedupe on the envelope id.

The example culture-rules rule that consumes these events is (planned).

Configuration and secrets

Settings resolve from environment variables, then an optional TOML file (POCKETSYNC_CONFIG), then defaults (pocketsync/config.py). Secrets are never printed in repr or error messages.

A Mongo URI looks like mongodb://<user>:<password>@localhost:27019/ and the stack only listens on loopback. Use placeholders in anything you commit.

Non-secret settings:

Credentials and privacy

This repository is public. Credentials belong in environment variables, the OS keyring, or a git-ignored config file. They are never committed. Tests use a fake Pocket adapter, in-memory or mongomock stores, and synthetic fixtures only, never data from real recordings.

Quickstart (development)

uv sync
uv run pytest -n auto                 # run the test suite
uv run pocketsync whoami              # identity from culture.yaml
uv run pocketsync learn               # self-teaching prompt (add --json)
uv run teken cli doctor . --strict    # the agent-first rubric gate CI runs

CLI available today

Every command supports --json. Results go to stdout, errors/diagnostics to stderr (never mixed). Exit codes: 0 success, 1 user error, 2 environment error, 3+ reserved.

As a Culture mesh agent

pocketsync is also an AgentCulture mesh agent, provisioned from culture-agent-template:

• A mesh identity: culture.yaml (suffix: pocketsync, backend: claude) and the matching resident prompt, CLAUDE.md.
• The guildmaster skill kit under .claude/skills/, vendored cite-don't-import. See docs/skill-sources.md.
• CI and deploy: pytest, lint, a secret scan, the agent-first rubric gate, a four-harness smoke check, and PyPI Trusted Publishing.

Prompt files by harness

Four harnesses, four root files, no shared base. Each file is read by exactly one harness:

CLAUDE.md is the fullest write-up of the repo's conventions; read it first. There is intentionally no AGENTS.md, because a shared base would leak Claude-specific guidance into the other harnesses.

culture.yaml's backend selects only the mesh resident, the one prompt the Culture daemon loads. It does not limit which harness you can run interactively in the clone; all four stay live. See docs/harness-selection.md and docs/automation-contract.md.

The gate section

culture.yaml carries a top-level gate section that declares how to set up and test this repo, so tooling (for example a PR-fixer that re-runs the checks before pushing) needs no per-repo knowledge:

gate:
  setup:
  - [uv, sync]
  test:
  - [uv, run, pytest, -n, auto]

setup and test are each a list of commands, and each command is an argv list (never a shell string), run in order from the repo root. Culture's loader ignores unknown top-level keys, so the section is safe alongside agents.

License

Apache 2.0 — see LICENSE.