ghostjournal

0.1.1 hardening release. Stop all 0.1.0 writers before upgrading. The lock protocol changed from age-based lockfiles to OS-held advisory locks. Do not mix old and new writers on one root. Existing valid entry JSON and existing manifest bytes are preserved; new roots receive a journal UUID. Read CHANGELOG.md.

ghostjournal is boring, local-first infrastructure for agents that need a durable reflective journal rather than a chat-log dump.

Each journal entry is immutable JSON. SQLite, FTS, and optional embedding vectors are derived state: delete index/, run ghostjournal reindex, and the searchable journal is rebuilt from entries/ without rewriting history.

Install

pip install -e .

For local sentence embeddings:

pip install -e '.[nn]'

The NN extra uses sentence-transformers with sentence-transformers/all-MiniLM-L6-v2 by default. The base runtime dependency is filelock; SQLite/FTS5 supplies lexical search. NN is now explicitly opt-in even when the extra is installed. Ordinary encoder operations use CPU and local_files_only=True, with trust_remote_code=False. Download the selected model only with this explicit setup command:

ghostjournal --root ./journal download-model

That command requires the NN extra and network access. Later --nn operations use the cached model and do not intentionally fetch model files. In strictly offline deployments also set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 and enforce network policy at the OS boundary. No GPU or API key is required for the default public model. Actual model download/inference was not exercised in this hardening build; the lexical path and adapter configuration were tested.

API

from ghostjournal import Journal

journal = Journal("./journal", enable_nn=False)

entry = journal.append({
    "kind": "pulse",
    "agent": "motoko",
    "voice": "I keep using practical light to explain where work happens.",
    "theme": "light as labor",
    "tags": ["light", "places"],
    "signals": {"focus": "light", "novelty": 0.5, "continuity_refs": []},
    "client_key": "motoko:pulse:2026-09-05T16",
    "meta": {"source_routine": "pulse-4h"},
})

print(journal.get(entry.id).to_dict())
print(journal.list(kind="pulse", limit=20))
print(journal.search("what did I care about in lighting?", k=8))
print(journal.relate(entry.id, k=8))
print(journal.digest())
print(journal.prompt_context("what production habits keep returning?", k=6))

append() supplies id, timezone-aware UTC ts, top-level schema_version, tags, and meta.schema_version when omitted. All supplied fields are validated strictly. Unknown top-level fields are rejected.

client_key is optional. When reused, append() returns the previously stored entry, allowing cron retries without duplicate pulses. For compatibility, the original payload wins even if a retry supplies different prose. This differs from deeprem's strict retry-key conflict rule. All generated IDs are UUIDs. Invalid/non-finite JSON values, invalid timestamps, and unsafe IDs are rejected.

CLI

ghostjournal --root ./journal init

echo '{
  "kind": "pulse",
  "agent": "motoko",
  "voice": "The crane silhouette is becoming a landmark.",
  "theme": "recurring landmarks",
  "tags": ["places", "continuity"],
  "meta": {"source_routine": "pulse-4h"}
}' | ghostjournal --root ./journal append

ghostjournal --root ./journal list --kind pulse --limit 20
ghostjournal --root ./journal search "recurring places" -k 8
ghostjournal --root ./journal relate ENTRY_ID -k 8
ghostjournal --root ./journal digest
ghostjournal --root ./journal prompt-context "what did I care about last week?" -k 6
ghostjournal --root ./journal reindex

Add --nn before the subcommand to enable semantic embeddings when ghostjournal[nn] is installed:

ghostjournal --root ./journal --nn search "what identity am I developing?"

On-disk layout

journal/
  manifest.json
  entries/
    YYYY/MM/DD/<uuid>.json
  index/
    journal.sqlite3
  models/
  .write.lock

The JSON entry files are canonical. SQLite contains metadata, FTS content, idempotency keys, and—when enabled—float32 embedding blobs. Keeping vectors in SQLite avoids an additional vector database and makes the derived index transactional and simple to rebuild. models/ is reserved for encoder/cache integrations; model caching itself follows the sentence-transformers/Hugging Face cache configuration.

Reads, writes, and reindex are serialized with an OS-held advisory lock. No live lock is stolen based on age. Entry files are flushed/fsynced and published with an atomic no-clobber hard link; newly created parent directories are synced on POSIX. SQLite connections are explicitly closed. The journal's canonical files, not a stale index, determine ID uniqueness and retry keys. An exact retry after a crash between file publication and indexing repairs the index. Digests remain computed views unless the caller explicitly appends a new kind="digest" entry.

Use a local filesystem supporting OS locks, hard links and atomic replacement. Network filesystems and hostile same-account processes are not a supported security boundary. This package is not encrypted or cryptographically signed; use deeprem for evidence seals and review decisions, and encrypted storage for source-journal confidentiality. Symlink entry paths and path/glob syntax in IDs are rejected. prompt_context() now emits bounded, escaped historical JSON data, not a privileged instruction block. It is not a complete prompt-injection defense.

Entry schema v1

The packaged JSON Schema lives at ghostjournal/schemas/entry-v1.schema.json.

Core fields:

• schema_version: 1
• id: UUID by default; custom IDs must match [A-Za-z0-9][A-Za-z0-9_-]{0,127}
• ts: timezone-aware ISO8601 timestamp
• kind: pulse | evening | note | digest
• agent: agent identity
• voice: reflective prose
• theme: short theme label
• tags: unique strings
• mood: optional short string
• anchors: optional {type, ref, label?} external references; ghostjournal never fetches them
• signals: optional focus, novelty, and continuity_refs
• embedding: optional numeric vector; normally embeddings are stored separately in the index
• client_key: optional idempotency key
• meta: extensible metadata object with required schema_version and optional source_routine

Why structured JSON helps a “ghost” develop

Raw markdown preserves prose but forces every later agent run to rediscover what the prose means. A ghostjournal entry preserves both levels at once: voice keeps the subjective record, while stable machine fields expose theme, focus, novelty, provenance, tags, and explicit continuity links.

That gives retrieval more than a transcript. An agent can ask for semantically similar past thoughts, restrict by entry kind or time, aggregate recurring themes, and carry compact “past-you” context into its next reflective prompt. The library does not claim to create identity and does not call an LLM; it makes the agent’s self-observations durable, addressable, and comparable over time.

Recovery contract

The entries/ tree is the source of truth. With writers stopped, remove the entire index/ directory and run:

ghostjournal --root ./journal reindex

All searchable metadata and optional vectors are recreated solely from immutable entry JSON. Historical JSON files are not modified. Opening an existing journal with a missing index also rebuilds it automatically. NN reindex requires the selected encoder to already be cached.

Development

python -m pytest
python examples/motoko_sim.py
python -m build

License

MIT