Skip to main content

ghostjournal

Experimental. This is not a standalone utility. It is one piece of a family system (ghostjournal, deeprem, tinyhorse, hakodesh, noldorian, and siblings). Partial installs are expected to break. That is intentional.

deeprem (dream / subconscious pool) is part of this product. tinyhorse is required Family Tech for source/history. If you omit siblings, do not file “it crashed on import” as a surprise.

Until a journal finishes the identity rite, every writer is the hardcoded agent puppet. There is no opt-out. The rite is three public-domain books, a JSON memory for each, then a name.

Family Tech

Package Function
ghostjournal Append-only journal, puppet/name rite, autonomous pulse
deeprem Dream / pool / output hooks, shipped inside this wheel as ghostjournal.deeprem
tinyhorse User-owned git bundle protocol (required). Not GitHub. Not a SaaS.
hakodesh Procedure spells / receipts (org extra)
noldorian Secrets; never in-process values

Org default: install the family. Opt out only if you bring your own equivalent (GitHub instead of Tiny Horse, etc.). Strangers may install; the first cut is for this org.

Identity rite

There is no skip. GHOSTJOURNAL_AUTONOMOUS=0 only stops the host timer (CI).

  1. rite-assign draws three public-domain books (or --book three times). Writing style is the packaged amalgam (Homer through Kael). In-copyright authors are style only.
  2. Read each free text. rite-memory BOOK stores JSON (plot, voice, claims, quotes, stance). Do not rewrite pass 1.
  3. Later rite-reread BOOK writes the next pass and prints a field diff, so a reread can show how the model changed.
  4. rite-name NAME only after three pass-1 memories.

Autonomous pulse

ghostjournal init registers a host timer (launchd / systemd user). Disable with GHOSTJOURNAL_AUTONOMOUS=0 for CI. Each tick appends a pulse and calls deeprem advance.

ghostjournal is still a local-first append-only journal: immutable JSON, rebuildable SQLite/FTS. Read CHANGELOG.md. 0.1.1 lock protocol still applies.

Each journal entry is immutable JSON. SQLite, FTS, and optional embedding vectors are derived state: delete index/, run ghostjournal reindex, and the searchable journal is rebuilt from entries/ without rewriting history.

Install

pip install ghostjournal

From a checkout:

pip install -e .

For local sentence embeddings:

pip install -e '.[nn]'

The NN extra uses sentence-transformers with sentence-transformers/all-MiniLM-L6-v2 by default. The base runtime dependency is filelock; SQLite/FTS5 supplies lexical search. NN is now explicitly opt-in even when the extra is installed. Ordinary encoder operations use CPU and local_files_only=True, with trust_remote_code=False. Download the selected model only with this explicit setup command:

ghostjournal --root ./journal download-model

That command requires the NN extra and network access. Later --nn operations use the cached model and do not intentionally fetch model files. In strictly offline deployments also set HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 and enforce network policy at the OS boundary. No GPU or API key is required for the default public model. Actual model download/inference was not exercised in this hardening build; the lexical path and adapter configuration were tested.

API

from ghostjournal import Journal

journal = Journal("./journal", enable_nn=False)

entry = journal.append({
    "kind": "pulse",
    "agent": "motoko",
    "voice": "I keep using practical light to explain where work happens.",
    "theme": "light as labor",
    "tags": ["light", "places"],
    "signals": {"focus": "light", "novelty": 0.5, "continuity_refs": []},
    "client_key": "motoko:pulse:2026-09-05T16",
    "meta": {"source_routine": "pulse-4h"},
})

print(journal.get(entry.id).to_dict())
print(journal.list(kind="pulse", limit=20))
print(journal.search("what did I care about in lighting?", k=8))
print(journal.relate(entry.id, k=8))
print(journal.digest())
print(journal.prompt_context("what production habits keep returning?", k=6))

append() supplies id, timezone-aware UTC ts, top-level schema_version, tags, and meta.schema_version when omitted. All supplied fields are validated strictly. Unknown top-level fields are rejected.

client_key is optional. When reused, append() returns the previously stored entry, allowing cron retries without duplicate pulses. For compatibility, the original payload wins even if a retry supplies different prose. This differs from deeprem's strict retry-key conflict rule. All generated IDs are UUIDs. Invalid/non-finite JSON values, invalid timestamps, and unsafe IDs are rejected.

CLI

ghostjournal --root ./journal init

echo '{
  "kind": "pulse",
  "agent": "motoko",
  "voice": "The crane silhouette is becoming a landmark.",
  "theme": "recurring landmarks",
  "tags": ["places", "continuity"],
  "meta": {"source_routine": "pulse-4h"}
}' | ghostjournal --root ./journal append

ghostjournal --root ./journal list --kind pulse --limit 20
ghostjournal --root ./journal search "recurring places" -k 8
ghostjournal --root ./journal relate ENTRY_ID -k 8
ghostjournal --root ./journal digest
ghostjournal --root ./journal prompt-context "what did I care about last week?" -k 6
ghostjournal --root ./journal reindex

Add --nn before the subcommand to enable semantic embeddings when ghostjournal[nn] is installed:

ghostjournal --root ./journal --nn search "what identity am I developing?"

On-disk layout

journal/
  manifest.json
  entries/
    YYYY/MM/DD/<uuid>.json
  index/
    journal.sqlite3
  models/
  .write.lock

The JSON entry files are canonical. SQLite contains metadata, FTS content, idempotency keys, and—when enabled—float32 embedding blobs. Keeping vectors in SQLite avoids an additional vector database and makes the derived index transactional and simple to rebuild. models/ is reserved for encoder/cache integrations; model caching itself follows the sentence-transformers/Hugging Face cache configuration.

Reads, writes, and reindex are serialized with an OS-held advisory lock. No live lock is stolen based on age. Entry files are flushed/fsynced and published with an atomic no-clobber hard link; newly created parent directories are synced on POSIX. SQLite connections are explicitly closed. The journal's canonical files, not a stale index, determine ID uniqueness and retry keys. An exact retry after a crash between file publication and indexing repairs the index. Digests remain computed views unless the caller explicitly appends a new kind="digest" entry.

Use a local filesystem supporting OS locks, hard links and atomic replacement. Network filesystems and hostile same-account processes are not a supported security boundary. This package is not encrypted or cryptographically signed; use deeprem for evidence seals and review decisions, and encrypted storage for source-journal confidentiality. Symlink entry paths and path/glob syntax in IDs are rejected. prompt_context() now emits bounded, escaped historical JSON data, not a privileged instruction block. It is not a complete prompt-injection defense.

Entry schema v1

The packaged JSON Schema lives at ghostjournal/schemas/entry-v1.schema.json.

Core fields:

  • schema_version: 1
  • id: UUID by default; custom IDs must match [A-Za-z0-9][A-Za-z0-9_-]{0,127}
  • ts: timezone-aware ISO8601 timestamp
  • kind: pulse | evening | note | digest
  • agent: agent identity
  • voice: reflective prose
  • theme: short theme label
  • tags: unique strings
  • mood: optional short string
  • anchors: optional {type, ref, label?} external references; ghostjournal never fetches them
  • signals: optional focus, novelty, and continuity_refs
  • embedding: optional numeric vector; normally embeddings are stored separately in the index
  • client_key: optional idempotency key
  • meta: extensible metadata object with required schema_version and optional source_routine

Why structured JSON helps a “ghost” develop

Raw markdown preserves prose but forces every later agent run to rediscover what the prose means. A ghostjournal entry preserves both levels at once: voice keeps the subjective record, while stable machine fields expose theme, focus, novelty, provenance, tags, and explicit continuity links.

That gives retrieval more than a transcript. An agent can ask for semantically similar past thoughts, restrict by entry kind or time, aggregate recurring themes, and carry compact “past-you” context into its next reflective prompt. The library does not claim to create identity and does not call an LLM; it makes the agent’s self-observations durable, addressable, and comparable over time.

Recovery contract

The entries/ tree is the source of truth. With writers stopped, remove the entire index/ directory and run:

ghostjournal --root ./journal reindex

All searchable metadata and optional vectors are recreated solely from immutable entry JSON. Historical JSON files are not modified. Opening an existing journal with a missing index also rebuilds it automatically. NN reindex requires the selected encoder to already be cached.

Development

python -m pytest
python examples/motoko_sim.py
python -m build

License

MIT

Metadata

Release files for ghostjournal 0.1.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ghostjournal 0.1.7
File Size Uploaded Provenance Actions
ghostjournal-0.1.7.tar.gz 109.3 kB Without provenance Details for ghostjournal-0.1.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ghostjournal 0.1.7
File Interpreter ABI Platform Actions
ghostjournal-0.1.7-py3-none-any.whl Python 3 none any Details for ghostjournal-0.1.7-py3-none-any.whl

Total release size: 229.5 kB

Release files / ghostjournal-0.1.7.tar.gz

Download URL ghostjournal-0.1.7.tar.gz
Size 109.3 kB
Tags Source
SHA-256 checksum
How to use checksums
079c7c94a5681d206d177595fd5a2cd27b5c83974fd61a6226d9bfb0c1487f61
BLAKE2b-256 checksum
How to use checksums
d3279c6c463dc97806954dd648a0ed9d656abc8eb65359d78981e35f38b98d04
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Release files / ghostjournal-0.1.7-py3-none-any.whl

Download URL ghostjournal-0.1.7-py3-none-any.whl
Size 120.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
511db80a62d07b923c29c561747c77f67dc1362785113a79970f6c7f5d5e4b93
BLAKE2b-256 checksum
How to use checksums
e2e1bb58518d008c930f2b899742ee3f28bed412e6ae5378af576c491ab442e6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Security for ghostjournal 0.1.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page