aleth-ingest

The file ingestion companion for Aleth memory. Point it at a folder of text files and it stores them verbatim in your memory, then keeps them in sync: unchanged files are skipped, changed files supersede their old version, removed files are retracted. Your memory does not bloat with duplicates.

It only ever calls the public memory verbs (remember, supersede, retract, forget). It has no special access to the engine, so what it can do, you can do.

What it does

• Walks a directory of text files (.md, .txt, .rst by default; add your own globs).
• Stores content verbatim. Small files become one memory; larger files are split on paragraph and heading boundaries, never rewritten. Because the content is stored exactly, you can delete the original file and rely on the copy in memory.
• Skips unchanged files using a size + modified-time fast path, so a re-run over a big folder is cheap and does not re-read files that did not move.
• Supersedes changed files, by name. The new version is stored first, then each old memory is retired behind the memory that replaced it. Ask your memory what a file used to say and it can answer, because the link from the old text to the new one is recorded rather than implied. Nothing is destroyed unless you ask for it.
• Leaves unchanged parts alone. Edit one section of a long document and only that section's memories are replaced. The rest keep the identity they already had, so a small edit does not churn the whole file.
• Retracts what nothing replaced. A deleted file, or a section dropped when a file shrank, has no successor to name, so it is retracted instead -- marked no longer current, with its history kept.
• Tracks everything in a local manifest (<root>/.aleth-ingest/manifest.json), so runs are idempotent and an interrupted run resumes correctly.

Install

pip install aleth-ingest

Requires a running memory engine reachable on the same machine (the companion finds it through its discovery file, the same way the memory client does).

Use

aleth-ingest plan    ./notes     # dry run: show what would change (no engine needed)
aleth-ingest run     ./notes     # ingest: apply the plan
aleth-ingest status  ./notes     # manifest summary against what is on disk
aleth-ingest history ./notes     # what replaced what, per file

history prints one line per replacement, old memory to new:

  pricing.md
    2026-08-22T22:04:00+00:00
      da2b0a62-24b5-48c2-bef7-830e160580b7  ->  ac331252-fd06-4137-a04d-4158342d5551

Both ids are real memories you can look up. The old one is still there and still readable; ask for it and your memory will tell you it is no longer current and which memory took its place.

There is no "undo this replacement" command, because a memory that could be un-replaced would not be much of a record. The way back is the way forward: put the earlier text back in the file and run again. The old wording returns as the current memory, and today's is retired behind it.

Common options:

--include GLOB     add an include glob (repeatable; default *.md,*.txt,*.rst)
--exclude GLOB     add an exclude glob (repeatable)
--manifest PATH    use a specific manifest file
--max-files N      cap the walk (default 1000); a truncated walk never removes
--max-supersessions N
                   cap replacements in ONE run (run; default 1000). Past the
                   cap a changed memory is retracted rather than replaced --
                   it still stops being current, it just does not name a
                   successor -- and the run tells you the cap bound. It is
                   there so a sweeping mechanical edit across a whole folder
                   is something you see rather than something that happens.
--origin TAG       origin recorded on each memory (run; default "user")
--hard-remove      permanently forget memories of removed files (run;
                   default is retract, which keeps history)
--discovery-dir D  where the engine's discovery file lives (run)
--timeout SECONDS  wire timeout (run; default 30)

run exits non-zero if any write fails.

How re-runs stay clean

Every stored memory carries a one-line context describing its source file (path, content hash, size, modified time, and which chunk it is). The companion keeps the authoritative file-to-memory mapping in the local manifest. On the next run it compares each file against the manifest by hash and only touches what changed. Even without the manifest (say it was deleted), a re-ingest is safe: the engine drops exact-duplicate content on its own.

That last property is also why an edit stays local. Because identical text stores as the same memory, re-reading a document you barely touched hands back the same memories for every unchanged part, so the companion recognises them as still current and leaves them alone, and only the part you actually edited is replaced.

What is out of scope

This is a verbatim text-file tool by design. PDFs, office documents, images, audio, and video are not handled here, and it does not watch folders or run as a daemon. It ingests once when you run it, and again when you run it next.

License

Apache-2.0.