aleth-ingest

The file ingestion companion for Aleth memory. Point it at a folder of text files and it stores them verbatim in your memory, then keeps them in sync: unchanged files are skipped, changed files supersede their old version, removed files are retracted. Your memory does not bloat with duplicates.

It only ever calls the public memory verbs (remember, supersede, retract, forget). It has no special access to the engine, so what it can do, you can do.

What it does

• Walks a directory of text files (.md, .txt, .rst by default; add your own globs).
• Reads documents, not just text. PDF, Word (.docx), OpenDocument (.odt), ePub, Excel (.xlsx), Outlook messages (.msg) and HTML, with --all-formats or an explicit --include '*.pdf'. Each format's library is an optional install; run aleth-ingest formats to see what your machine can read now and the one line that fixes anything it cannot.
• Stores content verbatim. Small files become one memory; larger files are split on paragraph and heading boundaries, never rewritten. Because the content is stored exactly, you can delete the original file and rely on the copy in memory.
• Skips unchanged files using a size + modified-time fast path, so a re-run over a big folder is cheap and does not re-read files that did not move.
• Supersedes changed files, by name. The new version is stored first, then each old memory is retired behind the memory that replaced it. Ask your memory what a file used to say and it can answer, because the link from the old text to the new one is recorded rather than implied. Nothing is destroyed unless you ask for it.
• Leaves unchanged parts alone. Edit one section of a long document and only that section's memories are replaced. The rest keep the identity they already had, so a small edit does not churn the whole file.
• Retracts what nothing replaced. A deleted file, or a section dropped when a file shrank, has no successor to name, so it is retracted instead: marked no longer current, with its history kept.
• Is safe with overlapping folders. Track a project and a folder inside it, the same folder twice, or two folders that each hold a copy of a file: a memory is retracted or erased only once nothing you track still holds that text. Your browser history and page text count too, so one article you read on two sites, or a page you also keep as a file, is not erased by forgetting only one of them. plan marks such a file HELD. If a tracked folder cannot be read (an unplugged drive, say), the run leaves alone anything that folder might hold, says so (plan marks it WITHHELD), and tries again next time.
• Tracks everything in a local manifest (<root>/.aleth-ingest/manifest.json), so runs are idempotent and an interrupted run resumes correctly.
• Imports your browser history, locally. Chrome, Edge, Brave and Firefox keep it in a database on your own disk; this reads a copy of that database, shows you the domains it would store, and stores nothing until you say so. No extension, no upload, no server asked about any site you visited. See Browser history below.

Install

pip install aleth-ingest

Requires a running memory engine reachable on the same machine (the companion finds it through its discovery file, the same way the memory client does).

Use

aleth-ingest plan    ./notes     # dry run: show what would change (no engine needed)
aleth-ingest run     ./notes     # ingest: apply the plan
aleth-ingest status  ./notes     # manifest summary against what is on disk
aleth-ingest history ./notes     # what replaced what, per file
aleth-ingest untrack ./old       # stop counting a folder you no longer track

aleth-ingest browser-preview     # what your browser history would store (no engine)
aleth-ingest browser-import --confirm-preview   # store it

history prints one line per replacement, old memory to new:

  pricing.md
    2026-08-22T22:04:00+00:00
      da2b0a62-24b5-48c2-bef7-830e160580b7  ->  ac331252-fd06-4137-a04d-4158342d5551

Both ids are real memories you can look up. The old one is still there and still readable; ask for it and your memory will tell you it is no longer current and which memory took its place.

There is no "undo this replacement" command, because a memory that could be un-replaced would not be much of a record. The way back is the way forward: put the earlier text back in the file and run again. The old wording returns as the current memory, and today's is retired behind it.

Common options:

--include GLOB     add an include glob (repeatable; default *.md,*.txt,*.rst)
--exclude GLOB     add an exclude glob (repeatable)
--all-formats      include every readable format, not just text

--manifest PATH    use a specific manifest file
--max-files N      cap the walk (default 1000); a truncated walk never removes
--max-supersessions N
                   cap replacements in ONE run (run; default 1000). Past the
                   cap a changed memory is retracted rather than replaced.
                   It still stops being current, it just does not name a
                   successor, and the run tells you the cap bound. It is
                   there so a sweeping mechanical edit across a whole folder
                   is something you see rather than something that happens.
--origin TAG       origin recorded on each memory (run; default "user")
--on-removed MODE  what happens to a memory when its file leaves the tree
                   (run): retract (default) stops it being current and keeps
                   the history; forget erases it permanently; keep leaves it
                   CURRENT. --hard-remove is a DEPRECATED alias for
                   "--on-removed forget", removed at the next major version.
--discovery-dir D  where the engine's discovery file lives (run)
--timeout SECONDS  wire timeout (run; default 30)
--scopes PATH      the list of tracked folders and records (default
                   ~/.aleth-ingest/scopes.json); every command that can
                   retract or erase takes it

run exits non-zero if any write fails, or if it held a change back because another tracked folder could not be read.

Each folder you run joins a short list of tracked folders, which is how a run knows what the others still hold. A tracked folder inside or above the one you run is found even before it is on the list; any other folder you tracked with an earlier version joins the list the next time you run it. aleth-ingest untrack <folder> takes a folder off the list, for one that is gone for good. It does not touch any memories.

Browser history

The pages you read are worth remembering, and the usual way to collect them is a browser extension that sends them somewhere. This does the opposite: your browser already keeps its history in a database on your disk, and this reads a copy of that file.

aleth-ingest browser-preview                    # look first. Stores nothing
aleth-ingest browser-import --confirm-preview   # then store what you saw

aleth-ingest browser-tracking                   # what is tracked, what is not
aleth-ingest browser-track   <domain>           # always store this one
aleth-ingest browser-block   <domain>           # stop storing new pages for it
aleth-ingest browser-category finance track     # switch a whole category
aleth-ingest browser-forget  <domain>           # erase everything you have of it

browser-preview prints one row per domain (pages, visits, first and last time you were there) and a count of what it refused. Nothing is stored. browser-import prints exactly the same table again and then refuses unless you pass --confirm-preview, so nothing can reach your memory without you having seen the list of places it came from.

Each page is stored as one memory holding the address, the title, how many times you visited and when you first and last did. Run it again next week and unchanged pages are skipped; a page whose title or visit count moved supersedes its previous memory by name, the same way a changed file does.

--browser NAME          only this browser (repeatable: chrome, edge, brave, firefox)
--exclude-domain D      never read anything under this domain (repeatable)
--exclude-file PATH     a file of such domains, one per line
--max-urls N            cap one run (default 5000, most recently visited first)
--rows N                how many domain rows to print (default 40)
--manifest PATH         where the local state lives
                        (default ~/.aleth-ingest/browser-history.json)
--policy PATH           where your decisions live
                        (default ~/.aleth-ingest/browser-tracking.json)
--confirm-preview       (import) you have read the table and want it stored
--origin TAG            (import) origin recorded on each memory (default "user")

Deciding what gets tracked

The categories below are blocked by default, not by decree. If you want an assistant that can actually help with your money, it has to be allowed to see your money. So you can switch a whole category on, and then still block the one place inside it you would rather it never saw.

aleth-ingest browser-category finance track    # the whole category
aleth-ingest browser-block    mybank.example   # ... except this one
aleth-ingest browser-track    news.example     # or open a single domain
aleth-ingest browser-tracking                  # show me where things stand

Your decisions live in ~/.aleth-ingest/browser-tracking.json, so you make them once. The narrower one always wins: a domain you named beats whatever its category says, in either direction. Anything you have never mentioned keeps the shipped default, so installing this changes nothing until you use it. An --exclude-domain you pass to browser-import is written into the same file, so you do not have to remember it next time.

When a category is switched on, the preview says so in its own line, under the counts, and names the policy file it read.

Stopping and forgetting are not the same thing

browser-block stops NEW records for a domain and KEEPS the existing ones. Telling a companion to stop following something is not telling it to delete what it already knows, and it will never make that leap on its own initiative.

browser-forget is the explicit erasure, and it is the only path that deletes. It permanently erases every memory this companion holds for that domain, drops them from the local record, and blocks the domain so the next run does not simply bring them back. It reports how many memories it erased, never which pages they were. A memory whose exact text a tracked file or another site's page also holds is kept and counted, because it is that source's memory too; erase it there as well to remove it.

aleth-ingest browser-block  somewhere.example    # stop. Keep what you have
aleth-ingest browser-forget somewhere.example    # erase what you have

Seeing all of it at once

The verbs above can change anything. None of them can SHOW you where you stand. For that there is one page:

aleth-ingest browser-page

It opens a page listing every site your memory actually holds: how many pages of each, when you last visited, whether it is tracked or blocked and why, with the category switches at the top and an erase button on every row that tells you exactly how much it will destroy before it destroys it. Erasing asks you to type the site's name, because it is the one thing on that page that cannot be undone.

The page never leaves your machine and never reaches the network. It is served on the loopback interface only, there is no flag to change that, the link carries a one-time key that dies when you stop the command, and the page itself loads no script, no font and no image from anywhere. A page that governs a local-only memory has no business making a request of its own, and if you are watching your network to check the claims in this file, this page must not be the first thing you see.

The switches write the same file the verbs write, so the page and your terminal can never disagree, and it re-reads that file on every request rather than caching it. Nothing on the page needs the engine to be running except erasing, which is the only action that has to reach it. You can always say "stop", even when there is nothing running to be told.

The privacy contract

Everything here happens on your machine. No page, domain or count leaves it. There is no service to ask whether a site is sensitive, which is the difference that matters: a tool that asks a server "is bank.example blocked?" has just told that server you bank there.

Your browser is never modified. The history database is copied to a temporary file, read from the copy, and the copy is deleted. That is also why it works while the browser is running.

These are never stored, and the preview shows only a count for each:

• Finance, health, adult, sign-in and mail sites, by a built-in list of well-known hosts and by name: any host with login, signin, sso, auth, account or passw in it is treated as a sign-in page, and any host with bank, banco, banca, banque or wallet in it is treated as a money page. Password managers and national e-ID portals are sign-in pages.
• Anything on your own machine or network: localhost, private addresses, any name without a dot, .local and .internal.
• Anything that is not a web page: file:, about:, chrome:, edge:, javascript:, data:.
• Tracking parameters (utm_*, fbclid, gclid and their family) and any parameter whose name looks like a credential (token, key, session, sig, password, code, state and neighbours). That last rule is deliberately over-eager: it will drop a harmless keywords= along with a real access_token=, because a coarser address is cheap and a stored session token is not.

The category lists are short and readable on purpose: open aleth_ingest/browser_history.py and check them. They are examples; the name rules above are what catch the bank, the insurer or the intranet that no list ever heard of. Anything they miss, --exclude-domain is yours.

A page that leaves your browser's history is not retracted. Browsers expire history on their own schedule, and that is not you saying you never read it. Memories of pages stay until you remove them yourself.

Page text

A list of addresses is an index, not a memory. "What did that article about sleep cycles actually say" is not answerable from a title, so this can fetch the pages you already read and keep the readable text. Then a recall by meaning lands on the words, with the source address attached.

The reader runs on your machine. The page is fetched by this program with Python's own HTTP client and the text is extracted in the same process. There is no cloud service, no language model and no third-party API anywhere in the path, so there is no server that could be told which pages you read. The only network traffic is to the page's own site: exactly the request your browser already made when you read it.

And it is a separate decision from tracking, per domain. Letting this read your browser history is letting it read a file that is already on your disk. Fetching is different: it opens a connection to somebody else's server, from your address, at a moment you did not choose. So page text has its own opt-in and nothing is fetched for a domain you have not named.

aleth-ingest page-text-opt-in  arstechnica.com   # yes, fetch pages from here
aleth-ingest page-text-preview                   # which pages? Fetches nothing
aleth-ingest page-text-fetch --confirm-preview   # fetch them, store the text
aleth-ingest page-text-status                    # what is opted in, what is stored
aleth-ingest page-text-opt-out arstechnica.com   # stop. Keep what you have
aleth-ingest page-text-forget  arstechnica.com   # erase the text you have

The urls come from the reading record browser-import wrote, so page text can only ever fetch pages you actually visited. It does not follow links and it does not discover anything: there is no seed list, no crawl and no way to point it at a site you have never been to.

page-text-preview prints one row per domain and a count of what it refused, and touches no network at all. page-text-fetch prints the same table again and then refuses unless you pass --confirm-preview.

--domain D         only this domain (repeatable). Narrows; it grants nothing
--limit N          cap one run (default 25 pages, most recently visited first)
--delay SECONDS    between two pages of the SAME domain (default 2.0)
--max-bytes N      refuse a page larger than this (default 2000000)
--min-chars N      refuse a page with less readable text than this (default 400)
--timeout SECONDS  per page (default 20)
--wire-timeout S   for the memory daemon (default 30)
--dry-run          fetch, report, store nothing
--consent PATH     your opt-in file (default ~/.aleth-ingest/page-text.json)
--manifest PATH    page-text state (default ~/.aleth-ingest/page-text-manifest.json)
--history-manifest P  the reading record (default ~/.aleth-ingest/browser-history.json)
--policy PATH      your tracking decisions (default ~/.aleth-ingest/browser-tracking.json)

What it will not fetch

The block always wins over the opt-in. A page is refused, before any connection is opened, if it falls in a blocked category (finance, health, adult, sign-in, mail), if you blocked its domain by name, if the domain is not opted in, if the address is not a public web page, or if you have no stored record of having visited it. Blocking a domain for history blocks it here too (you never have to say it twice), and the counts are reported per reason, never per site, exactly as the history preview does.

robots.txt is honoured, and there is no flag to turn that off. A local reader that ignores robots is a scraper with better manners, and that is not what this is. A site that refuses to show its rules (401 or 403 on robots.txt) is treated as a no.

It fetches HTML and plain text only. A redirect is followed within the same site and refused if it leaves it: storing another host's words under your address would make the source line a lie. A page larger than --max-bytes is refused rather than truncated: half an article stored as if it were the article is a memory nothing marks as incomplete. A page whose readable text is shorter than --min-chars is skipped: a cookie wall and a navigation menu match every query and answer none.

What gets stored, and how it stays attached to the page

The text exactly as extracted, split into chunks on its own structure. Every chunk is linked to the memory of your visit to that page, the one browser-import stored, whose first line is the address. Every chunk, not just the first: a recall lands on whichever chunk matched your question, and a chunk that cannot say where it came from is an unattributed quotation. Ask your memory a question in your own words, get the paragraph back, and the address is one step away from it.

That link is why a page you have not imported is not fetched. If there is no stored record of your visit, there is nothing to attach the text to, and text stored loose can never be traced back afterwards. So it is refused before anything is downloaded, and the preview says so. Run browser-import first.

Nothing is summarised, classified or rewritten. Re-run it next month and an unchanged page is skipped; a page that changed has its new text stored first and its old text superseded behind it, so what the page used to say stays readable as history. A page that has gone offline is not erased: a 404 is not you saying you never read it.

To erase what is stored for a site, aleth-ingest page-text-forget <domain> really deletes the text and stops fetching that domain. It is a separate store from the visit records, so erasing both is two commands and the output of each says so:

aleth-ingest page-text-forget somewhere.example   # erase the TEXT
aleth-ingest browser-forget   somewhere.example   # erase the VISIT records

How re-runs stay clean

Each stored memory is the file's text, verbatim. Where it came from (path, content hash, size, modified time, and which chunk it is) lives in the local manifest, the companion's authoritative file-to-memory mapping. On the next run it compares each file against the manifest by hash and only touches what changed. Even without the manifest (say it was deleted), a re-ingest is safe: the engine drops exact-duplicate content on its own.

That last property is also why an edit stays local. Because identical text stores as the same memory, re-reading a document you barely touched hands back the same memories for every unchanged part, so the companion recognises them as still current and leaves them alone, and only the part you actually edited is replaced.

What is out of scope

Images, audio and video are not handled here. A scanned page needs OCR, which needs software installed outside Python and gives you a confident guess rather than the document's own words. It does not watch folders or run as a daemon either: it ingests once when you run it, and again when you run it next.

For a document format, "verbatim" means the text the format's own reader gives back, in reading order, with nothing added and nothing summarised. Layout, images and styling are not stored. Plain text files are stored exactly as they are on disk.

If a page has no text layer, the file is reported and skipped rather than stored empty, so a later run can pick it up once you can read it. Nothing is recorded for it in the meantime.

License

Apache-2.0.