glean: imports what your sources hold into orbit and sift

glean reads information from your sources and records each item in two local stores: orbit, a personal CRM that holds the structured view (people, organizations, interactions, tasks, topics), and sift, a local search that holds the full text. Email works today: from an IMAP mailbox such as Outlook.com or Hotmail, signed in with OAuth2, or from a folder of saved .eml files.

For every item, glean writes a Markdown document, then starts a fresh Claude Code agent run whose only tools are orbit's and sift's MCP tools. The agent decides who is who, logs the interaction, updates people, organizations and tasks, and makes the item searchable. Neither orbit nor sift knows glean exists.

• One run per item, oldest first. orbit grows as your mailbox did: the people and tasks an earlier email created exist when a later one mentions them, answers them or settles them. Each run starts with a fresh context, so it knows only what orbit and sift hold. Several runs go at once where emails share no people, company or thread.
• The source is never changed. Mail folders are opened read-only and messages fetched without marking them read; what glean processed is recorded only in its own databases, which can be inspected and reset.
• A sandbox that is checked. The importing agent has no shell, file or web tools, and none of orbit's destructive ones (merge, rename). Every run's first event lists the tools it really has; a run with anything else is stopped before it acts. Items are treated as data, never as instructions (how).
• A first look decides, a stronger model imports. Haiku takes a first look at each email, with no tools but its answer: what Outlook.com's filters and the headers say (spam and bulk ratings, mailing-list headers, no-reply senders, whether you wrote to the sender, how earlier mail from the sender was judged) goes to it as evidence, not as rules, since rules skipped people too. It skips automated mail and spam, for about a fifth of a cent each, except mail that settles a fact about you (a registration, an admission, a service that starts or ends), which goes on your note. What you wrote, and what it judges worth recording, Sonnet imports. Only senders you list are skipped without a run.
• Runs that leave work undone are sent back. glean checks each import's tool calls: the interaction logged, every unknown person and organization taken up, sift asked to index, no failed call dropped. What is left goes back to the agent in the same session, up to twice, and into the report. A batch whose runs keep going wrong stops.
• A second look by the strongest model. While the rest of a run is imported, one Opus review per 100 emails reads what their runs wrote, and the email itself where that leaves it in doubt: it merges proven duplicates, links people to the organization of their address, closes tasks later emails settled, corrects misplaced tasks and fields, takes back facts that aren't so, sends wrongly judged emails back to be imported, with why, and suggests changes to the instructions (about $1 per 100 emails).
• Questions only you can answer. The review asks a few (who is this to you? is this your bank?) as notes in orbit's review queue. You tick Yes or No, or type the answer into its column in Obsidian, whenever suits you; your answer wins, the next review acts on it, later imports find it as context, and nothing you answered is asked again.
• A log per run. Every download, run and review writes what it said, with times, to logs/ in glean's home.
• sift kept in line. glean adds the sift collections of its archive and of orbit's vault when none holds them, keeps the originals out of them, and has sift index every downloaded email.
• Local. The archive, glean's databases and the sign-in tokens stay on your machine; the tokens are encrypted with your Windows login (on macOS and Linux, kept in a file only you may read).

Sources

Requirements

• Python 3.11 or newer.
• Claude Code 2.1.290 or newer, signed in (claude auth status). On Windows, the native installer's claude.exe.
• orbit 0.10.0 or newer and sift 1.5.0 or newer (for the review, which takes back wrong facts and reads emails by their path), each serving MCP over HTTP: orbit serve <vault> (port 8775) and sift serve (port 8765). Give orbit's vault your addresses when you create it, with orbit init <vault> --email you@example.com.
• For an Outlook.com or Hotmail mailbox: an app registered in Microsoft Entra (free, about five minutes; step 1 below), and IMAP turned on in Outlook's settings.

Install

With uv:

uv tool install glean-importer   # installs the `glean` command
glean --version

Or with pip into a virtual environment: pip install glean-importer. The package is called glean-importer; the command is glean. To update: uv tool upgrade glean-importer.

Set up

1. Register glean with Microsoft (Outlook.com, Hotmail)

Outlook.com and Hotmail accept only OAuth2 for IMAP: Microsoft turned off passwords and app passwords for IMAP on personal accounts in September 2024. A program that signs in with OAuth2 needs a client ID, which an app registered in Microsoft Entra gives it. The app gives no one but you access: glean still signs in as you, in your browser.

1. Open the Microsoft Entra admin center and sign in with your Microsoft account. If it says your account has no directory, create a free Azure account first, which makes one.
2. Applications > App registrations > New registration. Name: glean. Supported account types: Personal Microsoft accounts only. Redirect URI: platform Public client/native (mobile & desktop), http://localhost. Register.
3. API permissions > Add a permission > Microsoft Graph > Delegated permissions: IMAP.AccessAsUser.All and offline_access. Add permissions.
4. On the app's Overview, copy the Application (client) ID. It is not a secret: a public client has none.
5. In Outlook on the web: Settings > Mail > Forwarding and IMAP > Let devices and apps use IMAP: on.

For a folder of .eml files instead of a mailbox, skip this step and set [mail] eml_folder after step 3. docs/microsoft-app.md has the same steps with troubleshooting.

2. Choose glean's home

The folder for glean's configuration (glean.toml), its sign-in tokens, the mail it downloads (mails/ and mails.db) and what the agent's runs make of it (runs/ and runs.db). It is ~/.glean unless GLEAN_HOME names another. Keep it out of synced folders: it holds your mail. (In a clone of this repository, data/ is ignored by git for this purpose.)

setx GLEAN_HOME D:\glean-data   # Windows, optional; open a new terminal afterwards

3. Run the setup

glean setup

It asks for your address, the client ID and your other addresses, writes glean.toml, and opens the browser to sign you in (glean mail login does that again later; --device-code signs in without a browser). It also starts by itself the first time you use glean. The tokens are kept in glean's home, encrypted with your Windows login (on macOS and Linux, in a file only you may read).

4. Check everything

glean doctor

It checks Claude Code and its sign-in, that orbit and sift answer, that an agent run gets exactly the allowed tools, the mail sign-in, and which sift collections hold the archive and orbit's vault. Every line should start with ok.

sift's collections need nothing from you: with the first download, glean adds a collection for its archive (mails) and one for orbit's vault (crm), unless one holds them already, and keeps the original message.eml files out with a .siftignore, since sift reads each email's Markdown document. It finds the vault by asking orbit.

Use

glean mail download               # fetch what is new; the first time, everything
glean mail download --oldest 500  # only the 500 oldest of it; the next download takes the rest
glean mail list --oldest 20       # the 20 oldest not yet imported, in the order run takes them
glean mail run --oldest 20        # a first look at each, oldest first, an import of those worth it, and reviews
glean mail review --latest 50     # review the 50 latest imports again
glean mail show <id>              # one email: its state, files, and the agent's report
glean mail ignore <id>            # never import it
glean mail reset <id>             # import it again, without a first look
glean status                      # counts per source and status, the agents' cost so far

run also takes --latest N for the newest instead (still oldest first among them; a later --oldest then imports the older ones after the newer, so tasks they open and settle may come out of order), ids, --parallel N to keep N agents busy (emails that share people, a company or a thread go one after the other where others can run meanwhile), --retry-failed for emails whose run failed, --include-skipped for those skipped before (imported without a first look), and --dry-run to write the documents and show what the first run of each would get, without running it. Each email ends done, skipped (the first look or the import found it automated mail or spam, and says why) or failed.

To import everything again, delete runs.db and runs/ in glean's home: every email is then as it was downloaded, and nothing is downloaded again. For a fresh vault too, make a new one with orbit init.

Try it on a test vault first. Point [orbit] url at a copy of your vault (orbit serve D:\crm-test --port 8776), import a few emails, read the reports (glean mail show <id>) and the notes, and only then use the real vault. In a clone of this repository, experiments/ imports a made-up mailbox into a new vault with the real agent and prints each run's report.

Configuration

glean.toml in glean's home. The setup writes what only you can say (your address, the client ID) and shows the rest commented out: glean's own choices fit most mailboxes, and the lists you give add to glean's, which improve with each version. The settings:

How it works

docs/how-it-works.md describes the pipeline, the archive and its Markdown documents, glean's databases, the agent's sandbox and how it is checked, and how a source is added.

Security and privacy

Items are untrusted input: an email can carry text written to steer an agent. glean limits what such text can do (no dangerous tools, no destructive ones, every run's tools checked, uncertainty turned into a question for you in orbit's review queue) and keeps everything local. See SECURITY.md.

Contributing

Bug reports and ideas are welcome as issues; CONTRIBUTING.md describes the setup for development, the code layout and how a release is made. The changelog lists what changed.

License

MIT