vishalakshi

vishalakshi keeps what you collect in one SQLite file on your own machine, called a vault, and lets you search it and ask questions of it. One question can draw on a paper, a web page, a talk transcript, your own notes and the source code on disk at once, and the answer cites the sections it used.

It is built from smaller packages. fossick fetches pages, papers and videos. litesearch stores the text and finds it again by keyword and by meaning; a Vault is a litesearch Index, and its search defaults come from there. Urai is the chat interface the vault talks to models through, and Rishi supplies the model backends. kosha indexes source code, and rgapi searches files on disk that nothing has indexed yet. rahasya finds personal information and varga recognises what kind of document something is.

Install

pip install vishalakshi

Adding and searching

Vault() with no argument opens ~/.vishalakshi/vault.db. This page uses a throwaway file and fills it with this repository’s own docs and source.

from tempfile import mkdtemp
from litesearch import repo_root
from vishalakshi import Vault

root = str(repo_root() or Path('..'))   # works from nbs/ or the repo root
v = Vault(Path(mkdtemp())/'vault.db')
v.enc.note

'minishlab/potion-multilingual-128M (256d, float16, model2vec)'

add takes a directory, a file or a piece of text. grab takes almost anything else, such as an arXiv id, a YouTube link, a GitHub repository, a PDF or a web address, and reads each the right way (acquire). Adding a source that is already in the vault reads it again only when its content changed, and the result’s changed says whether it did. note writes your own note. A note with a key= or a title replaces the earlier note with the same key or title; a note with neither is added as a new one.

Every search result carries age, the seconds since its document was filed, and stale, which is True when the vault believes the document no longer matches its source. Stale results sort last. poll() (below) marks documents stale: a file that changed or vanished, a web page past its time limit that no watch refreshes, or an older copy that a watch has replaced. It deletes only documents that are both stale and replaced.

v.add(root)                       # README.md and every notebook under nbs/
v.note('federate fuses the legs by rank because they share no vector space: the vault embeds '
       'prose, kosha embeds identifiers, ripgrep embeds nothing.', tags=['retrieval', 'design'])
v.note('one document per key: writing it again replaces it', title='federation', key='federation')
v.stats()

To answer questions the vault needs a language model. The default is gemma-4-E2B, a small model that runs on this machine through LiteRT, on the GPU where there is one, and needs no API key. model= names any other model Rishi knows, local or hosted, and chat_kw= passes settings to it. $VISHALAKSHI_MODEL changes the default and $VISHALAKSHI_GPU=0 keeps LiteRT on the CPU. ask retrieves the four best sections (sections=4), shows the model up to 1,500 characters of each (max_chars=1500), and returns the answer with cited, the sections the answer cites (ask).

r = v.ask('why are rankings fused instead of distances?')
print(r.model, '·', r.runtime)
print(r.answer)
print(L(r.cited).map(lambda c: f"{c['n']} {c['breadcrumb']}"))
if (hit := first(r.cited)): print(v.read(hit['node_id'])['text'][:400])

litert/litert-community/gemma-4-E2B-it-litert-lm · litert
The provided sections do not explicitly state why rankings are fused instead of distances. However, one section mentions that `federate` fuses rankings with RRF and never distances [2].
['2 03 code › Fusing legs that share no vector space']
The vault embeds prose, kosha embeds identifiers, ripgrep embeds nothing. Legs share no vector space, so `federate` fuses rankings with RRF and never distances.


#| export
def _prose(v, q, n, kind=None, source:str='prose') -> L:
    'Vault sections, normalised to the federated row shape. `source` names the shelf they came from.'
    return L(v.sections(q, limit=n, kind=kind)).map(
        lambda 

Personal information and junk

A personal vault holds things that should not leave your machine: invoices, letters, bank statements. vishalakshi looks for personal information in everything it stores, with patterns and checksums rather than a model: email addresses, phone numbers, card and account numbers, national identity numbers, street addresses, dates of birth and API keys. On a held-out test set of 240 generated documents it missed nothing and raised no false alarm (evals/RESULTS.md). Names are harder. It finds only names that follow a title such as Dr or Mrs, and only when name detection is on (ner=True); a report’s scanned_ner says whether it looked.

What it finds sets each document’s privacy level: Open, Personal (an identifier was found, or someone marked it), or Local only (set by hand, or a sensitive type such as a passport or a payslip). Each shelf can also have a floor, a level none of its documents goes below. Before every model call the vault decides, with send_check, what each document may give a hosted model, meaning one that runs anywhere but this machine: an Open document goes whole, a Personal one masked or not at all, a Local only one never. A document’s title, headings and source path go under the same rule as its text. A model on this machine may read everything. Each send to a hosted model is recorded in the vault’s send log. The details are on pii.

Junk is a separate problem: cookie banners and site footers that match every search and answer nothing. suggest_noisy ranks likely junk without any labels (0.988 AUC on the test corpora) and mark_noisy takes a document out of search (quality).

from vishalakshi.pii import pii_report

r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying, r.kinds

(True,
 {'email': 1, 'card': 1, 'secret': 1},
 {'email': 1, 'card': 1, 'secret': 1})

# a person's judgement, over the top of the arithmetic
v.add('A letter about Jane, and what she said on Tuesday.',
      title='letter', source='/inbox/letter.md')
v.add('Cookie policy. All rights reserved. Privacy. Terms. Contact us.',
      title='footer', source='/inbox/footer.md')
v.mark_pii('/inbox/letter.md', reason='names a person')     # private even though nothing matched
v.mark_noisy('/inbox/footer.md', reason='site furniture')   # out of search, sections, context, ask

r = v.pii('/inbox/letter.md', ner=True)   # scanned_ner says whether names were looked for
r.scanned_ner, r.detected, r.has_pii, any('footer' in h['breadcrumb'] for h in v.search('cookie policy'))

pii= in ask	what happens when the retrieved sections hold personal information
local	a model on this machine answers, told never to repeat a personal detail; if the model is not local, nothing is sent
redact	each recognised identifier is masked, then the named model answers, hosted or not. The heading, title and source of each section are masked with its text. Names without a title are not masked
refuse	no model is asked, and the result says what was held back

The default is local when a local runtime is installed and redact when not. ask cannot run with detection off: pii='off' warns and applies the default. The result’s policy says which was used, and any identifier the model repeats is masked in the answer. Under every policy a hosted model gets a Local only section only as a placeholder, with no heading, title or source. Search and reading take pii= too (off, redact or refuse), and there the default is off, which returns the stored text. The MCP server defaults to redact for every tool and refuses off, since its client is a model the vault does not run.

v.mark_not_pii(doc_id, reason='my own invoice')          # a person clears a false alarm
v.set_level(doc_id, 'local')                              # never send this one to a hosted model
v.send_check(refs=[doc_id], model='gpt-4.1-nano').why     # what a send would do, in one sentence
v.accept_noisy(k=10, reason='reviewed')                   # mark the current top junk suggestions
v.fit_noise(save=True); v.use_noise(True)                 # optional: fit the junk score to your marks

Paperwork

A document’s kind says how it arrived; its doctype says what it is, such as an invoice, a receipt, a contract, a paper or a payslip. categorize decides the doctype from cue phrases and figures in the text, and asks a model only when the cues cannot decide. extract pulls fields out of a document, such as an invoice’s number, vendor and total; without a schema, the doctype picks one. ask_doc, or ask(ref=...), answers a question about named documents, read whole, with the rest of the vault behind them. More on extract and ask.

INVOICE = '''# INVOICE
Invoice No: ACM-2024-0117
Date: 2024-03-01
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd
Total due: 1,240.00 EUR
Due date: 2024-03-31
Items: widgets x10 @ 124.00
'''
v.add(INVOICE, title='Acme invoice 0117', source='/inbox/acme-0117.md')
v.categorize_all(llm='never')
v.doctypes()

{'code': 13, 'documentation': 4, 'invoice': 1, 'paper': 1, 'other': 1}

from dataclasses import fields
from vishalakshi.extract import SCHEMAS, as_schema

print(L(SCHEMAS), L(fields(as_schema('invoice'))).attrgot('name'))
e = v.extract('/inbox/acme-0117.md')                    # doctype picks schema
a = v.ask_doc('/inbox/acme-0117.md', 'what is owed, to whom, and by when?',
              schema='amount:float, payee:str, due:str')
e.schema, e.fields, a

[{'invoice': <class 'vishalakshi.extract.Invoice'>, 'purchase_order': <class 'vishalakshi.extract.Invoice'>, 'quote': <class 'vishalakshi.extract.Invoice'>, 'receipt': <class 'vishalakshi.extract.Receipt'>, 'catalogue': <class 'vishalakshi.extract.Catalogue'>, 'contract': <class 'vishalakshi.extract.Contract'>, 'resume': <class 'vishalakshi.extract.Resume'>, 'paper': <class 'vishalakshi.extract.Paper'>, 'meeting_notes': <class 'vishalakshi.extract.MeetingNotes'>, 'other': <class 'vishalakshi.extract.Summary'>}] ['number', 'date', 'due_date', 'vendor', 'vendor_tax_id', 'bill_to', 'ship_to', 'currency', 'subtotal', 'tax', 'total', 'payment_terms', 'items']

/Users/71293/code/personal/orgs/vishalakshi/vishalakshi/extract.py:580: UserWarning: ValueError on a constrained call for Answer (model neither called the tool nor returned JSON; reply: 'The invoice details are as follows:\n*   **Invoice No:** ACM-2024-0117 [1]\n*   **Date:** 202); retrying as a JSON reply.
  warnings.warn(f'{type(e).__name__} on a constrained call for {schema.__name__} '

('Invoice',
 {'number': 'ACM-2024-0117',
  'date': '2024-03-01',
  'due_date': '',
  'vendor': 'Acme Supplies Ltd',
  'vendor_tax_id': '',
  'bill_to': 'Contoso GmbH, Berlin',
  'ship_to': '',
  'currency': 'EUR',
  'subtotal': 1240.0,
  'tax': 0,
  'total': 1240.0,
  'payment_terms': '',
  'items': ['widgets x10 @ 124.00']},
 {'question': 'what is owed, to whom, and by when?',
  'model': 'litert/litert-community/gemma-4-E2B-it-litert-lm',
  'runtime': 'litert',
  'context': {'results': [{'node_id': '4c1bdc4b329e3c3d#0', 'title': 'Acme invoice 0117', 'doc_id': '4c1bdc4b329e3c3d', 'breadcrumb': 'Acme invoice 0117', 'filename': '/inbox/acme-0117.md', 'pages': None, 'text': '# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00'}, {'node_id': 'e8024c5880508178#0', 'title': '05 mcp', 'doc_id': 'e8024c5880508178', 'breadcrumb': '05 mcp', 'filename': None, 'pages': (0, 0), 'text': "    'files anything you point it at; `note` writes your own conclusions back so they are searched '\n    'alongside the sources. `add_tree` points at a directory and splits it: documents into the '\n    'vault, source files into kosha. When kosha has indexed '\n    'the repo, `context` and `ask` add code sections to what they retrieve on their own, so a '\n    'question about the user\\'s own system is\n\n    'For one document rather than the corpus: `document` returns the whole of it, `categorize` says '\n    'what kind of thing it is (invoice, catalogue, contract, paper, …), `extract` pulls its fields '\n    'out against a schema (a name like `invoice`, or a spec like `vendor:str, total:float` you '\n    'make up on the spot), and `ask_doc` answers a question about that document with the rest of '\n "}, {'node_id': 'bd083e908521c339#24', 'title': '`chat_kw` reaches the rest of the constructor: `CachedChat` takes `path`, and pointing it at', 'doc_id': 'bd083e908521c339', 'breadcrumb': '06 extract › `chat_kw` reaches the rest of the constructor: `CachedChat` takes `path`, and pointing it at', 'filename': None, 'pages': (0, 0), 'text': "test_eq(a.cited.attrgot('node_id')[0], f'{a.doc_id}#0')\nassert 'VAT (20%)' in a.prompt, 'the whole document must reach the model, not a retrieved chunk'\nassert a.prompt.endswith('Question: what is the total due?')\n# the same question as data instead of prose, through a schema built at the moment of asking\nd = _x.ask_doc('/inbox/acme-0117.md', 'what is owed and to whom?',\n               schema='amo\n\ntest_eq(_x.document(_p).origin, 'disk')\ntest_eq(_x.categorize(str(_p), llm='never', save=False).doctype, 'meeting_notes')\n\n#| eval:false\n#| hide\n# the route that had to wait: what acquisition could not tell at the door, the doctype can\nv.shelf('papers', offline=True)          # pre-registered, so this test pays for no encoder download\nr = v.reshelf('Acme invoice', llm='never')\ntest_eq((r.doctype, \n\nanswer), ('Answer', ['amount', 'currency', 'owed_to'], None))\ntest_eq(d.fields['amount'], 180.0)\n# and it is `ask` underneath\ntest_eq(_x.ask('what is the total due?', ref='/inbox/acme-0117.md', related=2, **_m).answer, a.answer)\n\n# a file the vault has never seen, straight off disk: no model needed to prove where it came from\nfrom fastcore.all import Path\nfrom tempfile import mkdtemp\n_p = Path(mkd"}, {'node_id': '10bdb7299a36ecf7#0', 'title': '02 ask', 'doc_id': '10bdb7299a36ecf7', 'breadcrumb': '02 ask', 'filename': None, 'pages': (0, 0), 'text': 'transcripts, files and their own notes. Answer only from those sections.\n\nRules:\n- Cite every claim with the bracketed number of the section it came from, like [2]. A sentence\n  drawing on two sections cites both.\n- If the sections do not answer the question, say exactly what is missing rather than filling the\n  gap from memory. A vault that admits a hole is useful; one that guesses is not.\n- Sect\n\n- Say what you are holding back and why, in one line, so the questioner knows something is \\\nthere rather than assuming there is nothing.\n- If the question cannot be answered without a personal detail, do not answer it. Say what \\\ninstruction would let you answer it usefully (a count, a comparison, a yes or no, a total) \\\nand stop. The questioner will send that instruction back and you will get an\n\nwhich must never be shown any of it.\n\nRules, in order of importance:\n- Never reproduce a personal detail. No names, addresses, emails, phone numbers, account or \\\ncard numbers, dates of birth, or medical specifics: not in your answer, not as an example, \\\nnot to show your working, not even partially or obfuscated.\n- Answer at the level of shape and quantity instead: how many, what kind, which peri'}],
   'related': [],
   'encoder': 'minishlab/potion-multilingual-128M (256d, float16, model2vec)',
   'doc': {'doc_id': '4c1bdc4b329e3c3d',
    'title': 'Acme invoice 0117',
    'source': '/inbox/acme-0117.md',
    'kind': 'file',
    'meta': {'doctype': 'invoice',
     'doctype_by': 'cues (ner+regex)',
     'doctype_score': 0.7},
    'pages': 1,
    'origin': 'vault',
    'nodes': 2,
    'chars': 179,
    'truncated': False,
    'text': '# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00'},
   'docs': [{'doc_id': '4c1bdc4b329e3c3d', 'title': 'Acme invoice 0117', 'source': '/inbox/acme-0117.md', 'kind': 'file', 'meta': {'doctype': 'invoice', 'doctype_by': 'cues (ner+regex)', 'doctype_score': 0.7}, 'pages': 1, 'origin': 'vault', 'nodes': 2, 'chars': 179, 'truncated': False, 'text': '# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00'}],
   'n_docs': 1,
   'note': '[1] is the document being asked about; the rest is context from elsewhere in the vault.\n\n'},
  'encoder': 'minishlab/potion-multilingual-128M (256d, float16, model2vec)',
  'prompt': "[1] Acme invoice 0117\n(source: /inbox/acme-0117.md)\n\n# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00\n\n---\n\n[2] 05 mcp\n(source: e8024c5880508178, pages 0–0)\n\n    'files anything you point it at; `note` writes your own conclusions back so they are searched '\n    'alongside the sources. `add_tree` points at a directory and splits it: documents into the '\n    'vault, source files into kosha. When kosha has indexed '\n    'the repo, `context` and `ask` add code sections to what they retrieve on their own, so a '\n    'question about the user\\'s own system is\n\n    'For one document rather than the corpus: `document` returns the whole of it, `categorize` says '\n    'what kind of thing it is (invoice, catalogue, contract, paper, …), `extract` pulls its fields '\n    'out against a schema (a name like `invoice`, or a spec like `vendor:str, total:float` you '\n    'make up on the spot), and `ask_doc` answers a question about that document with the rest of '\n \n\n---\n\n[3] 06 extract › `chat_kw` reaches the rest of the constructor: `CachedChat` takes `path`, and pointing it at\n(source: bd083e908521c339, pages 0–0)\n\ntest_eq(a.cited.attrgot('node_id')[0], f'{a.doc_id}#0')\nassert 'VAT (20%)' in a.prompt, 'the whole document must reach the model, not a retrieved chunk'\nassert a.prompt.endswith('Question: what is the total due?')\n# the same question as data instead of prose, through a schema built at the moment of asking\nd = _x.ask_doc('/inbox/acme-0117.md', 'what is owed and to whom?',\n               schema='amo\n\ntest_eq(_x.document(_p).origin, 'disk')\ntest_eq(_x.categorize(str(_p), llm='never', save=False).doctype, 'meeting_notes')\n\n#| eval:false\n#| hide\n# the route that had to wait: what acquisition could not tell at the door, the doctype can\nv.shelf('papers', offline=True)          # pre-registered, so this test pays for no encoder download\nr = v.reshelf('Acme invoice', llm='never')\ntest_eq((r.doctype, \n\nanswer), ('Answer', ['amount', 'currency', 'owed_to'], None))\ntest_eq(d.fields['amount'], 180.0)\n# and it is `ask` underneath\ntest_eq(_x.ask('what is the total due?', ref='/inbox/acme-0117.md', related=2, **_m).answer, a.answer)\n\n# a file the vault has never seen, straight off disk: no model needed to prove where it came from\nfrom fastcore.all import Path\nfrom tempfile import mkdtemp\n_p = Path(mkd\n\n---\n\n[4] 02 ask\n(source: 10bdb7299a36ecf7, pages 0–0)\n\ntranscripts, files and their own notes. Answer only from those sections.\n\nRules:\n- Cite every claim with the bracketed number of the section it came from, like [2]. A sentence\n  drawing on two sections cites both.\n- If the sections do not answer the question, say exactly what is missing rather than filling the\n  gap from memory. A vault that admits a hole is useful; one that guesses is not.\n- Sect\n\n- Say what you are holding back and why, in one line, so the questioner knows something is \\\nthere rather than assuming there is nothing.\n- If the question cannot be answered without a personal detail, do not answer it. Say what \\\ninstruction would let you answer it usefully (a count, a comparison, a yes or no, a total) \\\nand stop. The questioner will send that instruction back and you will get an\n\nwhich must never be shown any of it.\n\nRules, in order of importance:\n- Never reproduce a personal detail. No names, addresses, emails, phone numbers, account or \\\ncard numbers, dates of birth, or medical specifics: not in your answer, not as an example, \\\nnot to show your working, not even partially or obfuscated.\n- Answer at the level of shape and quantity instead: how many, what kind, which peri\n\n---\n\n[1] is the document being asked about; the rest is context from elsewhere in the vault.\n\nQuestion: what is owed, to whom, and by when?",
  'answer': None,
  'thinking': '',
  'cited': [],
  'schema': 'Answer',
  'fields': {'amount': 1240.0,
   'payee': 'Contoso GmbH, Berlin',
   'due': '2024-03-31'},
  'doc_id': '4c1bdc4b329e3c3d',
  'title': 'Acme invoice 0117',
  'source': '/inbox/acme-0117.md',
  'origin': 'vault',
  'chars': 179,
  'truncated': False,
  'pii': {'has_pii': False,
   'kinds': {},
   'identifying': {},
   'n': 0,
   'scanned': 3395,
   'density': 0.0,
   'spans': []},
  'usage': total=1,674|in=1,623|out=51|turns=1})

Code

index_code builds a kosha index of a repository: its functions and classes, their names and the call graph. Once that index exists, context and ask add code sections beside the prose (code=). A code hit has no node_id; it is cited by path:line on disk. grep searches the files on disk with ripgrep and needs no index, and federate merges prose, code and ripgrep results into one ranking.

v.index_code(root)                # fills .kosha/; context then appends code sections
c = v.context('where do topic nodes get rebuilt?', sections=3, related=0, code=3, dir=root)
# code hits have no node_id; their handle is path:line on disk
c.code, L(c.results).filter(lambda r: r.node_id is None).attrgot('breadcrumb')

parse files from /Users/71293/code/personal/orgs/vishalakshi: 100%|██████████| 18/18 [00:00<00:00, 225.97it/s]

<style> progress { appearance: none; border: none; border-radius: 4px; width: 300px; height: 20px; vertical-align: middle; background: #e0e0e0; } progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; } progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; } progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; } progress:not([value]) { background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); } progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; } progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; } progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; } progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; } progress.progress-bar-interrupted { background: #F44336; } table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; } table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; } table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; } table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; } </style> <style> progress { appearance: none; border: none; border-radius: 4px; width: 300px; height: 20px; vertical-align: middle; background: #e0e0e0; } progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; } progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; } progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; } progress:not([value]) { background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); } progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; } progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; } progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; } progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; } progress.progress-bar-interrupted { background: #F44336; } table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; } table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; } table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; } table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; } </style>

(3,
 ['repo › /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/core.py:520', 'grep › README.md:119', 'repo › /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/core.py:547'])

L(v.grep('rrf_all', root, limit=4)).attrgot('where')   # ripgrep; no kosha needed

['README.md:125', 'nbs/index.ipynb:573', 'vishalakshi/code.py:12', 'vishalakshi/code.py:128']

Watches and the queue

A watch repeats an acquisition on a schedule. Its action is an acquisition method (url, web, harvest, arxiv, youtube, crawl or path), remind to write a note, or folder; kind= is the same thing under another name. A folder watch keeps a snapshot of a directory and fires with what was added, changed or removed, together with its instructions for whoever reviews the change. A schedule is every='6h' or cron='0 9 * * 1', and pobblebonk keeps the schedules in the vault file.

poll() is the tick to call from cron or an app’s timer. It returns jobs whose worker died to the queue, queues the watches that are due, runs the queue, and every ten minutes runs housekeep. A failed fetch is retried with a growing delay and set aside as dead after five attempts; jobs(state='dead') lists those. See jobs.

v.watch('https://example.com/changelog', action='url', every='6h')
v.watch('late chunking retrieval', action='web', every='1d', n=5)
v.watch('Re-read the evals', action='remind', every='1w')
v.watch(Path(root)/'nbs', kind='folder', every='1h', instructions='flag any print() left in a notebook', pattern='*.ipynb')
L(v.watches()).map(lambda w: (w['kind'], w['target'][:34], w['every'], w['pattern']))

Books and scans

A scanned book has no usable text layer, and an old print set in a legacy font may have one that reads as gibberish. read_doc reads a PDF or a page image page by page and picks a route for each page (parse='auto'): a text layer that passes its checks is kept, a scanned page is OCRed, and a page still scanned or flagged as garbled goes to a vision model, local or hosted. preset= picks the reading prompt (generic, sanskrit, tamil or tables), and hint= adds a sentence about the book. Given a folder, it reads every PDF and image under it. Each page’s text, route and flags are kept as numbered versions, so reading a page again with a better model (reparse) never loses the earlier reading. A page image cannot be masked, so a Personal page goes to a hosted model only with allow=True, and a Local only page never does. Readings are recorded and replayed through CachedChat, and a page with no recording yet reaches the model only when $VISHALAKSHI_RECORD_CHAT is set or a chat= is passed (read).

proofread has other models read the pages again and files the word changes each reading would make. A change two readers agree on is proposed, and one that only one reader makes is in doubt. Nothing changes until the owner decides: accept applies changes as a new version of the page, reject sets them aside, and undo_decision and revert_page take decisions back (proof).

v.read_doc('books/', parse='auto', model='claude/sonnet', preset='sanskrit', hint='verses end in ॥ n ॥')
doc = v.read_doc('scan.pdf', parse='llm', model='gemini/gemini-2.5-flash')['doc_id']
v.pages(doc)                                   # text, route (text, ocr, model, proof), flags, version
v.reparse(doc, pages=[3, 4], model='claude/opus')
v.proofread(doc, ['claude/sonnet', 'gemini/gemini-2.5-pro'])
v.changes(doc, 'proposed'); v.accept(doc); print(v.proof_md(doc))
read_doc('scan.pdf', parse='auto', model='claude/sonnet')   # no vault: the pages alone

The rest

page	what is on it
core	Vault, documents and sections, shelves, marks, context, topic maps
acquire	grab, url, web, crawl, arxiv, pdf, youtube, github, apis, harvest, images, watches
ask	ask, citations, the privacy policies, send_check and the send log, CachedChat
code	kosha, symbol, where_to_add, grep, federate
cli	vault methods as vishalakshi commands
mcp	vishalakshi-mcp, the vault as MCP tools
extract	doctypes from varga: categorize, extract, extract_all, ask_doc, reshelf
concepts	embedding models, model backends, shelves, and when to change the defaults
skill	the cheat sheet an agent loads, and what it may call
pii	personal information over rahasya: levels, the floor, findings, marks, gated
quality	the noise score, suggest_noisy and accept_noisy, feedback, the ranker
jobs	Queue: retries, leases, dead jobs
read	read_doc, reparse, page versions, presets, caching, progress and cancelling
proof	proofread and the review queue: changes, accept, reject, edit_change, undo_decision, revert_page

vishalakshi grab https://example.com/post
vishalakshi ask "why does late chunking help"

variable	what it sets
$VISHALAKSHI_VAULT	the vault file
$VISHALAKSHI_MODEL	the default model
$VISHALAKSHI_PII_MODEL	the local model that answers when sections are private
$VISHALAKSHI_GPU=0	keeps LiteRT on the CPU
$VISHALAKSHI_OFFLINE	never download a model; use the hashing encoder
$VISHALAKSHI_CHAT_CACHE	the folder CachedChat records replies in
$VISHALAKSHI_RECORD_CHAT	lets CachedChat ask a real model when it has no recording

The MCP client configuration is on mcp. Retrieval trade-offs and the measured defaults are in concepts and evals/RESULTS.md.

Development

The notebooks in nbs/ are the source, and nbdev-export writes the modules from them.

pip install -e .
nbdev-prepare