prepembd

Installation

Install prepembd:

python3 -m pip install prepembd --upgrade

Requirements

• Python >= 3.10

Why

I've been using markdown now for a long time to take notes in every possible scenario. I even manage my Anki cards with markdown (inka2) so finding relevant information again is paramount. With the advent of semantic search via Embeddings search became so much more powerfull. However, to create the embeddings out of markdown the files have to be prepared in order to reduce noice and create the correct chunk sizes.

This Python script automates the process and creates a json representation of all the markdown files which then can be fed into an embedding model. It is basically just a thin wrapper aroung LangChain combined with some bespoke filter to eliminated noise.

Usage

prepembd tokenize <directory> | tee -a output.ndjson