needlesearch-local 0.1.0

A lightweight, local-first full-text search service.

Needle Search

Needle is a small full-text search engine built from scratch in Python. Give it JSON documents, tell it which fields matter, and it will handle indexing, ranking, filters, synonyms, and typo-tolerant search.

It is not tied to one kind of data. The same engine can search jobs, products, recipes, articles, or anything else that can be represented as JSON.

Search:  nike runing sneakers
                    ↓
Found:   Nike Pegasus Running Shoes
         Corrected "runing" to "running"

Why I built it

Needle started as a way to improve search in Recruit Hub, but the interesting part was bigger than job search. I wanted to understand what happens between a user typing a query and receiving useful, ranked results.

So instead of hiding everything behind an existing search platform, Needle implements the core pieces directly:

• an inverted index for fast text lookup;
• BM25-style relevance ranking;
• different weights for important fields;
• typo correction based on the indexed vocabulary;
• synonyms, filters, and sorting;
• separate collections for unrelated datasets.

Python 3.11 or newer is all you need. There are no third-party dependencies.

Installation

Once the first release is published, install Needle with:

python3 -m pip install needlesearch-local

Until then, install the current repository locally:

python3 -m pip install .

Architecture

JSON documents
      │
      ▼
Schema validation
      │
      ▼
Text analysis and tokenization
      │
      ├──────────────► Trigram index ───► Typo correction
      │
      ▼
Inverted index
      │
      ▼
BM25 ranking + field weights + filters
      │
      ▼
CLI / HTTP API / browser

Each collection has its own schema, documents, index, synonyms, and autocorrect vocabulary. Words learned from the jobs collection never leak into products or recipes.

Quick start

After installing Needle, initialize its local data and configuration:

needle init

Create a product collection using the example schema from this repository:

needle create products --schema schemas/products.json

Add the example products:

needle ingest examples/products.json --collection products

Now try a search—with a typo on purpose:

needle search "nike runing sneakers" --collection products

Needle corrects runing, expands sneakers using the collection's synonyms, and ranks the Nike Pegasus result first.

To see every collection:

needle collections

To run the local server:

needle serve --port 8080

Then open http://127.0.0.1:8080. The current browser page searches the original default index; named collections are available through the collection API shown below.

If a collection already exists, you do not need to create it again. You can keep ingesting documents or search the existing collection.

Local files

Needle keeps its indexes and configuration outside the installed Python package:

Platform	Default data directory
macOS	~/Library/Application Support/NeedleSearch/
Linux	~/.local/share/needlesearch/
Windows	%LOCALAPPDATA%\NeedleSearch\

Check the installation and local indexes with:

needle --version
needle doctor

Another project can keep its search data alongside the application:

needle --data-dir ./search-data serve

The same location can be set through the NEEDLE_DATA_DIR environment variable. Use NEEDLE_CONFIG_FILE to override the configuration file.

Try another kind of data

The workflow stays the same no matter what you are searching.

Jobs

needle create jobs --schema schemas/jobs.json
needle ingest examples/jobs.json --collection jobs
needle search "amazn intership" --collection jobs

Recipes

needle create recipes --schema schemas/recipes.json
needle ingest examples/recipes.json --collection recipes
needle search "spicy chiken" --collection recipes

How a search works

Suppose you search for:

amazn intership

Needle handles it in a few stages:

1. The query is normalized and split into amazn and intership.
2. Neither word exists in the collection's vocabulary, so Needle checks its trigram index for similar words.
3. Edit distance confirms that amazon and internship are close matches.
4. The corrected terms point directly to matching documents in the inverted index.
5. BM25 scores the results. Matches in important fields such as title and company receive a larger boost.
6. The highest-scoring documents are returned with the suggested corrections.

The original query is still considered. Corrected words receive a smaller ranking weight, so an exact match always has the advantage.

Defining your own collection

A schema is a small JSON file that explains what your documents look like. There is no need to change Needle's Python code.

Here is a shortened product schema:

{
  "id_field": "sku",
  "fields": [
    {
      "name": "name",
      "type": "text",
      "searchable": true,
      "weight": 5
    },
    {
      "name": "brand",
      "type": "keyword",
      "searchable": true,
      "filterable": true,
      "weight": 2
    },
    {
      "name": "price",
      "type": "number",
      "searchable": false,
      "filterable": true,
      "sortable": true
    }
  ],
  "synonyms": {
    "sneakers": ["shoes", "trainers"]
  }
}

That schema accepts documents such as:

{
  "sku": "shoe-934",
  "name": "Nike Pegasus Running Shoes",
  "brand": "Nike",
  "price": 129.99
}

Available field types:

Type	Good for
text	Names, titles, descriptions, and other searchable writing
text[]	Ingredients, tags, skills, or any list of text
keyword	Brands, categories, or exact labels
number	Prices, salaries, ratings, and numeric ranges
boolean	Values such as remote or in_stock
date	ISO dates such as 2026-07-18

Needle validates every document before indexing it. If price is configured as a number and a document sends "cheap", ingestion fails with a clear error.

Using the HTTP API

The CLI is convenient for experimenting. Applications such as Recruit Hub would normally talk to Needle over HTTP.

Add documents:

POST /v1/collections/products/documents
Content-Type: application/json

Search:

POST /v1/collections/products/search
Content-Type: application/json

{
  "query": "nike runing shoes",
  "filters": {
    "price": {"lte": 150},
    "in_stock": true
  },
  "sort": [
    {"field": "_score", "order": "desc"}
  ],
  "limit": 20
}

API routes

Method	Route	What it does
POST	/v1/collections	Creates a collection
GET	/v1/collections	Lists existing collections
GET	/v1/collections/{name}	Shows its schema and statistics
POST	/v1/collections/{name}/documents	Adds one or more documents
GET	/v1/collections/{name}/search?q=...	Runs a simple search
POST	/v1/collections/{name}/search	Searches with filters and sorting
DELETE	/v1/collections/{name}/documents/{id}	Removes a document
GET	/health	Checks whether the server is running

Core files

File	Purpose
src/needlesearch/analysis.py	Normalizes text, creates tokens and trigrams, and calculates edit distance
src/needlesearch/engine.py	Builds indexes, finds corrections, filters documents, and ranks results
src/needlesearch/models.py	Defines schemas, field types, validation rules, and search results
src/needlesearch/collections.py	Keeps named collections and their indexes separate
src/needlesearch/server.py	Exposes search through the browser and HTTP API
src/needlesearch/cli.py	Handles the create, ingest, search, and serve commands
schemas/	Example configurations for jobs, products, and recipes
examples/	Small datasets you can use while learning the project
tests/	Tests for indexing, ranking, corrections, collections, and the API

Project structure

Search/
├── src/
│   └── needlesearch/
│       ├── analysis.py
│       ├── cli.py
│       ├── collections.py
│       ├── engine.py
│       ├── models.py
│       └── server.py
├── schemas/
│   ├── jobs.json
│   ├── products.json
│   └── recipes.json
├── examples/
│   ├── jobs.json
│   ├── products.json
│   └── recipes.json
└── tests/
    ├── test_collections.py
    └── test_engine.py

Running the tests

python3 -m unittest discover -s tests -v

The tests cover ranking, typo correction, typed validation, filtering, sorting, persistence, isolated collection vocabularies, and the full HTTP lifecycle.

Every push and pull request runs the test suite on Python 3.11 through 3.14, builds the wheel and source archive, validates the PyPI metadata, and installs the wheel in a clean environment. Releases use PyPI Trusted Publishing, so the repository does not store publishing tokens.

Current scope

Needle is currently a single-node learning project. Text searches use posting lists instead of scanning every document, but filters still scan document metadata and saved indexes are rebuilt in memory when the engine starts.

The next meaningful improvements would be indexed filters, immutable disk segments, a write-ahead log, and background compaction. Distributed search can come later, once the single-node engine has real limits worth solving.

License

Needle Search is available under the MIT License.