needlesearch-local 0.1.1

A lightweight, local-first full-text search service.

Needle Search

Needle is a small full-text search engine built from scratch in Python. Give it JSON documents, tell it which fields matter, and it will handle indexing, ranking, filters, synonyms, and typo-tolerant search.

It is not tied to one kind of data. The same engine can search jobs, products, recipes, articles, or anything else that can be represented as JSON.

Search:  nike runing sneakers
                    ↓
Found:   Nike Pegasus Running Shoes
         Corrected "runing" to "running"

Why I built it

Needle started as a way to improve search in Recruit Hub, but the interesting part was bigger than job search. I wanted to understand what happens between a user typing a query and receiving useful, ranked results.

So instead of hiding everything behind an existing search platform, Needle implements the core pieces directly:

• an inverted index for fast text lookup;
• positional postings for phrase and proximity matching;
• BM25-style relevance ranking;
• different weights for important fields;
• context-aware typo correction with confidence scoring;
• bidirectional, multi-word synonyms, filters, and sorting;
• separate collections for unrelated datasets.

Python 3.11 or newer is all you need. There are no third-party dependencies.

Installation

Install the latest release from PyPI:

python3 -m pip install needlesearch-local

Confirm that the command is available:

needle --version

To work on the project itself, clone the repository and install it in editable mode:

git clone https://github.com/fishfoodfish/NeedleSearch.git
cd NeedleSearch
python3 -m pip install --editable .

Architecture

JSON documents
      │
      ▼
Schema validation
      │
      ▼
Text analysis and tokenization
      │
      ├──────────────► Trigram index ───► Context-aware typo correction
      │
      ▼
Inverted index
      │
      ▼
BM25 ranking + coverage + phrases + proximity + filters
      │
      ▼
CLI / HTTP API / browser

Each collection has its own schema, documents, index, synonyms, and autocorrect vocabulary. Words learned from the jobs collection never leak into products or recipes.

Quick start

After installing Needle, initialize its local data and configuration:

needle init

Create a product collection using the example schema from this repository:

needle create products --schema schemas/products.json

Add the example products:

needle ingest examples/products.json --collection products

Now try a search—with a typo on purpose:

needle search "nike runing sneakers" --collection products

Needle corrects runing, expands sneakers using the collection's synonyms, and ranks the Nike Pegasus result first.

For a stricter search, require a minimum number of query terms:

needle search "remote python internship" --collection jobs --minimum-should-match 2

By default, Needle keeps broad OR-style retrieval but reduces the score of documents that match less of the query. minimum_should_match removes partial matches entirely when an application needs more precision.

Use quotation marks when word order must be exact:

needle search '"software engineer"' --collection jobs

Unquoted terms can appear anywhere in a document, but terms that occur near one another receive a proximity boost.

To see every collection:

needle collections

To run the local server:

needle serve --port 8080

Then open http://127.0.0.1:8080. The current browser page searches the original default index; named collections are available through the collection API shown below.

If a collection already exists, you do not need to create it again. You can keep ingesting documents or search the existing collection.

Measuring search quality

Needle can evaluate a collection against queries with human-assigned relevance grades. Each line in a JSONL judgment file contains one query and the documents that should match it:

{"query":"amazon software internship","relevance":{"job-001":2}}
{"query":"python machine learning","relevance":{"job-003":2,"job-001":1}}

Use 2 for a highly relevant document, 1 for a partially relevant document, and 0 for an irrelevant document. Typo examples can also include the corrections Needle should make:

{"query":"amazn intership","relevance":{"job-001":2},"corrections":{"amazn":"amazon","intership":"internship"}}

Evaluate the example jobs collection:

needle evaluate examples/jobs-judgments.jsonl --collection jobs

The report includes nDCG, mean reciprocal rank, recall, zero-result rate, expected-no-result accuracy, correction accuracy, and average/P95 latency. Add --details to inspect the returned document IDs and score for each query. Keep the judgment file stable while refining the algorithm so improvements can be compared against the same baseline.

The larger benchmark in benchmarks/jobs/ contains 20 documents and 50 hand-labeled queries for comparing ranking changes.

Local files

Needle keeps its indexes and configuration outside the installed Python package:

Platform	Default data directory
macOS	~/Library/Application Support/NeedleSearch/
Linux	~/.local/share/needlesearch/
Windows	%LOCALAPPDATA%\NeedleSearch\

Check the installation and local indexes with:

needle --version
needle doctor

Another project can keep its search data alongside the application:

needle --data-dir ./search-data serve

The same location can be set through the NEEDLE_DATA_DIR environment variable. Use NEEDLE_CONFIG_FILE to override the configuration file.

Try another kind of data

The workflow stays the same no matter what you are searching.

Jobs

needle create jobs --schema schemas/jobs.json
needle ingest examples/jobs.json --collection jobs
needle search "amazn intership" --collection jobs

Recipes

needle create recipes --schema schemas/recipes.json
needle ingest examples/recipes.json --collection recipes
needle search "spicy chiken" --collection recipes

How a search works

Suppose you search for:

amazn intership

Needle handles it in a few stages:

1. The query is normalized and split into amazn and intership.
2. Neither word exists in the collection's vocabulary, so Needle checks its trigram index for similar words.
3. Damerau-Levenshtein distance handles insertions, deletions, replacements, and adjacent transpositions.
4. Candidate confidence combines edit similarity, trigram overlap, term popularity, and overlap with the other query terms.
5. High-confidence corrected terms point directly to matching documents in the inverted index. Uncertain candidates are returned as suggestions without changing the search.
6. BM25 scores the results. Matches in important fields such as title and company receive a larger boost.
7. Query coverage reduces the score of documents that match only part of the query, and minimum_should_match can remove them entirely.
8. The highest-scoring documents are returned with the suggested corrections.

Correction responses distinguish applied corrections from uncertain suggestions:

{
  "corrections": {"amazn": "amazon"},
  "suggestions": {},
  "correction_details": {
    "amazn": {
      "term": "amazon",
      "confidence": 0.749,
      "applied": true
    }
  }
}

When two candidates are too close in confidence, Needle leaves the original query unchanged and places the best candidate in suggestions.

The original query is still considered. Corrected words receive a smaller ranking weight, so an exact match always has the advantage.

Defining your own collection

A schema is a small JSON file that explains what your documents look like. There is no need to change Needle's Python code.

Here is a shortened product schema:

{
  "id_field": "sku",
  "fields": [
    {
      "name": "name",
      "type": "text",
      "searchable": true,
      "weight": 5
    },
    {
      "name": "brand",
      "type": "keyword",
      "searchable": true,
      "filterable": true,
      "weight": 2
    },
    {
      "name": "price",
      "type": "number",
      "searchable": false,
      "filterable": true,
      "sortable": true
    }
  ],
  "synonyms": {
    "swe": ["software engineer"],
    "intern": ["internship"]
  }
}

Synonyms are bidirectional. This configuration lets swe find software engineer, software engineer find swe, and internship find documents that use intern. A synonym value may contain one word or an entire phrase.

That schema accepts documents such as:

{
  "sku": "shoe-934",
  "name": "Nike Pegasus Running Shoes",
  "brand": "Nike",
  "price": 129.99
}

Available field types:

Type	Good for
text	Names, titles, descriptions, and other searchable writing
text[]	Ingredients, tags, skills, or any list of text
keyword	Brands, categories, or exact labels
number	Prices, salaries, ratings, and numeric ranges
boolean	Values such as remote or in_stock
date	ISO dates such as 2026-07-18

Needle validates every document before indexing it. If price is configured as a number and a document sends "cheap", ingestion fails with a clear error.

Using the HTTP API

The CLI is convenient for experimenting. Applications such as Recruit Hub would normally talk to Needle over HTTP.

Add documents:

POST /v1/collections/products/documents
Content-Type: application/json

Search:

POST /v1/collections/products/search
Content-Type: application/json

{
  "query": "nike runing shoes",
  "filters": {
    "price": {"lte": 150},
    "in_stock": true
  },
  "sort": [
    {"field": "_score", "order": "desc"}
  ],
  "limit": 20,
  "minimum_should_match": 2,
  "coverage_boost": 1.0
}

API routes

Method	Route	What it does
POST	/v1/collections	Creates a collection
GET	/v1/collections	Lists existing collections
GET	/v1/collections/{name}	Shows its schema and statistics
POST	/v1/collections/{name}/documents	Adds one or more documents
GET	/v1/collections/{name}/search?q=...	Runs a simple search
POST	/v1/collections/{name}/search	Searches with filters and sorting
DELETE	/v1/collections/{name}/documents/{id}	Removes a document
GET	/health	Checks whether the server is running

Core files

File	Purpose
src/needlesearch/analysis.py	Normalizes text, creates tokens and trigrams, and calculates bounded Damerau-Levenshtein distance
src/needlesearch/engine.py	Builds positional indexes, expands synonyms, finds corrections, matches phrases, filters documents, and ranks results
src/needlesearch/evaluation.py	Measures ranked relevance, corrections, zero results, and query latency
src/needlesearch/models.py	Defines schemas, field types, validation rules, and search results
src/needlesearch/collections.py	Keeps named collections and their indexes separate
src/needlesearch/config.py	Initializes configuration and checks local installation health
src/needlesearch/paths.py	Chooses safe data and configuration paths for each operating system
src/needlesearch/server.py	Exposes search through the browser and HTTP API
src/needlesearch/cli.py	Handles the create, ingest, search, and serve commands
schemas/	Example configurations for jobs, products, and recipes
examples/	Small datasets you can use while learning the project
tests/	Tests for indexing, ranking, corrections, collections, and the API

Project structure

Search/
├── src/
│   └── needlesearch/
│       ├── analysis.py
│       ├── cli.py
│       ├── collections.py
│       ├── config.py
│       ├── engine.py
│       ├── evaluation.py
│       ├── models.py
│       ├── paths.py
│       └── server.py
├── .github/workflows/
│   ├── ci.yml
│   └── release.yml
├── schemas/
│   ├── jobs.json
│   ├── jobs-judgments.jsonl
│   ├── products.json
│   └── recipes.json
├── examples/
│   ├── jobs.json
│   ├── products.json
│   └── recipes.json
└── tests/
    ├── test_collections.py
    ├── test_config.py
    ├── test_evaluation.py
    └── test_engine.py

Running the tests

python3 -m unittest discover -s tests -v

The tests cover ranking, typo correction, typed validation, filtering, sorting, persistence, isolated collection vocabularies, and the full HTTP lifecycle.

Every push and pull request runs the test suite on Python 3.11 through 3.14, builds the wheel and source archive, validates the PyPI metadata, and installs the wheel in a clean environment. Releases use PyPI Trusted Publishing, so the repository does not store publishing tokens.

Current scope

Needle is currently a single-node learning project. Text searches use posting lists instead of scanning every document, but filters still scan document metadata and saved indexes are rebuilt in memory when the engine starts.

The next meaningful improvements would be indexed filters, immutable disk segments, a write-ahead log, and background compaction. Distributed search can come later, once the single-node engine has real limits worth solving.

License

Needle Search is available under the MIT License.