nlp4j-local-search-embedding

nlp4j-local-search-embedding provides simple embedding-based semantic search built on top of nlp4j-local-search.

It converts text into embeddings using a multilingual E5 model and stores the resulting vectors in a local vector search index.

Features

• Simple semantic search API
• Text-to-vector embedding with multilingual E5
• Local vector search using nlp4j-local-search
• E5 query/passage prefixes handled internally
• Supports document IDs, text, and metadata
• Field search — filter search results by document fields (added in v0.3.0)
• Designed as a lightweight bridge between local search and embedding models

Installation

nlp4j-local-search-embedding is available on PyPI.

Install the latest version with:

pip install nlp4j-local-search-embedding==0.3.0

This package depends on nlp4j-local-search and sentence-transformers.

The first run may take some time because the embedding model is downloaded and loaded locally.

Local Development Installation

For local development with editable installs:

git clone https://github.com/oyahiroki/nlp4j-local-search.git
git clone https://github.com/oyahiroki/nlp4j-local-search-embedding.git

python -m venv .venv
source .venv/bin/activate

python -m pip install -U pip setuptools wheel
python -m pip install -e ./nlp4j-local-search
python -m pip install -e ./nlp4j-local-search-embedding

On Windows PowerShell:

git clone https://github.com/oyahiroki/nlp4j-local-search.git
git clone https://github.com/oyahiroki/nlp4j-local-search-embedding.git

python -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install -U pip setuptools wheel
python -m pip install -e ./nlp4j-local-search
python -m pip install -e ./nlp4j-local-search-embedding

Quick Start

from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch("en")

app.add({
    "doc1": "Kyoto is a historic city in Japan.",
    "doc2": "Tokyo is the capital city of Japan.",
    "doc3": "Python is a popular programming language.",
})

app.commit()

results = app.search("an old Japanese capital", limit=10)

print("=== Search results ===")
print(f"number of results: {len(results)}")

for i, result in enumerate(results):
    print(f"result[{i}].id: {result.id}")
    print(f"result[{i}].text: {result.text}")
    print(f"result[{i}].score: {result.score}")
    print(f"result[{i}].metadata: {result.metadata}")
    print("---")

Example output:

=== Search results ===
number of results: 3
result[0].id: doc1
result[0].text: Kyoto is a historic city in Japan.
result[0].score: 0.91
result[0].metadata: {}
---
result[1].id: doc2
result[1].text: Tokyo is the capital city of Japan.
result[1].score: 0.88
result[1].metadata: {}
---
result[2].id: doc3
result[2].text: Python is a popular programming language.
result[2].score: 0.75
result[2].metadata: {}
---

Scores may vary depending on the model version and runtime environment.

Using Metadata

Documents can include metadata.

from nlp4j_local_search_embedding import SemanticSearch

documents = [
    {
        "id": "doc1",
        "text": "Kyoto is a historic city in Japan.",
        "metadata": {
            "category": "city",
            "country": "Japan"
        },
    },
    {
        "id": "doc2",
        "text": "Nintendo is a video game company headquartered in Kyoto.",
        "metadata": {
            "category": "company",
            "country": "Japan"
        },
    },
    {
        "id": "doc3",
        "text": "Python is widely used for data science and machine learning.",
        "metadata": {
            "category": "technology"
        },
    },
]

app = SemanticSearch("en")
app.add(documents)
app.commit()

results = app.search("a Japanese game company", limit=3)

for result in results:
    print(result.id)
    print(result.text)
    print(result.score)
    print(result.metadata)
    print("---")

Adding a Single Document

from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch("en")

app.add(
    "doc1",
    "Kyoto is known for temples, shrines, and traditional culture.",
    metadata={"category": "travel"}
)

app.commit()

results = app.search("traditional Japanese culture", limit=5)

for result in results:
    print(result.id, result.score, result.text)

Field Search

Added in v0.3.0.

Documents can be registered with fields — key-value pairs used for exact-match filtering. Use fields= when adding documents and filters= when searching.

filters conditions are combined with AND when multiple fields are specified. Field values are evaluated by exact (term) match and do not affect the similarity score.

Registering documents with fields

from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch("en")

app.add([
    {"id": "1", "text": "Kyoto is a historic city in Japan.",
     "fields": {"category": "city", "country": "Japan"}},
    {"id": "2", "text": "Nintendo is headquartered in Kyoto, Japan.",
     "fields": {"category": "company", "country": "Japan"}},
    {"id": "3", "text": "Tokyo is the capital city of Japan.",
     "fields": {"category": "city", "country": "Japan"}},
    {"id": "4", "text": "Paris is the capital city of France.",
     "fields": {"category": "city", "country": "France"}},
    {"id": "5", "text": "Sony is a Japanese multinational company.",
     "fields": {"category": "company", "country": "Japan"}},
])
app.commit()

Filtering by a single field

results = app.search("", limit=10, filters={"category": "city"})
for r in results:
    print(r.id, r.text)
# [1] Kyoto is a historic city in Japan.
# [3] Tokyo is the capital city of Japan.
# [4] Paris is the capital city of France.

Filtering by multiple fields (AND)

results = app.search("", limit=10, filters={"category": "city", "country": "Japan"})
for r in results:
    print(r.id, r.text)
# [1] Kyoto is a historic city in Japan.
# [3] Tokyo is the capital city of Japan.

Semantic search combined with field filtering

results = app.search("Japanese company", limit=10, filters={"category": "company"})
for r in results:
    print(r.id, r.score, r.text)
# Only documents with category="company" are returned, ranked by semantic similarity.

Two-argument form with fields

app.add("doc1", "Kyoto is a historic city in Japan.",
        fields={"category": "city", "country": "Japan"})

Using the Document class with fields

from nlp4j_local_search_embedding import Document, SemanticSearch

app = SemanticSearch("en")
app.add([
    Document(id="1", text="Kyoto is a historic city in Japan.",
             fields={"category": "city", "country": "Japan"}),
    Document(id="2", text="Nintendo is headquartered in Kyoto, Japan.",
             fields={"category": "company", "country": "Japan"}),
])
app.commit()

results = app.search("old capital", limit=5, filters={"category": "city"})

Default Model

The default embedding model is:

intfloat/multilingual-e5-large

SemanticSearch uses the E5-style prefixes internally:

passage: <document text>
query:   <search query>

Therefore, users can simply add plain document text and search with plain query text.

Specifying a Model

from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch(
    "en",
    model_name="intfloat/multilingual-e5-large"
)

Architecture

This package is designed as an embedding layer for nlp4j-local-search.

text documents
    |
    v
E5 embedding model
    |
    v
vectors
    |
    v
nlp4j-local-search vector index
    |
    v
semantic search results

The base package nlp4j-local-search is responsible for local search and vector indexing. This package is responsible for converting text into embeddings and providing a convenient SemanticSearch API.

Relationship with nlp4j-local-search

nlp4j-local-search can perform keyword search and vector search with user-provided vectors.

from nlp4j_local_search import SearchEngine

search = SearchEngine("en", vector_dimension=2)
search.add("east", [1.0, 0.0])
search.add("north", [0.0, 1.0])
search.commit()

results = search.search([0.9, 0.1], limit=10)

nlp4j-local-search-embedding adds text embedding support on top of that.

from nlp4j_local_search_embedding import SemanticSearch

search = SemanticSearch("en")
search.add({
    "doc1": "Kyoto is a historic city in Japan.",
    "doc2": "Tokyo is the capital city of Japan.",
})
search.commit()

results = search.search("old Japanese capital", limit=10)

Saving Document Text and Metadata

The vector index is managed by nlp4j-local-search.

This package also keeps document text and metadata on the Python side so that search results can include the original text.

app.save_documents("documents.json")

To restore the document text and metadata:

app.load_documents("documents.json")

Note: index persistence and document-store persistence may be handled separately depending on the version of nlp4j-local-search.

Development

Install the package in editable mode:

python -m pip install -e .

Run an example:

python examples/simple_semantic_search.py

Run tests:

python -m pip install pytest
python -m pytest

Project Structure

nlp4j-local-search-embedding/
  src/
    nlp4j_local_search_embedding/
      __init__.py
      document.py
      e5_embedder.py
      errors.py
      result.py
      semantic_search.py
  examples/
    simple_semantic_search.py
    example_0.3.0.py
  tests/
  pyproject.toml
  README.md
  README_build.md
  LICENSE

Notes

• The first run may take time because the embedding model needs to be downloaded and loaded.
• The package depends on sentence-transformers.
• The base vector search functionality is provided by nlp4j-local-search.
• This package is intended for local semantic search, experimentation, and lightweight RAG-style applications.

License

Apache License 2.0