Skip to main content

search-expert



 search-expert

Natural language → structured search queries, instantly.
Fine-tuned Qwen3.5-0.8B LoRA adapters for search query parsing across 10 domains.


PyPI version Python License: MIT Open In Colab




What it does

"Non-stop business class from JFK to Tokyo under $3,000"
{
  "domain":      "flights",
  "origin":      "JFK",
  "destination": "Tokyo",
  "cabin_class": "business",
  "stops":       "lte:0",
  "price":       "lt:3000"
}

Search Expert uses a custom fine-tuned Small Language Model to understand natural language queries and extract only the fields explicitly mentioned — never hallucinating values that aren't there. It works across 10 search verticals out of the box.


Install

pip install search-expert

Usage

Basic

from search_expert import SearchExpert, ModelFormat, ParseResult

expert = SearchExpert()  # loads the JSON adapter by default

result = expert.parse("noise cancelling headphones any colour but red or green, under $200")
print(result.fields)
{
    'domain':  'ecommerce',
    'product': 'headphones',
    'feature': 'noise cancelling',
    'color':   ['ne:red', 'ne:green'],
    'price':   'lt:200'
}

Why hybrid search beats the alternatives

Text-to-SQL Pure vector search Hybrid search (this pipeline)
Hard constraints (price, brand, color) ✅ ❌ ✅
Semantic intent ("good for travel") ❌ ✅ ✅
Ranked results by relevance ❌ ✅ ✅
Works on unstructured descriptions ❌ ✅ ✅
Respects exclusions ("not black") ✅ ❌ ✅
Price is a hard cutoff, not a soft signal ✅ ❌ ✅

Text-to-SQL is a lookup tool — it returns rows that match, but can't rank by relevance or understand semantics.
Pure vector search is a semantic tool — it understands meaning, but treats "$200" as a soft hint, not a hard rule. A $350 product can rank above a $180 one if its description is more similar to the query.
This pipeline is a retrieval tool — structured filters enforce the hard constraints first, then vector search ranks the surviving candidates by semantic relevance.

In production, we generally use a version of this pattern:
structured pre-filtering → ANN (approximate nearest neighbour) vector search → learning-to-rank re-ranker.

search-expert makes step 1 trivial with a tiny, fast, locally-runnable model.


Operator reference

All numeric and exclusion constraints use a consistent prefix so downstream filters need zero NLP — just parse the string.

Query phrase Output value
"under $200", "below $200" lt:200
"up to $200", "max $200" lte:200
"over $150k", "above $150k" gt:150000
"at least $150k", "$150k+" gte:150000
"around $200", "~$200" approx:200
"$100–$200", "between $100 and $200" between:100:200
"any colour but red or green" ["ne:red", "ne:green"]

Applying a filter in one line:

result = expert.parse("apartments under $2,500/month in Austin")
salary = result.get_numeric_constraint("price")
# {'operator': 'lt', 'value': 2500.0, 'value_hi': None}

filtered = [l for l in listings if l["price"] < salary["value"]]

Domains present in the training data

Domain Example query
real_estate "2BR apartment in Austin under $1,500/month"
ecommerce "Sony noise cancelling headphones under $300"
jobs "Remote senior ML engineer paying over $150k"
flights "Non-stop business class JFK to Tokyo under $3,000"
hotels "5-star hotel in Paris with breakfast under $400/night"
cars "Electric SUV with 300+ mile range under $50k"
restaurants "Vegan Italian in NYC with outdoor seating under $40"
movies "Thriller on Netflix with 8+ IMDB rating"
healthcare "Female therapist in Chicago accepting Aetna"
courses "Python ML course for beginners under $30"
events "Taylor Swift concert in London in July"

The model

Each adapter is a LoRA fine-tune of Qwen3.5-0.8B trained on 100,000 (query, structured output) pairs spanning all 10 domains above.

Adapter HuggingFace Format
JSON (default) sarthakrastogi/search-expert-json-0.8b JSON
YAML sarthakrastogi/search-expert-yaml-0.8b YAML

Format leaderboard (held-out test set, 300 samples per format):

Rank Format Key F1 Value Acc Parse Rate
🥇 JSON 0.913 0.874 98.2%
🥈 YAML 0.901 0.861 97.6%
🥉 TOML 0.887 0.843 96.1%
4. XML 0.871 0.829 94.8%
5. CSV key=value 0.856 0.812 93.3%

Both public adapters return the same Python dict — the format only affects the model's internal generation language.


I evaluated search-expert's hybrid pipeline against Amazon's native search and pure vector search on 25 complex, multi-constraint ecommerce queries (e.g. "Sony noise cancelling headphones, not black, under $250").

Each pipeline's top-6 results were scored on whether they actually satisfied the hard constraints in the query — price limits, color exclusions, and color requirements. No relevance proxies; just "did the result follow the rules?"

Metric Amazon Pure Vector Hybrid (search-expert)
Price satisfaction 0.95 0.76 1.00
Color exclusion 0.66 0.78 1.00
Color match 0.62 0.47 1.00
Overall (all constraints) 0.76 0.56 1.00
Perfect@6 (all 6 correct) 0.42 0.21 1.00

Perfect@6 is the strictest metric: 1.0 only if every result in the top 6 satisfies every constraint simultaneously.

The hybrid pipeline scores 1.0 across the board. Amazon's search respects price constraints reasonably well but frequently returns wrong-colored results (only 62% color match on average). Pure vector search is the weakest on color — it understands what kind of thing you want, but treats "not black" as a soft suggestion rather than a hard rule.

Full benchmark code and dataset: benchmarks/amazon/ · Dataset on HuggingFace


Comparison: Why not just use spaCy/NER instead?

Capability Description spaCy / NER search-expert
Named entity extraction Pulls out cities, brands, product names, job titles from free text ✅ good ✅
Numeric value extraction Detects numbers like $200, 5 stars, 2 bedrooms ✅ good ✅
Comparison operator inference Maps phrases like "under", "at least", "starting from" to lt: / gte: operators ⚠️ partial, brittle rules ✅
Range constraints Understands "between $100 and $300" or "$100–$300" as between:100:300 ⚠️ regex only ✅
Approximation operator Maps "around $200", "roughly $200", "~$200" to approx:200 ❌ ✅
Exclusion constraints Maps "any colour but red or green" to ["ne:red", "ne:green"] ❌ needs custom logic ✅
Multi-domain field assignment Correctly assigns "noise cancelling" to feature vs. a brand or city depending on domain ❌ explodes in rules ✅
Novel phrasing generalisation Handles phrasings not seen during development without rule updates ❌ misses anything not in rules ✅ generalises
Inference speed Time to parse one query ✅ ~1 ms (CPU) ⚠️ ~400 ms (GPU) / ~3 s (CPU)
GPU requirement Whether a GPU is needed for production-speed inference ✅ none ⚠️ optional
Maintenance cost Effort required to extend coverage to new domains, fields, or phrasings ❌ high — rules accumulate and conflict ✅ low — add training data

Repo structure

search-expert/
├── search_expert/        # Library source
│   ├── __init__.py
│   ├── expert.py         # SearchExpert class (main API)
│   ├── config.py         # Model IDs, prompts, format enum
│   ├── loader.py         # HF model loading (unsloth / peft / plain)
│   ├── parser.py         # Raw output → dict parsers
│   ├── result.py         # ParseResult dataclass
│   └── exceptions.py     # Custom exceptions
├── training/             # Fine-tuning pipeline
│   ├── finetune.py       # Training script
│   └── evaluate.py       # Format comparison leaderboard
├── tests/
│   └── test_search_expert.py
├── examples/
│   └── search_expert_colab.ipynb
├── pyproject.toml
└── README.md

Development

git clone https://github.com/sarthakrastogi/search-expert
cd search-expert
pip install -e ".[dev]"

pytest tests/ -v                                          # unit tests (no GPU needed)
SEARCH_EXPERT_RUN_MODEL_TESTS=1 pytest tests/ -v          # includes model inference tests

License

MIT © Sarthak Rastogi

Contributing

Contributions are very welcome! Please open an issue or submit a pull request with any improvements.

Contact

For questions, feedback, or just to say hi, you can reach me at:

Email

LinkedIn

Metadata

Release files for search-expert 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for search-expert 0.1.2
File Size Uploaded Provenance Actions
search_expert-0.1.2.tar.gz 26.9 kB Without provenance Details for search_expert-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for search-expert 0.1.2
File Interpreter ABI Platform Actions
search_expert-0.1.2-py3-none-any.whl Python 3 none any Details for search_expert-0.1.2-py3-none-any.whl

Total release size: 50.1 kB

Release files / search_expert-0.1.2.tar.gz

Download URL search_expert-0.1.2.tar.gz
Size 26.9 kB
Tags Source
SHA-256 checksum
How to use checksums
52b4b67732ecd9847bce17e2a5a7b23c0e3396b680df049d2086c6b038a79c2b
BLAKE2b-256 checksum
How to use checksums
f45e440e3d7d41395bdc52256705fbde8a7204ce239e8f29c16778ee08d7b975
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.6

Release files / search_expert-0.1.2-py3-none-any.whl

Download URL search_expert-0.1.2-py3-none-any.whl
Size 23.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9f92e15353815be207d68b6bfb34b9c8969f039f5370135f007844077ce49196
BLAKE2b-256 checksum
How to use checksums
4c8d3fd398efcf27799f2e3d753826d9335ce6d1a863292397b282deed4c08eb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.6

Security for search-expert 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page