termflow-ir 0.1.2

English text analysis for information retrieval

termflow

English text analysis for information retrieval workloads in C++ and Python.

termflow is a library-first analysis stack for search, indexing, tagging, and query normalization. It provides a built-in English analyzer, term extraction helpers, and a lightweight query rewrite layer without trying to be a full search engine.

Why termflow

• C++20 core library with optional Python bindings
• English analyzer with configurable stemming, stop words, possessive handling, and ASCII folding
• Term extraction API for finalized search/index terms
• Query parser and rewrite support for canonicalization, equivalents, and expansions
• Installable Python wheels for Linux and macOS
• CMake install flow for downstream C++ consumers

Install

Python package:

pip install termflow-ir

Python import:

import termflow

CLI quick check:

termflow analyze "The Running Cars"

For C++ installation and find_package(termflow) usage, see docs/installation.md.

Quick Start

Python:

import termflow

analyzer = termflow.EnglishAnalyzer()
terms = analyzer.analyze_terms("The Running Cars")
normalized = analyzer.normalize("Running Café")

print(terms)  # ['run', 'car']
print(normalized)  # 'running café'

C++:

#include <iostream>
#include "termflow/analysis/english_analyzer.hpp"

int main() {
  termflow::EnglishAnalyzer analyzer;
  const auto terms = analyzer.analyze_terms("The Running Cars");

  for (const auto& term : terms) {
    std::cout << term << "\n";
  }
}

Features

Area	What it includes
Analysis	EnglishAnalyzer, token analysis, normalization, stemming, stop words, ASCII folding
Term extraction	TermExtractor with length, numeric, and character-policy filtering
Query processing	clause parsing, analyzed query terms, rewrite loading, validation, and alternatives
Python bindings	built-in analyzer, term extractor, and query module under termflow.query
CLI	termflow analyze, termflow extract, and termflow analyze-query for quick validation
C++ consumption	installable CMake package and external find_package example

Documentation

• docs/usage.md for day-to-day analyzer, term extraction, query, and Python usage
• docs/customization.md for pipeline tuning, query rewrites, and custom analyzers in C++
• docs/installation.md for Python and C++ installation paths
• docs/installation-roadmap.md for packaging and distribution priorities

Runnable examples:

• examples/analyze_text.cpp
• examples/extract_terms.cpp
• examples/custom_analyzer.cpp
• examples/analyze_query.cpp
• examples/find_package_consumer/CMakeLists.txt

Scope

termflow currently focuses on:

• English text analysis
• Batch-oriented APIs
• Query parsing and rewrite preparation
• Reusable components for embedding in larger applications

termflow does not currently provide:

• indexing or retrieval
• ranking or scoring
• token graphs
• phrase execution logic
• multilingual analyzers

Build From Source

Local build:

cmake -S . -B build -G Ninja
cmake --build build
ctest --test-dir build --output-on-failure

Build Python bindings from source:

cmake -S . -B build -G Ninja -DTERMFLOW_BUILD_PYTHON=ON
cmake --build build
PYTHONPATH=build/python python3 -c 'import termflow; print(termflow.EnglishAnalyzer().analyze_terms("Running Cars"))'

Build Python distributions:

python3 -m build --sdist --wheel
python3 -m twine check dist/*

Project Status

termflow is early-stage and intentionally narrow in scope. The current focus is making the built-in English analysis and packaging story solid before expanding into more languages or broader IR features.

Contributing

Issues and pull requests are welcome. If you want to make a larger API or packaging change, open an issue first so the direction is clear before implementation work starts.

License

This repository does not yet include a LICENSE file. Until that is added, do not assume open source usage terms.