swarmauri_tool_lexicaldensity 0.10.1.dev2

Lexical Density Tool for Swarmauri.

---

Swarmauri Tool · Lexical Density

A Swarmauri-compatible NLP utility that measures the lexical density of text—the ratio of content words (nouns, verbs, adjectives, adverbs) versus the total token count. Use it to monitor writing complexity, automate readability checks, or surface style signals inside agent conversations.

• Tokenizes input with NLTK and tags parts of speech to isolate lexical items.
• Returns a percentage score so changes in density are easy to compare between drafts.
• Ships as a Swarmauri tool, ready for registration inside agents or pipelines.

Requirements

• Python 3.10 – 3.13.
• nltk with the punkt_tab and averaged_perceptron_tagger_eng resources available (downloaded at runtime).
• textstat for robust lexicon counting.
• Core dependencies (swarmauri_base, swarmauri_standard, pydantic).

Installation

Pick the installer that matches your project; each command resolves the transitive requirements.

pip

pip install swarmauri_tool_lexicaldensity

Poetry

poetry add swarmauri_tool_lexicaldensity

uv

# Add to the current project and update uv.lock
uv add swarmauri_tool_lexicaldensity

# or install into the running environment without touching pyproject.toml
uv pip install swarmauri_tool_lexicaldensity

Tip: If you deploy in an offline environment, download the required NLTK models during build time (python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng).

Quick Start

from swarmauri_tool_lexicaldensity import LexicalDensityTool

text = "This report summarizes quarterly revenue growth across all segments."

lexical_density_tool = LexicalDensityTool()
result = lexical_density_tool(text)

print(result)
# {'lexical_density': 58.333333333333336}

The tool returns a floating-point percentage. Use the same instance to score multiple passages.

Usage Scenarios

Enforce Style Guidelines in Content Pipelines

from swarmauri_tool_lexicaldensity import LexicalDensityTool

product_copy = "Introducing our new AI-powered workstation with modular expansion."

checker = LexicalDensityTool()
score = checker(product_copy)["lexical_density"]

if score < 40:
    raise ValueError(f"Copy too simple (density={score:.1f}%). Add more substantive language.")

Gate marketing copy or documentation PRs based on desired complexity thresholds.

Analyze Conversations in a Swarmauri Agent

from swarmauri_core.agent.Agent import Agent
from swarmauri_standard.tools.registry import ToolRegistry
from swarmauri_tool_lexicaldensity import LexicalDensityTool

registry = ToolRegistry()
registry.register(LexicalDensityTool())
agent = Agent(tool_registry=registry)

utterance = "Could you elaborate on the architectural trade-offs in the data plane?"
result = agent.tools["LexicalDensityTool"](utterance)
print(result)

Use lexical density as a signal to adjust agent tone or escalate queries to human operators.

Batch Score Documents and Track Trends

from pathlib import Path
from swarmauri_tool_lexicaldensity import LexicalDensityTool

lexical_density = LexicalDensityTool()
corpus_dir = Path("reports/")

scores = []
for doc in corpus_dir.glob("*.txt"):
    text = doc.read_text(encoding="utf-8")
    scores.append((doc.name, lexical_density(text)["lexical_density"]))

for name, score in sorted(scores, key=lambda item: item[1], reverse=True):
    print(f"{name}: {score:.2f}% lexical words")

Monitor writing complexity across a corpus of articles or support responses.

Troubleshooting

• LookupError for NLTK resources – Ensure punkt_tab and averaged_perceptron_tagger_eng are downloaded prior to calling the tool (see nltk.download(...)).
• Low density on short texts – Very short messages yield coarse percentages. Aggregate multiple utterances or relax thresholds for brief content.
• Non-English text – POS tagging models target English. Swap in language-specific models before using the tool with multilingual corpora.

License

swarmauri_tool_lexicaldensity is released under the Apache 2.0 License. See LICENSE for full details.