The Lexos Python Library

The Lexos Python library reproduces and extends most of the text analysis tools in the Lexos web app. Lexos is designed to implement many common text analysis procedures and create workflows in a way that saves the user having to re-invent the wheel or figure out how to combine multiple Python packages to achieve a given result. It is intended to be used as a library in other projects to build backend functions for applications, but it can be used in standalone scripts or in Jupyter notebooks. As with the original web app, it is designed to be accessible to entry-level users whilst offering power functionality for students and researchers, particularly in the Humanities. It is also designed to be as language-agnostic as possible so that it can be used for a wide variety of historical and under-resourced languages. Users can leverage existing language models or train their own.

📖 Documentation

A full discussion of the use of the API can be found on the website Documentation website.

⭐️ Major Features

• Loads texts from a variety of sources into a common data structure.
• Manages a corpus of texts and generates stastics about the corpus.
• Performs text pre-processing ("scrubbing") and splitting ("cutting").
• Performs tokenization and trains language models using spaCy.
• Creates assorted visualizations of term/ngram vectors.
• Performs hierarchical and kmeans clustering with a variety of visualizations for text comparisons..
• Performs classification using spaCy, scikit-learn, and custom architectures.
• Generates topic models and topic model visualizations using MALLET and DFR Browser 2.

And more!

What's New in v0.2.1

• Smoother spaCy model installation process.
• A refactored mallet module now provides access to both Java and a pyrmallet backends. This allows users to install the Rust version of MALLET with pip install pyrmallet, rather than the more complicated procedure of installing downloading and installing MALLET and JDK.

What's New in v0.2.0

• Performance optimizations and bug fixes througout.
• A simplified public API.
• Better HTML and XML tag handling in Scrubber.
• New and improved clustering visualizations.
• Improved plotting functions in Bootstrap Consensus clustering.
• Silhouette score analysis for KMeans clustering.
• More advanced topic modelling features, including termite plots, customisable labels, and LLM-generated labels with local or cloud-based models.
• A new Classification module that performs text classification using a variety of architectures.
• A new Structural Stylometry module that allows you to assess the significance of punctuation and whitespace patterns in your documentation.
• Language model training features.
• Improved documentation and tutorials for new users (now in a separate lexos-docs repository).

⏳ Installation

pip install lexos

To update to the latest version, use

pip install -U lexos

Or, if you are using uv:

uv add lexos

Some Lexos functions use spaCy language models to obtain language-specific information about texts. To download spaCy models, run the following command:

lexos-download-spacy-models xx_sent_ud_sm en_core_web_sm

or

uv run lexos-download-spacy-models xx_sent_ud_sm en_core_web_sm

You can install multiple models by separating their names with spaces. If you do not run this command, Lexos will install the default xx_sent_ud_sm model the first time you attempt to use a feature that requires a spaCy language model.

If you are working in another language or need a larger language model, you can download instructions for additional models from the spaCy models page. Use the following command:

python -m spacy download en_core_web_md # Replace with the name of your model

Add uv run to the beginning of the command if you are using a uv environment.

If you need to install a model from within Python, you can use Lexos' helper funtion:

from lexos.util import download_spacy_model

download_spacy_model("fr_core_web_sm")

🚦Project Status

The Lexos API is currently in beta. Most of the core functionality of the Lexos web app, along with new features, has been implemented and documented. As of September 2026, the API is considered feature complete and stable for general use, but some rough edges remain. Feedback is welcome.

I will continue to fix bugs and improve the documentation as issues arise, but no major new features are planned at this time. The beta release coincides with at a time when the landscape of digital tools is rapidly evolving and AI-assisted coding is becoming more prevalent. I am waiting to see whether there is significant adoption of the Lexos library before investing more time in developing new features. If you like Lexos, you can help by requesting new features in the GitHub issues (labelled as "enhancement") or contributing them yourself.

💝 Contribute

• If are looking for help using Lexos, please post you question on the GitHub Discussions board.
• Bug reports and feature requests: Please use GitHub issues.
• For other types of contributions see the Documentation website.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📝 Citation Information

Kleinman, S., (2026). Lexos. v0.2.1 https://github.com/scottkleinman/lexos. 10.5281/zenodo.18112379.