Python Package: impresso-pipelines
Overview
This repository contains a Python package designed for modular and efficient text processing workflows. Currently, it includes the following subpackages:
- Language Identification Pipeline: Identifies the language of input text and returns a probability score.
- OCR QA Pipeline: Assesses the quality of OCR text by estimating the proportion of recognized vocabulary items (0–1), using efficient language-specific Bloom filters.
- LDA Topic Modeling Pipeline: Soft clustering of input texts using LDA-based topic modeling.
- News Agencies Pipeline: Extracts and ranks news agency entities from text, providing relevance scores and optional links to Wikidata.
- Advertisement Classifier: Identifies advertisements in historical newspaper content using a fine-tuned XLM-RoBERTa model with rule-based features.
- Lucene/Solr normalization Pipeline: Replicates Solr's language-specific text normalization to clarify how input text is tokenized and indexed in impresso.
Installation
Quick Install (with uv - recommended)
uv is an extremely fast Python package installer (10-100x faster than pip):
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install the package with all dependencies
uv pip install "impresso-pipelines[all]"
Standard Install (with pip)
To install the full package with all submodules:
pip install "impresso-pipelines[all]"
The [all] extra installs all dependencies required for each component.
Install Individual Modules
To install individual modules without unnecessary dependencies, use:
pip install "impresso-pipelines[langident]" # Language Identification
pip install "impresso-pipelines[ocrqa]" # OCR QA
pip install "impresso-pipelines[ldatopics]" # LDA Topics
pip install "impresso-pipelines[newsagencies]" # News Agencies
pip install "impresso-pipelines[adclassifier]" # Advertisement Classifier
pip install "impresso-pipelines[solrnormalization]" # Solr text normalization
Development Setup
For contributors, we support both uv (faster) and Poetry:
# Clone the repository
git clone https://github.com/impresso/impresso-pipelines.git
cd impresso-pipelines
# Option 1: Using uv (recommended - 3-6x faster)
uv sync --extra all --extra dev
# Option 2: Using Poetry
poetry install --all-extras --with dev
# Or use Make (auto-detects uv or Poetry)
make install-dev
See UV_MIGRATION.md for more details on using uv.
Usage
Each pipeline is instantiated from a corresponding class.
from impresso_pipelines.langident import LangIdentPipeline
from impresso_pipelines.ocrqa import OCRQAPipeline
from impresso_pipelines.ldatopics import LDATopicsPipeline
from impresso_pipelines.newsagencies import NewsAgenciesPipeline
from impresso_pipelines.adclassifier import AdClassifierPipeline
from impresso_pipelines.solrnormalization import SolrNormalizationPipeline
Pipeline Examples
For usage examples, refer to the individual README files:
- Langident Pipeline
- OCR QA Pipeline
- LDA Topics Pipeline
- News Agencies Pipeline
- Advertisement Classifier
- Solr normalization Pipeline
See also the interactive notebooks for further examples:
- langident_pipeline_demo.ipynb
- ocrqa_pipeline_demo.ipynb
- ldatopics_pipeline_demo.ipynb
- newsagencies_pipeline_demo.ipynb
- solrnormalization_pipeline_demo.ipynb.
Future Plans
Additional functionality will be added to extend use cases and support further processing tasks.
Local Development
For contributors and developers who want to test locally before pushing to GitHub:
Quick Start
# Clone and install
git clone https://github.com/impresso/impresso-pipelines.git
cd impresso-pipelines
# Option 1: Poetry (recommended for full development)
make install-dev
# Option 2: Pip editable mode (faster for testing changes)
make install-editable-dev
# Run tests
make test
# Run all QA checks (mimics CI)
make qa
Available Commands
make help # Show all available commands
make install # Install package with all extras
make install-dev # Install with dev dependencies
make test # Run tests (skipping JVM tests)
make test-all # Run all tests including JVM tests
make test-ocrqa # Run only OCRQA tests
make test-cov # Run tests with coverage report
make lint # Run linting checks
make format # Format code with black
make type-check # Run type checking
make qa # Run all QA checks
make clean # Remove build artifacts
For detailed development instructions, see CONTRIBUTING.md.
About Impresso
Impresso project
Impresso - Media Monitoring of the Past is an interdisciplinary research project that aims to develop and consolidate tools for processing and exploring large collections of media archives across modalities, time, languages and national borders. The first project (2017-2021) was funded by the Swiss National Science Foundation under grant No. CRSII5_173719 and the second project (2023-2027) by the SNSF under grant No. CRSII5_213585 and the Luxembourg National Research Fund under grant No. 17498891.
Copyright
Copyright (C) 2025 The Impresso team.
License
This program is provided as open source under the GNU Affero General Public License v3 or later.
1 maintainer
Metadata
1 maintainer
Release files for impresso-pipelines 0.6.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| impresso_pipelines-0.6.1.2.tar.gz | 90.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| impresso_pipelines-0.6.1.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 194.3 kB
Release files / impresso_pipelines-0.6.1.2.tar.gz
| Download URL | impresso_pipelines-0.6.1.2.tar.gz |
|---|---|
| Size | 90.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
828a209e5d1e0feba3e2c5d6b50b624e4ae4f9efd6f6cd4cc9954ff8d244b892
|
|
BLAKE2b-256 checksum How to use checksums |
23bdb281180a20db86fb83cc3bfc0cdfe3a05d25f112d9f40aa76c89a33825b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / impresso_pipelines-0.6.1.2-py3-none-any.whl
| Download URL | impresso_pipelines-0.6.1.2-py3-none-any.whl |
|---|---|
| Size | 103.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d0678c1075ca0b5e756379d96a11f9d34e6d52bc50c8635946d7ab71fd7dcbb0
|
|
BLAKE2b-256 checksum How to use checksums |
83aadabf0809340efe1390f6df5d8d36fb250359322203ab1c53deb2d8a4ed50
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
