tau-eval 0.1.0

Text Anonymization Evaluation Library

𝜏 Tau-Eval: A Unified Evaluation Framework for Useful and Private Text Anonymization

Tau-Eval is a user-friendly, modular, and customizable Python library designed to benchmark and evaluate text anonymization algorithms. It enables granular analysis of anonymization impacts from both privacy and utility perspectives. Tau-Eval seamlessly integrates with LiteLLM and 🤗 Hugging Face to support a wide range of datasets, models, and evaluation metrics.

Installation

From PyPI

Install Tau-Eval via pip:

pip install tau-eval

From source

To install from source:

1. Clone this repository on your own path:

git clone https://github.com/gabrielloiseau/tau-eval.git
cd tau-eval

2. Create an environment with your own preferred package manager. We used python 3.9 and dependencies listed in pyproject.toml. If you use conda, you can just run the following commands from the root of the project:

conda create --name taueval python=3.9         # create the environment
conda activate taueval                         # activate the environment
pip install --user -r pyproject.toml           # install the required packages

Quickstart

Tau-Eval is designed for flexibility. With just a few lines of code, you can set up and run evaluations.

1. Define Your Anonymization Model

Create a custom anonymization model by extending the Anonymizer interface:

from tau_eval.models import Anonymizer

class TestModel(Anonymizer):
    def __init__(self):
        self.name = "Test Model"

    def anonymize(self, text: str) -> str:
        # Implement anonymization logic
        return text

    def anonymize_batch(self, texts: list[str]) -> list[str]:
        # Batch processing
        return texts

Or use prebuilt models from tau_eval.models.

2. Configure Evaluation Metrics

Use built-in metrics from tau_eval.metrics or define your own following this signature:

Callable[[str | list[str], str | list[str]], dict]

This allows complete control over what and how you evaluate.

3. Instantiate Tasks

Tasks can be created using prebuilt options in tau_eval.tasks, or customized using CustomTask. Tau-Eval also supports tasksource for dataset integration.

4. Configure and Run Your Experiment

Define an experiment configuration:

from tau_eval.config import ExperimentConfig

config = ExperimentConfig(
    exp_name="test-experiment",
    classifier_name="medicalai/ClinicalBERT",
    train_task_models=True,
    train_with_generations=False,
)

Run the experiment:

from tau_eval.experiment import Experiment

Experiment(
    models=[m1, m2, m3, m4, m5],
    metrics=["bertscore"],
    tasks=[mednli, pii],
    config=config
).run()

5. Visualize Results

Tau-Eval includes built-in visualization tools to compare model anonymization strategies and evaluation results. You can find them with tau_eval.visualization.

Tutorials

You can find our tutorials to learn how to better leverage Tau-Eval in the examples/ folder.

There you can also find a set of interesting case studies using real-world datasets! :chart_with_upwards_trend:

Contributors

• Gabriel Loiseau, Hornetsecurty, Inria Lille

Citation

If you use 𝜏 Tau-Eval in your work, please cite our paper as follows: