telescope-llm-text-detection 0.1.0

Improving Zero-Shot Detection of LLM-Generated Content With Token Repetition

Telescope: Improving Zero-Shot Detection of LLM-Generated Content With Token Repetition

A novel approach to detecting AI-generated text through token repetition analysis

Paper (Coming Soon on Arxiv) • [Datasets](https://huggingface.co/datasets/Aanimated/telescope_datasets) • [Raw Results](https://huggingface.co/datasets/Aanimated/telescope_experiment_results)

Overview

Telescope introduces a new metric for detecting LLM-generated content in zero-shot settings by analyzing token repetition patterns. This repository contains the complete implementation and links to the datasets and experimental results from our research.

Installation

Prerequisites

• Python 3.10 or higher
• Miniconda or Anaconda
• Hugging Face account (for accessing certain models)

Setup Instructions

1. Install Miniconda
Follow the official Miniconda installation guide for your operating system.

2. Install Git and Git LFS
Follow the official guides to install Git and Git LFS for your operating system.

3. Create and activate the environment

conda env create -f telescope_env.yml
conda init
conda activate telescope

4. Install the package You can install the telescope package directly from PyPI:

pip install telescope_llm_text_detection

Alternatively, to install it locally in development/editable mode:

pip install -e .

All required packages should now be installed. If you encounter any missing dependencies, issues, or other hiccups during installation or usage, please open an issue.

Hugging Face Authentication

Some models used in this work require authentication. To set up your Hugging Face token:

1. Generate a token at huggingface.co/settings/tokens

2. Set the HF_TOKEN environment variable (replace XXXXXXXX with your actual token):

Unix/Linux/macOS:

export HF_TOKEN=XXXXXXXX

Windows (PowerShell):

$env:HF_TOKEN="XXXXXXXX"

To make this permanent, add the export line to your ~/.bashrc, ~/.zshrc, or equivalent shell profile like so:

echo "export HF_TOKEN=XXXXXXXX" >> ~/.bashrc

Datasets and Experiment Results

Download the datasets and experiment results using the following commands. Please note that they are fairly large and will consume approximately 38 GB of storage.

# Download experiment results
git lfs clone https://huggingface.co/datasets/Aanimated/telescope_experiment_results experiment_results

# Download datasets
git lfs clone https://huggingface.co/datasets/Aanimated/telescope_datasets datasets

Project Structure

telescope/                      # Repository root
├── llm_text_detectors/          # Core package folder (packaged as telescope_llm_text_detection)
│   ├── __init__.py             # Exports Telescope, utils
│   ├── llm_text_detectors.py   # Telescope detector class
│   └── utils.py                # Utility functions (model loading, auth, shared helpers)
├── ablations/                  # Ablation studies
│   ├── per_token/              # Per-token analysis metrics
│   ├── sampling/               # LLM sampling utilities
│   ├── sequence_modeling/      # Sequence modeling dataset tools
│   ├── single_sample/          # Single sample analysis
│   ├── single_token_distribution/  # Token distribution analysis
│   └── training/               # Training utilities and logging
├── scripts/                    # Analysis and experiment scripts
│   ├── generate_experiment_results.py
│   ├── compute_roc_and_f1score_from_metrics.py
│   ├── generate_*.py           # Various plotting/analysis scripts
│   └── ...
├── ghostbusters_dataset_creation/ # Dataset conversion and creation tools
├── datasets/                   # Dataset files
├── experiment_results/         # Pre-computed experiment results
├── config.yaml                 # Global configuration (model/dataset/metric names)
├── pyproject.toml              # Package configuration
└── telescope_env.yml           # Conda environment specification

Key Concepts and Definitions

Metrics

A metric is a numerical value computed from a reference model's outputs. Examples include:

• Telescope Perplexity
• Binoculars Score
• Perplexity
• DetectLLM LRR

Additional experimental metrics are implemented in llm_text_detectors/llm_text_detectors.py. Effective metrics show correlation with whether text was LLM-generated.

Experiment Results

Experiment results are CSV files containing data from running detection algorithms on specific datasets with specific reference models. Each result includes:

• Original text samples
• Ground truth labels (human vs. LLM-generated)
• Computed metric values

Browse the experiment_results directory to examine the data format.

Codenames vs. Display Names

To maintain file naming conventions while preserving publication-ready formatting:

• Codenames: lowercase with underscores (e.g., telescope_perplexity)
• Display Names: formatted for publication (e.g., "Telescope Perplexity")

Performer and Reference Models

This is a concept from the Binoculars paper:

• Performer Model: Computes both perplexity and cross-perplexity
• Observer Model: Only needed for cross-perplexity computation

For single-model techniques, the reference model defaults to "performer model" by convention.

See the Binoculars paper for detailed explanations.

Configuration

The config.yaml file stores global variables including:

• Codenames -> display name mappings for models, datasets, and metrics
• Plot colors

Usage

In lieu of having command line arguments for every script, this codebase instead uses global variables at the top of each runnable script where you can set which arguments you want for things like which datasets or metrics to use. The reason for this is because specifying all of the metrics, datasets, models, etc takes up a lot of space and is annoying to keep track of in the runtime arguments of a script, so we just have all of in an easy place to see and edit.

Running Experiments

If you would like to Generate new experiment results by running detection algorithms on datasets:

python scripts/generate_experiment_results.py

This script:

• Runs reference models on text samples
• Calculates metrics (Telescope Perplexity, Binoculars Score, etc.)
• Saves raw results to CSV files

[!IMPORTANT] Running experiments requires significant computational resources and time. Pre-computed results are provided to facilitate analysis without rerunning experiments.

Analyzing Experiments

Analyze existing experiment results to generate:

• ROC curves
• F1-scores
• Threshold transfer characteristics
• Data visualizations

Available analysis scripts (all located in scripts/):

• scripts/compute_roc_and_f1score_from_metrics.py
• scripts/generate_adversarial_perturbation_plot.py
• scripts/generate_calibration_charts.py
• scripts/generate_error_independence_table.py
• scripts/generate_length_vs_score_plot.py
• scripts/generate_misclassification_plots.py
• scripts/generate_score_distribution_plots.py
• scripts/generate_threshold_transfer_plot.py

Experiment results contain only raw metric values. Analysis scripts compute performance metrics like AUROC, precision, and recall.

Additional Metrics

Various additional metrics are implemented in llm_text_detectors/llm_text_detectors.py from our initial large-scale testing phase. While none proved as promising as Telescope Perplexity in our experiments, they remain available for further research and analysis.

Citation

If you use Telescope in your research, please cite our paper:

@inproceedings{telescope2026,
  title={Telescope: Improving Zero-Shot Detection of {LLM} Generated Content By Measuring Token Repetition Probability},
  author={Christopher Nassif and Josh Cooper},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year={2026}
}

License

CC BY-NC-SA 4.0

Contact

For questions or collaboration opportunities, please open an issue or contact [contact information].