The United States Secretary of State Speech Dataset (USSOS)

The US Secretary of State Speech Dataset (USSOS Dataset) is a corpus of U.S. Secretary of State public communication from Madeline Albright (January 1997) through the present, with Rubio-era speeches updated automatically each week. Public communication that are featured in the dataset includes major addresses, public statements, interviews, and press conferences, among others.

The dataset is at the speech-level and captures the location of the remarks, the date, the type of speech, the URL, the raw scraped text, the full cleaned text, a column that captures only the remarks of the Secretary of State, as well as a suite of additional variables.

Download

The full corpus is available via four channels:

GitHub Release (canonical)

# Download the latest release using the gh CLI
gh release download --pattern "corpus_v*.csv" --dir .

Or browse Releases on the right sidebar to download a specific version.

Python package loader (recommended)

pip install sos-speeches

import sos_speeches

# Download and cache
csv_path = sos_speeches.load()

# Load directly as a Pandas DataFrame
df = sos_speeches.load(as_dataframe=True)
print(df.shape)  # e.g. (17701, 19)

R package (recommended)

The sosspeech R package provides a convenient way to download and load the Secretary of State speech corpus directly into R. The package includes compressed data files, so you do not need to download the CSV separately.

Install the package directly from GitHub:

install.packages("remotes")
remotes::install_github(
  "revolins/sos-corpus",
  subdir = "r-package"
)

library(sosspeech)

# Load the full corpus
speeches <- sos_load()

# Load specific years (optional)
speeches_2020 <- sos_load(years = 2020)

# Inspect the data
print(dim(speeches))

Hugging Face Hub

# Using the datasets library
from datasets import load_dataset
ds = load_dataset("JayRevolinsky/sos-secretary-speeches")

# Or with a direct hf_hub_download
from huggingface_hub import hf_hub_download
csv_path = hf_hub_download(
    repo_id="JayRevolinsky/sos-secretary-speeches",
    filename="corpus.csv",
    repo_type="dataset",
)

Dataset columns

Each row is one public communication by a sitting Secretary of State. Every release has 19 columns: 11 core columns and 8 derived columns added by the enrichment stage. The table follows the grouping in the paper (Table A1); in the CSV itself, Canonical URL is the last column.

The five list-valued columns (Speakers Present, both People Mentioned columns, and both Countries columns) share one convention: entries are joined by "; ", appear in first-appearance order, are deduplicated case-insensitively, and are empty strings when nothing was found.

Live update pipeline

Every Monday at 06:00 UTC, a GitHub Actions workflow:

1. Downloads the latest corpus release
2. Runs scripts/rubio_backfill.py to discover new Rubio speeches on state.gov
3. If new rows are found, runs scripts/corpus_rubio_update.py to merge, clean, and verify
4. Publishes a new versioned release only if the verifier passes all checks

The workflow can also be triggered manually via Actions → Rubio Corpus Update → Run workflow.

If a release reached GitHub but the Hugging Face upload failed, run Actions → Publish to Hugging Face → Run workflow. It re-publishes an existing release (the latest by default) without scraping.

The sos-speeches Python package is released separately from the data: it always fetches the latest corpus release, so it only needs a new version when the loader itself changes. To release it, bump version in pyproject.toml and __version__ in sos_speeches/__init__.py, then push a matching tag (e.g. git tag py-v0.1.1 && git push origin py-v0.1.1). The Publish to PyPI workflow builds and uploads it via PyPI Trusted Publishing.

The PyPI package, R package and Hugging Face dataset badges above come from check workflows that run every Monday at 12:00 UTC, after the update. Each one installs or downloads the data the way a user would and validates it: the PyPI check runs sos_speeches.load(), the R check runs remotes::install_github() + sos_load() + the package tests, and the Hugging Face check confirms the mirror matches the latest GitHub release row for row.

Verification

Each release passes 85 checks in scripts/corpus_verify_research_grade.py (A through Z):

python scripts/corpus_verify_research_grade.py --input corpus_v*.csv --max-gap-days 90

Manual Rubio update

# Pipeline dependencies (scraping, enrichment, verification, publishing).
# The offline STM / analysis stack is deliberately separate -- it pins
# numpy<2, which is incompatible with the parquet engine used to publish.
pip install -r scripts/requirements.txt
# For the topic-modeling and stance scripts only:
#   pip install -r scripts/requirements-analysis.txt

# Discover new speeches (deduplicates against existing corpus)
python scripts/rubio_backfill.py --enriched corpus_v*.csv --output rubio_new.csv

# Merge, clean, verify
python scripts/corpus_rubio_update.py \
    --corpus  corpus_v*.csv \
    --backfill rubio_new.csv \
    --output  corpus_updated.csv

Citation

If you use this corpus in research, please cite:

BibTeX:

@dataset{yoel2026sos,
  author={Yoel, Benjamin and Revolinsky, Jay},
  title={Introducing the United States Secretary of State Speech Corpus},
  year={2026},
  howpublished={\url{https://github.com/revolins/sos-corpus}},
  note={Under Review}
}

APA: Yoel B., Revolinsky, J. (2026).Introducing the United States Secretary of State Speech Corpus. Retrieved from https://github.com/revolins/sos-corpus

Note: The underlying speeches are works of the U.S. Government and are in the public domain. Please cite this curated corpus when reusing the dataset.

License

• Code (Python scripts, pipeline, package): MIT License
• Dataset (corpus CSVs and Parquet mirrors): CC0 1.0 Universal — public domain dedication

The underlying content of the corpus (speeches and remarks by U.S. Secretaries of State) are works of the United States Government and are in the public domain.

Contact

Benjamin Yoel - Postdoctoral Fellow, CDDRL, Stanford University (PhD, Michigan State University): benyoel@stanford.edu

Jay Revolinsky - Computer Science PhD Student, Michigan State University: revolins@msu.edu