topiclayers

From posts to topical multilayer networks — hardened and social-scientist-friendly.

Quickstart

# Requires Python 3.12
pip install topiclayers

# copy runnable examples (configs + toy data) into ./examples/
topiclayers init

topiclayers run examples/generic.yml

That's it. You get a GML network file in ./out/generic_toy/networks/.

What does it do?

1. Loads your posts (Twitter JSONL, CSV, Parquet).
2. Cleans text (removes URLs, @mentions, newlines).
3. Models topics using BERTopic with SentenceTransformer embeddings.
4. Labels topics with an LLM (local via Ollama by default) and propagates topic labels through retweet chains.
5. Builds networks: single-layer retweet, multilayer per-topic, and bipartite temporal-text networks.

Minimal config

Save this as my_config.yml:

input:
  format: csv_posts
  posts: my_data.csv

name: my_dataset

topic_model:
  embedder: all-MiniLM-L6-v2
  min_topic_size: 50

network:
  type: multilayer_repost

output_dir: ./out/my_dataset

Run it: topiclayers run my_config.yml

Checking progress

Long runs write a live status file to <output_dir>/run_status.json. Check the current stage, elapsed time, and ETA from another terminal without touching the running job:

topiclayers status out/my_dataset

Example output:

Stage 3/7: embed
Status: running
Run: my_dataset
Progress: 412000/1200000 (34%)
Stage elapsed: 18m 12s
Stage ETA: ~35m 20s
Total elapsed: 21m 05s

Stages are load → data → embed → topic_model → label → network → manifest. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in <output_dir>/stage_timings.json. While running, the terminal itself shows a live progress line (spinner, counter, ETA, elapsed time) that keeps ticking even in stages without a counter, plus a per-stage duration summary when the run finishes.

Topic labeling with an LLM (optional)

After clustering, the pipeline can ask a local LLM to write one short, readable label per topic (instead of raw keyword lists like -1_proclamation_plante_trending). Default is Ollama — free, local, nothing leaves your machine:

# one-time setup
pip install 'topiclayers[labeling]'
ollama pull qwen2.5:7b

# start the server in another terminal
ollama serve

Then just run the pipeline as usual (label_model is on by default). Any LiteLLM model string works:

topic_model:
  label_model: ollama/qwen2.5:7b      # default
  # label_model: gpt-4o-mini          # OpenAI (needs OPENAI_API_KEY)
  # label_model: anthropic/claude-3-haiku-20240307
  # label_model: ""                   # disable labeling

Custom / self-hosted OpenAI-compatible endpoints

Any server speaking the OpenAI protocol (Unsloth, llama.cpp, vLLM, LM Studio...) works via label_api_base:

topic_model:
  label_model: openai/unsloth/Qwen3.8-27B-GGUF   # note the openai/ prefix
  label_api_base: http://localhost:8888/v1

with the token in .env (key resolution order: LABEL_API_KEY → OPENAI_LIKE_API_KEY → OPENAI_API_KEY). You can also set LABEL_API_BASE in .env instead of the YAML. Find the exact model name your server exposes with curl http://localhost:8888/v1/models.

Labeling is non-fatal: if the model server is unreachable, topics keep their keyword labels and everything else proceeds normally.

Choosing an embedder

embedder is any SentenceTransformer model (or an OpenAI text-embedding-* model). The pipeline applies the model-specific input treatment automatically — E5 gets its "query: " prefix, Qwen3-Embedding its instruction format, the MiniLM/MPNet families plain text — and L2-normalises, because clustering uses cosine/UMAP. Override with embedder_prompt / embedder_normalize:

topic_model:
  embedder: Qwen/Qwen3-Embedding-0.6B   # strong multilingual; ~1.8x the ARI of MiniLM on noisy data
  # embedder: all-mpnet-base-v2         # English-only fallback
  # embedder: paraphrase-multilingual-MiniLM-L12-v2   # lightweight multilingual
  # embedder_prompt: ""                 # override the auto prefix
  # embedder_normalize: true
  # embedder_device: ""                 # "" = auto (MPS/CUDA if available)
  umap_n_neighbors: 15
  umap_n_components: 10                 # 10 worked better than 5 for Qwen3

Rule of thumb (from the benchmark in experiments/): the embedder matters more than any clustering knob. all-MiniLM-L6-v2 (the default) is fine for English toy data but collapses multilingual corpora; use a multilingual model (Qwen3-Embedding, multilingual MiniLM/mpnet) for real, mixed-language data.

Input formats

CSV optional columns: created_at, lang, interaction_type (original/repost/quote/reply), target_post_id, mentions (semicolon-separated), extra_* passthrough columns.

What if something fails?

Output files

For a dataset named <name> (e.g. my_dataset), output goes to <output_dir>/:

<output_dir>/
├── run_manifest.json              # Reproducibility metadata
├── run_status.json                # Live stage/ETA tracking (while running)
├── stage_timings.json             # Per-stage durations for ETA prediction
└── networks/
    ├── <name>_retweet.gml                  # Single-layer retweet network
    ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
    ├── <name>_ttt.gml                      # Temporal-text bipartite network
    └── projected/
        └── <name>__prj_<topic>.gml         # Per-topic projected networks

Reusable artifacts (processed data, embeddings, the BERTopic model) go to the cache, not the output dir. By default the cache lives in a cache/ folder next to the input data file, so it persists across runs regardless of output_dir and can be shared between configs (files are prefixed with the dataset name). Override the location with cache_dir: in the config:

<cache_dir>/
├── data/
│   ├── tweets_<name>.pkl/.csv          # Full tweet table
│   ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
│   └── manifest_<name>.json            # Cache validity key
└── tm/<embedder_name>/
    ├── embeddings_<name>.pkl           # Document embeddings
    ├── topics_<name>.csv               # Topic table (with LLM labels)
    └── model_<name>/                   # BERTopic safetensors dir

Requirements

• Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
• Optional: Ollama + topiclayers[labeling] extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)