Synthetic Surveys

A CLI for exploring policy and program questions with synthetic survey respondents. Generate a ten-question survey, review or edit it, simulate a panel, and save a Markdown report with results and an interpretation.

These are model-generated responses, not measured public opinion. Use them to explore questions and hypotheses before research with people.

Install

Synthetic Surveys is available on PyPI. With uv installed, download or update the package and launch it in one command:

uvx --upgrade --from synthetic-surveys synthetic-surveys

For a persistent command available in your terminal:

uv tool install --upgrade synthetic-surveys
synthetic-surveys

Python 3.10 or later is required. Alternatively, use pip in a virtual environment:

python -m venv .venv
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1
python -m pip install --upgrade synthetic-surveys
synthetic-surveys

Optional native executable previews are built separately for Windows, macOS, and Linux. Publishing a Python wheel does not rebuild those previews. See executable builds and installer preparation.

First run

The default provider is OpenRouter. One OPENROUTER_API_KEY covers survey generation, respondent simulation, interpretation, and Pro-mode embeddings. Missing credentials trigger a connection menu with browser authorization and hidden API-key entry. The CLI validates OpenRouter/OpenAI keys using an account/models request before offering to save them.

• API usage is billed to your provider account. Start with two or three respondents; Pro mode makes additional requests. Key validation does not generate model tokens and does not guarantee access to every model.
• Saving a key is optional. Credentials use macOS Keychain or the Linux desktop Secret Service through Python keyring. If no secure store is available, the app keeps the key only for the current process. Non-secret preferences remain in ~/.synthetic_surveys/settings.json.
• Environment variables override saved settings. A bad exported key must be changed or unset; changing the saved key alone will not override it on the next launch.
• On first use, choose browser login (default), paste an API key, or authorize on another device and paste a one-time code. Browser login opens OpenRouter and returns authorization to a temporary localhost listener. Ctrl+C or a timeout returns to setup options; no separate OpenRouter CLI is required.

To replace a key or select a provider:

synthetic-surveys configure
synthetic-surveys configure --provider openai

Direct OpenAI configuration sets OpenAI as both the LLM and embedding provider. Remove any old OpenRouter model overrides when switching providers.

Your first report

synthetic-surveys run --mode normal --sample-size 3

1. Type or paste your question into the text box. Enter adds a new line; use the arrow keys to edit. Press Tab, then Enter to continue; an arrow appears in the bracket when the continue control is selected.
2. Optionally add survey-design guidance and supporting documents.
3. Choose a sampling profile, review the draft cost, and approve generation. You can change the draft model or question before paying.
4. Review the survey, its respondent background, and any private framing concerns. Accept it, edit the background or questions for free, or approve a paid revision.
5. Choose the respondent model and count, then review an itemized estimate including the final interpretation and any Pro work. Change settings or cancel before starting.
6. Read the results, reported charges, and saved report path.

The question and survey-design context boxes preserve pasted paragraphs and blank lines. A character count shows the limit; if you exceed it, you can edit your text without starting over. Shift+Tab moves back to the text box. Ctrl+C exits. Redirected input uses ordinary line prompts; scripts can pass a multiline question with --question.

Reports go into ~/synthetic-surveys/reports/YYYY-MM-DD/, with a readable survey-title-and-timestamp filename. The title comes from the generated survey, so saving does not require another model request. Choose another reports root with --output-dir /path/to/reports (--dev-dir remains an alias); date folders are created beneath that root. The CLI is interactive even when you supply a question and sample size. generate-survey and sample-personas are separate utility commands.

synthetic-surveys --help
synthetic-surveys generate-survey "Should the city pedestrianize Main Street?"
synthetic-surveys sample-personas 3

Normal mode directly selects Likert answers. Pro mode generates free-text responses and maps them through embeddings to Likert labels; it needs additional requests and fails clearly if embedding requests fail.

Simulation size

Choose the number of synthetic respondents based on your budget and the variety of profiles you want to explore. Larger runs can reduce random fluctuations but do not guarantee more realistic answers. There is no validated sample-size recommendation or real-world margin of error for this tool; target population headcount does not determine the required simulation size. A run needs at least one respondent.

Respondent context

The survey draft includes a neutral background paragraph for respondents to read. The original question and private design context guide survey generation; respondent models receive the approved background, survey items, persona, and supporting materials. Private framing notes remain available for your review. Supporting materials are passed through without being replaced by the background. Original research inputs and private context can still appear in your saved report metadata.

Cost transparency

The interactive survey flow reads OpenRouter's public model and embedding catalogs without sending your survey text. Estimates use current list prices, approximately three UTF-8 bytes per input token (including system prompts and JSON schemas), and explicit output allowances: 2,200 tokens for a draft, 800/1,000 per Normal/Pro respondent, 1,600 per anchor-generation call, and 400 per interpretation. The review shows approximate token counts, dated per-million-token rates, and an estimated cost for each stage. Allowances are planning assumptions, not measured usage averages. Expanded estimates, including configured output limits, are retained in report metadata; they are not spending caps.

Pro adds one anchor-generation request per survey item, then six embedding requests per item per respondent. The estimate includes repeated anchors and answers (40 tokens per anchor statement and 80 per answer). Its additional work is itemized before you run the simulation. There is no fixed or empirically validated Pro/Normal cost multiplier.

Actual charges can differ because of reasoning, output length, caching, routing, retries, or price changes. Native image/file processing, direct-provider prices, unknown models and embedding fallback costs may be unavailable; incomplete estimates are labeled explicitly. Credit-purchase fees and taxes are excluded. No price lookup makes an inference request.

OpenRouter response-reported charges and token counts are shown after the session and saved in completed reports alongside approved estimates. Missing costs are marked as incomplete, including embeddings if their response omits cost. Failed requests without usage data may still incur charges: your provider's billing activity remains authoritative. Canceling stops future steps, not charges already incurred. These checkpoints apply to the interactive CLI; utility commands and the desktop preview do not use this review flow.

Weighting and IPUMS

Profile	Included with the app?	Setup
General demographic sampling (menu label: No Demographics Weights)	Built-in sampling proportions	None
IPUMS Population Sample	No; individual-record extract	Your own IPUMS USA access and a one-time download

The profiles affect which personas are sampled, not post-hoc weighting of completed responses. Even the general profile has built-in demographic sampling proportions.

Select IPUMS Population Sample, then enter exactly one state abbreviation or DC (for example, MA or CA). Multiple states are not accepted. The app checks for a valid local extract before offering a download or a return to the profile menu. You can also prepare it explicitly:

synthetic-surveys configure --provider ipums
synthetic-surveys download-ipums --state MA

Use your own IPUMS USA account and API key. Existing IPUMS_API_KEY credentials take precedence; otherwise the app prompts with hidden entry and offers the existing optional secure-keyring storage. The default sample is us2024a; override it with --ipums-sample or SYNTHETIC_SURVEYS_IPUMS_SAMPLE. Preparation may take several minutes. Valid cached data needs no API key.

Extracts are stored under ~/.synthetic_surveys/data/ipums/<sample>/<STATE>.csv.gz (or beneath SYNTHETIC_SURVEYS_HOME when configured). A companion JSON file records state, sample ID, extract number, download time, and checksum. Reuse verifies those details, required columns, state records, and usable PERWT values. Invalid or incomplete caches trigger a new download; failed downloads preserve existing data. State/sample combinations have separate caches.

Personas are drawn with replacement from the chosen state's records using PERWT. Each persona is also told its state of residence. Small synthetic samples can differ from the state's population proportions, and matching demographics does not validate the simulated opinions. State, sample, and extract provenance appear in reports.

The old download-ipums-ma command remains an alias defaulting to MA and writes the new cache layout. Programmatic sampling and the desktop runner can still read a compatible ~/.synthetic_surveys/data/ipums/massachusetts.csv.gz when no sample override is requested and no valid modern cache exists. That legacy file is recorded as having unknown sample/extract provenance. The interactive CLI downloads a verified extract instead of assuming a legacy file matches the requested sample.

Do not commit or redistribute IPUMS microdata. Redistribution requires IPUMS permission outside the stated publication exception. Follow IPUMS USA terms, and retain the extract's citation for reports/publications. The downloader specification is in synthetic_surveys/ipums_api.py; no IPUMS data is bundled.

Documents and limits

PDF, JPG, PNG, WebP, and plain-text attachments work without installing a local OCR engine. PDFs and images go to compatible model providers; text files and HTML are extracted locally. Convert Word documents to PDF first. The app checks visual-model support before paid requests and asks you to change incompatible models. Exact model IDs are required when a routing alias cannot be verified.

Models with native PDF support receive PDFs directly. For image-capable models without PDF support, the app renders every page locally and sends the pages as images in order. The renderer installs automatically with the app; no OCR engine is needed. Conversion supports up to 20 pages per PDF at 144 DPI, capped at 2,048 pixels on the longest edge. Longer or unreadable PDFs are rejected before generation; pages are never silently omitted. Rendered pages are reused in memory across calls, but image input remains billable on each request. Small text may be less legible after rendering; native PDF input is preferred.

Visual capability checks use the public OpenRouter model catalog (also as a conservative capability reference for matching direct OpenAI model IDs). If support cannot be verified, visual requests stop before inference. Provider-specific size, context, and access limits can still reject a request. See OpenRouter PDF inputs.

Limits: four attachments, 4 MB per file, 12 MB total, a 4,000-character question, and 2,000 characters of regeneration feedback. Questions, document content, and generated personas are sent to the selected providers. Reports contain research inputs and run metadata, so choose their storage location accordingly.

Environment variables

All are optional unless needed by the selected feature. Environment values take precedence over saved settings.

Variable	Purpose/default
OPENROUTER_API_KEY	Default LLM and embedding credential
OPENAI_API_KEY	Direct OpenAI credential; optional embedding fallback
GEMINI_API_KEY	Only for direct Gemini embeddings
IPUMS_API_KEY	Only for downloading an IPUMS extract
SYNTHETIC_SURVEYS_HOME	App settings/data/cache root; defaults to ~/.synthetic_surveys
SYNTHETIC_SURVEYS_LLM_PROVIDER	openrouter (default), openai, or offline stub
SYNTHETIC_SURVEYS_EMBEDDING_PROVIDER	openrouter, openai, or gemini; CLI defaults to the selected LLM provider
SYNTHETIC_SURVEYS_MODEL	Override all LLM stages
SYNTHETIC_SURVEYS_SURVEY_MODEL	Override survey generation model
SYNTHETIC_SURVEYS_SIMULATION_MODEL	Override simulation model
SYNTHETIC_SURVEYS_SUMMARY_MODEL	Override interpretation model
SYNTHETIC_SURVEYS_OPENROUTER_EMBEDDING_MODEL	openai/text-embedding-3-small
SYNTHETIC_SURVEYS_OPENAI_EMBEDDING_MODEL	text-embedding-3-small
SYNTHETIC_SURVEYS_EMBEDDING_MODEL	Direct Gemini embedding model
SYNTHETIC_SURVEYS_IPUMS_SAMPLE	Download sample override; default us2024a
SYNTHETIC_SURVEYS_SSR_EPSILON	SSR bias offset; default 0.0
SYNTHETIC_SURVEYS_SSR_TEMPERATURE	SSR temperature; default 1.0

The checked-in defaults use openai/gpt-5.5 for OpenRouter survey/summary stages and openai/gpt-5.4-mini for simulation. Provider availability can change; use model overrides if necessary.

Local development

Ablation is a development diagnostic, available through the hidden --ablation CLI flag. It is absent from the interactive menus and desktop controls:

synthetic-surveys --mode normal --ablation

This adds a second run without demographic, state, or personality persona information and reports distribution similarity. Shared survey content and materials remain unchanged. The control always uses Normal mode; use a Normal primary run to avoid also changing the response method. When explicitly enabled, its additional costs are included in the estimate. Similarity measures profile influence, not agreement with real people.

uv sync --extra dev
cp .env.example .env
# Edit .env locally to add your key. Never commit it.

The application does not load .env itself. If you use direnv, create .envrc containing:

dotenv_if_exists .env
if [[ -d .venv/bin ]]; then
  PATH_add .venv/bin
fi

Then run direnv allow with direnv installed and hooked into your shell. Alternatively export the variables in your shell. Both files are gitignored.

uv run synthetic-surveys
uv run ruff check .
uv run ty check
uv run pytest -q

For an offline demo with placeholder responses, use uv run synthetic-surveys --provider stub --mode normal. This is not a real simulation. Live OpenAI tests require SYNTHETIC_SURVEYS_RUN_OPENAI_TESTS=1 and OPENAI_API_KEY.

See research foundations and audit findings, methodology, and release verification. See LICENSE for licensing terms.