Synthetic Surveys

A CLI for exploring policy and program questions with synthetic survey respondents. Generate a ten-question survey, review or edit it, simulate a panel, and save a Markdown report with results and an interpretation.

These are model-generated responses, not measured public opinion. Use them to explore questions and hypotheses before research with people.

Install

The macOS/Linux MVP will use a single-command installer powered by uv. The installer downloads Python 3.12 as needed, installs the Python package and dependencies, then starts the CLI. Public download URLs are not configured yet; see distribution preparation.

Self-contained downloads are built for Windows x64, Linux x64, and both Apple Silicon and Intel Macs. They include Python and dependencies. Extract the archive, then launch synthetic-surveys.exe on Windows or Start Synthetic Surveys.command on macOS. On Linux, run ./synthetic-surveys in a terminal. Preview builds are unsigned and may require operating-system approval. Downloads remain private until public distribution is arranged.

Each archive includes START-HERE.txt. See executable builds for build and verification instructions.

Python package alternative

Requires Python 3.10 or later. Once the public wheel is published, download it from the release location linked on the website, then install it in an isolated environment:

python -m venv .venv
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1
python -m pip install /path/to/synthetic_surveys-0.1.0-py3-none-any.whl
synthetic-surveys

Until a release is available, install from a source checkout with python -m pip install . inside the environment. Developers can use the workflow below. This guide does not assume a PyPI publication.

First run

The default provider is OpenRouter. One OPENROUTER_API_KEY covers survey generation, respondent simulation, interpretation, and Pro-mode embeddings. Missing credentials trigger a connection menu with browser authorization and hidden API-key entry. The CLI validates OpenRouter/OpenAI keys using an account/models request before offering to save them.

• API usage is billed to your provider account. Start with two or three respondents; Pro mode and ablation make additional requests. Key validation does not generate model tokens and does not guarantee access to every model.
• Saving a key is optional. Credentials use macOS Keychain or the Linux desktop Secret Service through Python keyring. If no secure store is available, the app keeps the key only for the current process. Non-secret preferences remain in ~/.synthetic_surveys/settings.json.
• Environment variables override saved settings. A bad exported key must be changed or unset; changing the saved key alone will not override it on the next launch.
• On first use, choose browser login (default), paste an API key, or authorize on another device and paste a one-time code. Browser login opens OpenRouter and returns authorization to a temporary localhost listener. Ctrl+C or a timeout returns to setup options; no separate OpenRouter CLI is required.

To replace a key or select a provider:

synthetic-surveys configure
synthetic-surveys configure --provider openai

Direct OpenAI configuration sets OpenAI as both the LLM and embedding provider. Remove any old OpenRouter model overrides when switching providers.

Your first report

synthetic-surveys run --mode normal --sample-size 3

1. Type or paste your question into the text box. Enter adds a new line; use the arrow keys to edit. Press Tab to select Continue, then Enter to submit (or click Continue).
2. Optionally add survey-design guidance and supporting documents.
3. Choose a sampling profile, review the draft cost, and approve generation. You can change the draft model or question before paying.
4. Review the generated survey; accept it, edit questions for free, or approve a paid revision.
5. Choose the respondent model and count, then review an itemized estimate including the final interpretation and any Pro/ablation work. Change settings or cancel before starting.
6. Read the results, reported charges, and saved report path.

The question and survey-design context boxes preserve pasted paragraphs and blank lines. A character count shows the limit; if you exceed it, you can edit your text without starting over. Shift+Tab moves back to the text box, and Cancel or Ctrl+C exits. Redirected input uses ordinary line prompts; scripts can pass a multiline question with --question.

Reports go into ~/synthetic-surveys/reports/YYYY-MM-DD/, with a readable survey-title-and-timestamp filename. The title comes from the generated survey, so saving does not require another model request. Choose another reports root with --output-dir /path/to/reports (--dev-dir remains an alias); date folders are created beneath that root. The CLI is interactive even when you supply a question and sample size. generate-survey and sample-personas are separate utility commands.

synthetic-surveys --help
synthetic-surveys generate-survey "Should the city pedestrianize Main Street?"
synthetic-surveys sample-personas 3

Normal mode directly selects Likert answers. Pro mode generates free-text responses and maps them through embeddings to Likert labels; it needs additional requests and fails clearly if embedding requests fail. --ablation adds a second control run without demographic/personality signals and reports distribution similarity.

Cost transparency

The interactive survey flow reads OpenRouter's public model and embedding catalogs without sending your survey text. Estimates use current list prices, approximately three UTF-8 bytes per input token (including system prompts and JSON schemas), and explicit output allowances: 2,200 tokens for a draft, 800/1,000 per Normal/Pro respondent, 1,600 per anchor-generation call, and 400 per interpretation. It also shows costs at configured output limits; these are not spending caps. Allowances are planning assumptions, not measured usage averages.

Pro adds one anchor-generation request per survey item, then six embedding requests per item per respondent. The estimate includes repeated anchors and answers (40 tokens per anchor statement and 80 per answer). Its displayed Pro/Normal ratio depends on your models and panel size, includes the remaining report work, and excludes the already-generated draft. There is no fixed or empirically validated multiplier.

Actual charges can differ because of reasoning, output length, caching, routing, retries, or price changes. Native image/file processing, direct-provider prices, unknown models and embedding fallback costs may be unavailable; incomplete estimates are labeled explicitly. Credit-purchase fees and taxes are excluded. No price lookup makes an inference request.

OpenRouter response-reported charges and token counts are shown after the session and saved in completed reports alongside approved estimates. Missing costs are marked as incomplete, including embeddings if their response omits cost. Failed requests without usage data may still incur charges: your provider's billing activity remains authoritative. Canceling stops future steps, not charges already incurred. These checkpoints apply to the interactive CLI; utility commands and the desktop preview do not use this review flow.

Weighting and IPUMS

Profile	Included with the app?	Setup
General demographic sampling (menu label: No Demographics Weights)	Built-in sampling proportions	None
Harm reduction	Built-in normalized overdose-rate weights	None
Massachusetts IPUMS	No; individual-record extract	Your own IPUMS USA access and a one-time download

The profiles affect which personas are sampled, not post-hoc weighting of completed responses. Even the general profile has built-in demographic sampling proportions. Harm-reduction values are an approximation, not a population-adjusted epidemiologic model; see methodology.

Selecting IPUMS without local data offers setup/download or a return to the profile menu. You can also prepare it explicitly:

synthetic-surveys configure --provider ipums
synthetic-surveys download-ipums-ma

Use your own IPUMS USA account and API key. The app requests sample us2024a, filters STATEFIP=25, and uses PERWT to sample Massachusetts records. Preparation may take several minutes. Later runs use the local file without an IPUMS API key.

The extract is saved as ~/.synthetic_surveys/data/ipums/massachusetts.csv.gz. A successful replacement is atomic; failed downloads preserve the old extract. For a compatible existing local extract, copy it to that path. Old development checkouts may have data/ipums/massachusetts_us2024a_3.csv.gz; copy that file to the new location to reuse it.

Do not commit or redistribute IPUMS microdata. Redistribution requires IPUMS permission outside the stated publication exception. Follow IPUMS USA terms, and retain the extract's citation for reports/publications. The downloader specification is in synthetic_surveys/ipums_api.py; no IPUMS data is bundled.

Documents and limits

Executable downloads include text and PDF extraction. Text-like files work with the base Python installation. Install .[ocr] from a checkout for PDF extraction and image OCR; image OCR also requires the Tesseract executable. Model-native attachment support varies by provider and model.

Limits: four attachments, 4 MB per file, 12 MB total, a 4,000-character question, and 2,000 characters of regeneration feedback. Questions, document content, and generated personas are sent to the selected providers. Reports contain research inputs and run metadata, so choose their storage location accordingly.

Environment variables

All are optional unless needed by the selected feature. Environment values take precedence over saved settings.

Variable	Purpose/default
OPENROUTER_API_KEY	Default LLM and embedding credential
OPENAI_API_KEY	Direct OpenAI credential; optional embedding fallback
GEMINI_API_KEY	Only for direct Gemini embeddings
IPUMS_API_KEY	Only for downloading an IPUMS extract
SYNTHETIC_SURVEYS_HOME	App settings/data/cache root; defaults to ~/.synthetic_surveys
SYNTHETIC_SURVEYS_LLM_PROVIDER	openrouter (default), openai, or offline stub
SYNTHETIC_SURVEYS_EMBEDDING_PROVIDER	openrouter, openai, or gemini; CLI defaults to the selected LLM provider
SYNTHETIC_SURVEYS_MODEL	Override all LLM stages
SYNTHETIC_SURVEYS_SURVEY_MODEL	Override survey generation model
SYNTHETIC_SURVEYS_SIMULATION_MODEL	Override simulation model
SYNTHETIC_SURVEYS_SUMMARY_MODEL	Override interpretation model
SYNTHETIC_SURVEYS_OPENROUTER_EMBEDDING_MODEL	openai/text-embedding-3-small
SYNTHETIC_SURVEYS_OPENAI_EMBEDDING_MODEL	text-embedding-3-small
SYNTHETIC_SURVEYS_EMBEDDING_MODEL	Direct Gemini embedding model
SYNTHETIC_SURVEYS_IPUMS_SAMPLE	Download sample override; default us2024a
SYNTHETIC_SURVEYS_SSR_EPSILON	SSR bias offset; default 0.0
SYNTHETIC_SURVEYS_SSR_TEMPERATURE	SSR temperature; default 1.0

The checked-in defaults use openai/gpt-5.5 for OpenRouter survey/summary stages and openai/gpt-5.4-mini for simulation. Provider availability can change; use model overrides if necessary.

Local development

uv sync --extra dev
cp .env.example .env
# Edit .env locally to add your key. Never commit it.

The application does not load .env itself. If you use direnv, create .envrc containing:

dotenv_if_exists .env
if [[ -d .venv/bin ]]; then
  PATH_add .venv/bin
fi

Then run direnv allow with direnv installed and hooked into your shell. Alternatively export the variables in your shell. Both files are gitignored.

uv run synthetic-surveys
uv run ruff check .
uv run ty check
uv run pytest -q

For an offline demo with placeholder responses, use uv run synthetic-surveys --provider stub --mode normal. This is not a real simulation. Live OpenAI tests require SYNTHETIC_SURVEYS_RUN_OPENAI_TESTS=1 and OPENAI_API_KEY.

See release verification and methodology. License: Apache-2.0.