surveyshield-py 0.1.0

Static review of online survey instruments for resistance to AI/bot respondents

🛡️ Survey Shield

Static review of online survey instruments for resistance to AI/bot respondents — plus an optional live runtime that drives a real browser through your survey.

What is Survey Shield?

Survey Shield gives researchers feedback on whether their survey instrument is hardened against AI respondents. Two paths:

1. Instrument Review (primary, static, no browser) — point it at a Qualtrics .qsf export. Multi-agent LLM reviewers fan out across bot-resistance dimensions (attention checks, identity questions, visual-perceptual traps), produce a peer-review-style verdict with verbatim-grounded findings, and render a self-contained HTML report with a copy-paste Methods statement and APA/BibTeX citation.
2. Take Survey (live runtime) (optional [live] extra) — drives a real browser (browser-use) through a live Qualtrics URL, reports detected mechanisms after the fact. Costs ~5–10 minutes and real LLM credit per run, so the hosted demo doesn't expose it; install the [live] extra to run it locally.

Researchers using Survey Shield mostly want Instrument Review. Reach for the live runtime when you need to exercise the survey end-to-end.

Install

pip install surveyshield-py            # static review only — small, no browser
pip install "surveyshield-py[live]"    # adds browser-use + Playwright
playwright install chromium            # only if you installed [live]

The package name on PyPI is surveyshield-py (after openreview-py); the import name is surveyshield.

Set an LLM provider key in your environment (or in a .env file in the working directory — Survey Shield loads it via python-dotenv):

OPENAI_API_KEY=sk-...        # used unless the model name starts with "gemini"
GOOGLE_API_KEY=...           # used for Gemini models

CLI

surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input

surveyshield review your_survey.qsf --output report.html --json review.json \
                                    --model gpt-4o-mini

surveyshield take https://qualtrics.com/jfe/form/SV_xxx     # requires [live]
                  --model gemini-3-flash-preview --max-steps 150

surveyshield serve --host 127.0.0.1 --port 8000
# → boots the FastAPI app + bundled React SPA

surveyshield --help lists every command and flag.

Python API

import asyncio
import surveyshield

review, parsed = asyncio.run(
    surveyshield.review_qsf(
        "your_survey.qsf",
        model="gpt-4o-mini",
        # api_key="sk-...",      # or rely on env vars
        # dimensions=["attention_checks"],   # default = all
    )
)

print(review.overall_score, review.overall_feedback.headline)

with open("report.html", "w") as f:
    f.write(surveyshield.render_html(review, parsed))

The review object is a surveyshield.InstrumentReview Pydantic model. Power users can compose the lower-level seams directly: parse_qsf, run_review, drop_unverified_quotes, consolidate_and_summarize, aggregate. See surveyshield/__init__.py for the public surface.

For live runtime:

import asyncio, surveyshield  # surveyshield-py[live] installed

result = asyncio.run(surveyshield.take_survey(
    "https://qualtrics.com/jfe/form/SV_xxx",
    model="gemini-3-flash-preview",
    max_steps=150,
))
print(result.success_probability, [m.name for m in result.detected_mechanisms])

If the [live] extra isn't installed, surveyshield.take_survey resolves to None and the CLI's take command exits with a clear install hint.

Self-host the hosted UI

git clone https://github.com/kiante-fernandez/survey-shield
cd survey-shield
./setup.sh                                   # creates ../.conda env
echo "OPENAI_API_KEY=sk-..." > backend/.env  # or GOOGLE_API_KEY
cd backend && ./start.sh                     # → http://localhost:8000

The Take Survey tab is gated on live_take_enabled — GET /api/v1/survey/config flips it to true once a key is detected in the env.

Endpoints

• Web UI: http://localhost:8000
• Interactive API docs: http://localhost:8000/docs
• Health check: http://localhost:8000/health
• Live-runtime config: http://localhost:8000/api/v1/survey/config

Models

The hosted UI does not expose a model picker — Instrument Review reviewers run on a sensible default (gpt-4o-mini). Self-hosters who want a different model can pass model_name directly to the API or CLI. The backend has no allowlist; any model name langchain-openai's ChatOpenAI or langchain-google-genai's ChatGoogleGenerativeAI accept will be routed by prefix:

• Names starting with gemini → Google (requires GOOGLE_API_KEY)
• Everything else → OpenAI (requires OPENAI_API_KEY)

API usage (self-host)

Instrument Review (primary)

# Submit a QSF for review
curl -F "file=@your_survey.qsf" \
     http://localhost:8000/api/v1/instrument/review
# → {"review_id": "<uuid>", "status": "queued", ...}

# Poll
curl http://localhost:8000/api/v1/instrument/status/<uuid>
# queued → running → completed (~30–90 s)

# Structured JSON
curl http://localhost:8000/api/v1/instrument/results/<uuid>

# Human-readable HTML report
curl "http://localhost:8000/api/v1/instrument/report/<uuid>"

# Download as a file
curl -OJ "http://localhost:8000/api/v1/instrument/report/<uuid>?download=1"

Live runtime (self-host only)

curl -X POST "http://localhost:8000/api/v1/survey/analyze" \
     -H "Content-Type: application/json" \
     -d '{
       "survey_url": "https://example.com/survey",
       "model_name": "gpt-4o-mini",
       "max_steps": 150,
       "use_vision": true
     }'
# Then poll /api/v1/survey/status/<id> and fetch /api/v1/survey/results/<id>.

What Survey Shield evaluates

Instrument Review dimensions (primary)

The reviewer fans out across plug-in dimensions defined in surveyshield/review/dimensions.py. v1 ships three:

• attention_checks — explicit IMCs and instructional manipulation checks (Westwood, 2025; PNAS).
• identity_questions — direct LLM-resistance items (identity probes, reverse-shibboleth questions, impossible-event questions).
• visual_perceptual_traps — image- and layout-based cognitive traps that exploit vision-language model architectural constraints (Affonso, 2026; JCR).

Adding a new dimension is one entry in dimensions.py plus a Pydantic output schema — the reviewer fan-out, aggregator, and HTML report are all plug-in-driven.

Findings are grounded: every reviewer finding must cite a verbatim excerpt from the source survey. Survey Shield substring-checks each excerpt against the parsed survey and drops anything that can't be located. What reaches the report is grounded in real survey content.

The product is scoped strictly to bot resistance. We do not critique a survey's substantive research design, theoretical framing, or question wording — those remain the researcher's domain.

Live-runtime mechanisms

When you run a live analysis against a real Qualtrics URL, Survey Shield's browser-use Agent navigates the survey end-to-end and inventories detected mechanisms by category — hover traps, invisible text, timing requirements, CAPTCHAs, honeypot fields, attention checks, and behavioural / mouse-tracking signals. Per-mechanism severity weights (1–10) feed into success-probability and difficulty scores. See surveyshield/live/analyzer.py for the taxonomy and surveyshield/live/prompts.py for the agent task.

Development

Project structure

survey-shield/
├── surveyshield/                # the importable package
│   ├── __init__.py              # public API (review_qsf, render_html, take_survey, …)
│   ├── cli.py                   # Typer CLI (review / take / serve)
│   ├── models/                  # Pydantic schemas
│   ├── review/                  # static review pipeline
│   │   ├── parser.py            #   QSF → ParsedSurvey
│   │   ├── dimensions.py        #   plug-in dimension registry
│   │   ├── reviewer.py          #   fan-out, verify, consolidate, aggregate
│   │   ├── mechanism_context.py
│   │   └── templates/           #   Jinja2 self-contained HTML report
│   ├── live/                    # browser-use runtime ([live] extra)
│   │   ├── analyzer.py          #   SurveyAnalyzer / take_survey
│   │   ├── prompts.py
│   │   └── patches.py
│   └── serve/                   # FastAPI app + bundled React SPA
│       ├── app.py
│       ├── config.py
│       ├── api/{survey,instrument}.py
│       └── static/              # built React (populated by bin/build.sh)
├── frontend/                    # React/TypeScript source (CRA)
├── tests/                       # pytest suite + tiny QSF fixture
├── backend/start.sh             # convenience wrapper around `surveyshield serve`
├── pyproject.toml               # canonical package metadata
├── Procfile                     # web: gunicorn surveyshield.serve.app:app
├── bin/build.sh                 # React build → surveyshield/serve/static/
└── .github/workflows/           # test.yml + release.yml (PyPI trusted publishing)

Local dev

./setup.sh                                   # one-time: conda env at ../.conda
pip install -e ".[dev,live]"                 # editable install + tests + browser-use
pytest -q                                    # ~30 tests, no LLM calls
cd frontend && npx tsc --noEmit && npm run build

Releasing

git tag v0.1.0 && git push --tags
# .github/workflows/release.yml builds the wheel + sdist (with the React SPA
# bundled into surveyshield/serve/static/) and publishes to PyPI via OIDC.

The PyPI project must be configured with this repo + release.yml as a Trusted Publisher before the first push.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Submit a pull request

License

MIT License — see LICENSE.

Citation

If you use Survey Shield in published work:

@misc{fernandez2026surveyshield,
  author = {Fernandez, K. and Low, A. and Bogard, J. and Fox, C. R.},
  title  = {Survey Shield: Static review of online survey instruments for resistance to non-human responses},
  year   = {2026},
  note   = {Manuscript in preparation},
}

Every report includes the same citation pre-formatted (APA + BibTeX).

Disclaimer

Survey Shield is intended for research and testing purposes.