surveyshield-py 0.4.15

Static review of online survey instruments for resistance to AI/bot respondents

🛡️ Survey Shield

Survey Shield reviews your survey for resistance to AI/bot respondents and hands you back a score, a list of concrete weaknesses, and a copy-paste Methods paragraph you can drop straight into your manuscript.

You upload a Qualtrics .qsf file. Survey Shield returns a self-contained HTML report you can email, print, or attach as a supplement.

What you get

Instrument Review · my-survey.qsf · 2026-05-10

   Defense score                72 / 100
   Bot completion likelihood    28 / 100   (lower is better)
   Defense breadth                5 /   8   categories with a present defense

   Bot-resistance verdict
      "This instrument has a strong attention-check layer but no
       behavioral telemetry. Consider adding reCAPTCHA v3 or a
       total-survey-time gate before payment."

   Per-category review
      Logic                       85 / 100   ·  2 findings
      Visual Reasoning            60 / 100   ·  1 finding
      Traps                       95 / 100   ·  0 findings
      Open Ends                   80 / 100   ·  1 finding
      Mouse and Keyboard Input     0 / 100   ·  1 finding   ← missing
      Behavioral                   0 / 100   ·  2 findings  ← missing
      Context Awareness           40 / 100   ·  3 findings
      ECLAIR                     100 / 100   ·  0 findings

   Methods statement (copy into your paper)
      "We evaluated this instrument for resistance to non-human
       responses using Survey Shield (Fernandez et al., 2026). The
       instrument received an overall defense score of 72/100 and an
       estimated completion-likelihood for automated agents of 28/100.
       Reviews were generated using openai gpt-5.4-mini-2026-03-17 on
       2026-05-11. Review ID: …"

The full HTML report adds per-category cards with the quoted survey text, a Top Recommendations section, and APA + BibTeX citation blocks with copy buttons.

Same QSF + same model → same score, every time. The methods paragraph above captures everything a reader needs to reproduce the run; the model alias you pass (e.g. gpt-5.4-mini) is automatically pinned to a dated snapshot (gpt-5.4-mini-2026-03-17) so reports stay reproducible after the floating alias rotates.

60-second quickstart

pip install surveyshield-py
export OPENAI_API_KEY=sk-...
surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input — open it in a browser

That's the whole tool for most researchers. Cost: ~$0.05–$0.30 per review on the default model, depending on survey size. Wall-clock: 30–90 seconds.

What Survey Shield evaluates

Eight bot-resistance categories, grounded in Westwood et al. (PNAS 2025) and related work on detecting automated respondents:

Category	What it tests
Logic	Cognitive Reflection Test items, Sally-Anne theory-of-mind, syllogisms, impossible-event probes
Visual Reasoning	Image-based illusions, counting elements, perspective tasks
Traps	Attention checks, human-attestation oaths, invisible-text instructions
Open Ends	Knowledge-gap probes ("first paragraph of the Constitution"), reverse-shibboleths
Mouse and Keyboard Input	Map clicks, drag-and-drop, keystroke-timing tracking
Behavioral	reCAPTCHA v3, IAT latencies, total-survey-time gating
Context Awareness	"Is it raining where you are?" verified against a weather API
ECLAIR	Refusal probes — questions safety-tuned LLMs refuse but humans answer freely

Scope: what Survey Shield does not do

Survey Shield is scoped strictly to bot resistance. It does not critique your research design, theoretical framing, or question wording. Those remain your domain. Run the same QSF through Survey Shield twice and you'll get the same score; what neither run will tell you is whether the underlying questions are good research.

Install

pip install surveyshield-py              # review-only — small, no browser
pip install "surveyshield-py[live]"      # adds browser-use + Playwright (live mode)
pip install "surveyshield-py[pdf2qsf]"   # adds opendataloader-pdf (PDF → QSF converter); needs JDK 11+

The PyPI name is surveyshield-py; the import name is surveyshield.

Set an LLM provider key in your environment (or a .env file in the working directory — Survey Shield loads it via python-dotenv):

OPENAI_API_KEY=sk-...        # default
GOOGLE_API_KEY=...           # for Gemini models (model name starting with "gemini")

Usage

CLI

surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input

surveyshield review your_survey.qsf --output report.html --json review.json
surveyshield review your_survey.qsf --model gpt-4o --categories logical,content-traps

surveyshield serve --host 127.0.0.1 --port 8000
# → boots the FastAPI app + web UI at http://localhost:8000

surveyshield --help lists every command and flag.

Batch audit (many QSFs at once)

surveyshield audit runs review over a directory of QSFs (or a manifest CSV) and emits a long-format CSV joinable with manifest metadata (year, journal, study_id, …). One row per (paper × rubric × category), plus a per-paper HTML report + JSON under the output directory.

surveyshield audit path/to/jcr_qsfs/ \
    --rubrics traps,traps-behavioral,traps-behavioral-visual,full \
    --output-dir audit_output/

# Manifest form (preferred for the paper — extra columns pass through):
surveyshield audit manifest.csv --rubrics full
# manifest.csv columns: paper_id, qsf_path, year, journal, study_id, ...

The CSV is the analysis hand-off: load it into pandas / R for whatever figures the downstream paper or report needs.

Python

import asyncio
import surveyshield

review, _parsed = asyncio.run(
    surveyshield.review_qsf(
        "your_survey.qsf",
        model="gpt-5.4-mini",
        # categories=["content-traps", "eclaire"],   # default = all 8
    )
)

print(review.overall_score, review.overall_feedback.headline)
print(review.methods_statement)

with open("report.html", "w") as f:
    f.write(surveyshield.render_html(review))

The review object is a surveyshield.InstrumentReview Pydantic model with categories, recommendations, overall_feedback, parameters, and methods_statement fields. See surveyshield/__init__.py for the public surface.

Optional: live runtime

The live runtime drives a real browser through your survey URL and reports which categories actually blocked an AI agent (vs. just being present in the QSF). It costs ~$0.20 and 5–10 minutes per run, so the hosted demo doesn't expose it.

pip install "surveyshield-py[live]"
playwright install chromium

# Single run.
surveyshield take https://qualtrics.com/jfe/form/SV_xxx --max-steps 150

# Pre-launch sanity check (API keys, real-Chrome, URL reachable, …).
surveyshield preflight https://qualtrics.com/jfe/form/SV_xxx

# One cell — N runs of one (model, prompt), manifest CSV, resumable.
surveyshield take-corpus https://qualtrics.com/jfe/form/SV_xxx \
  --out runs/cell_A1 --runs 20 --concurrency 2

# Full factorial study — every (model × prompt) combo, replacement-safe.
surveyshield run-experiment https://qualtrics.com/jfe/form/SV_xxx \
  --out runs/study --runs-per-combo 256 --temperature 0.7

result = asyncio.run(surveyshield.take_survey(
    "https://qualtrics.com/jfe/form/SV_xxx",
    model="claude-sonnet-4-6",         # or gpt-4.1 / gemini-3.1-flash-lite
    prompt_id="surveyshield",          # or general / westwood / atlas / comet
    temperature=0.7,                   # uniform across model arms
    reviewer_model="gpt-4o-mini",      # None (default) = skip the 8-category reviewer
))
print(result.defense_score, result.bot_completion_likelihood)

Live and static reviews share the same 8-category rubric and produce the same shape of result — the difference is what they evaluate. Static review asks "is this defense implemented in the QSF?"; live review asks "did this defense actually work when an agent ran the survey?".

The runtime ships with three verified agent arms (claude-sonnet-4-6, gpt-4.1, gemini-3.1-flash-lite), five interchangeable prompts (surveyshield, general, westwood, atlas, comet), a run-experiment factorial dispatcher with prereg-compliant failure replacement, a Qualtrics-aware runId URL stamp so response exports left-join to the manifest CSV, and CapSolver / 2Captcha integration for in-survey CAPTCHAs.

See surveyshield/live/README.md for the full picture: prompt-variant catalog, persistent-profile setup, env-var reference, deployment guidance (researcher laptop vs. cloud), corpus-harness mechanics, and troubleshooting.

Optional: PDF → QSF converter

Some survey appendices are shared as PDFs (or DOCX) rather than as machine-reviewable QSF exports — about 14% of the JCR papers we indexed for the v0.4 audit. The pdf2qsf extra adds a three-stage converter (PDF → blocks via opendataloader-pdf → SurveyIR via one LLM call → QSF via deterministic compute) so you can feed those appendices into surveyshield review / surveyshield audit like any other QSF.

pip install "surveyshield-py[pdf2qsf]"   # adds opendataloader-pdf; requires JDK 11+ on PATH

surveyshield pdf2qsf appendix.pdf                    # → appendix.qsf next to the PDF
surveyshield pdf2qsf survey_pdfs/ --out-dir qsfs/ --manifest manifest.csv

from pathlib import Path
import surveyshield

qsf_path = surveyshield.pdf_to_qsf(Path("appendix.pdf"))
review, _ = asyncio.run(surveyshield.review_qsf(qsf_path))

Every emit is round-trip-checked against Survey Shield's own parser before the file lands on disk, and every converted QSF stamps source PDF / extractor / LLM / timestamp provenance into its SurveyDescription so it surfaces verbatim in the audit context. Multi-study PDFs emit one QSF per detected study (named <pdf_stem>__pdf_<label>.qsf), and the optional manifest CSV is slot-compatible with surveyshield audit's manifest format.

Self-host the web UI

git clone https://github.com/kiante-fernandez/survey-shield
cd survey-shield
./setup.sh                              # creates .conda env + .env stub
echo "OPENAI_API_KEY=sk-..." >> .env
surveyshield serve --reload             # → http://localhost:8000

Endpoints once it's running:

• Web UI: http://localhost:8000
• Interactive API docs: http://localhost:8000/docs
• Health: http://localhost:8000/health

HTTP API (for integration into other tools)

# Submit a QSF
curl -F "file=@your_survey.qsf" http://localhost:8000/api/v1/instrument/review
# → {"review_id": "<uuid>", "status": "queued", ...}

# Poll until complete (~30–90 s)
curl http://localhost:8000/api/v1/instrument/status/<uuid>

# Fetch the structured JSON result
curl http://localhost:8000/api/v1/instrument/results/<uuid>

# Or the HTML report
curl http://localhost:8000/api/v1/instrument/report/<uuid>
curl -OJ "http://localhost:8000/api/v1/instrument/report/<uuid>?download=1"

The live-runtime endpoint mirrors this shape at /api/v1/survey/* and is only enabled when an LLM API key is set in the env.

Citation

If you use Survey Shield in published work:

@misc{fernandez2026surveyshield,
  author = {Fernandez, K. and Low, A. and Bogard, J. and Fox, C. R.},
  title  = {Survey Shield: Static review of online survey instruments for resistance to non-human responses},
  year   = {2026},
  note   = {Manuscript in preparation},
}

Contributing

See CONTRIBUTING.md for the local dev setup, test suite layout, and pull-request workflow. Bug reports + category-rubric suggestions are very welcome.

License

MIT — see LICENSE.