surveyshield-py 0.4.0

Static review of online survey instruments for resistance to AI/bot respondents

🛡️ Survey Shield

Survey Shield reviews your survey for resistance to AI/bot respondents and hands you back a score, a list of concrete weaknesses, and a copy-paste Methods paragraph you can drop straight into your manuscript.

You upload a Qualtrics .qsf file. Survey Shield returns a self-contained HTML report you can email, print, or attach as a supplement.

What you get

Instrument Review · my-survey.qsf · 2026-05-10

   Defense score                72 / 100
   Bot completion likelihood    28 / 100   (lower is better)

   Bot-resistance verdict
      "This instrument has a strong attention-check layer but no
       behavioral telemetry. Consider adding reCAPTCHA v3 or a
       total-survey-time gate before payment."

   Per-category review
      Logic                       85 / 100   ·  2 findings
      Visual Reasoning            60 / 100   ·  1 finding
      Traps                       95 / 100   ·  0 findings
      Open Ends                   80 / 100   ·  1 finding
      Mouse and Keyboard Input     0 / 100   ·  1 finding   ← missing
      Behavioral                   0 / 100   ·  2 findings  ← missing
      Context Awareness           40 / 100   ·  3 findings
      ECLAIR                     100 / 100   ·  0 findings

   Methods statement (copy into your paper)
      "We evaluated this instrument for resistance to non-human
       responses using Survey Shield (Fernandez et al., 2026),
       reviewing it across 8 categories. The instrument received
       an overall defense score of 72/100 and an estimated
       completion-likelihood for automated agents of 28/100.
       Reviews were generated using openai gpt-5.4-mini-2026-03-17
       (temperature=0, seed=2026; stateless calls, no fine-tuning)
       on 2026-05-11. Review ID: …"

The full HTML report adds per-category cards with the quoted survey text, a Top Recommendations section, and APA + BibTeX citation blocks with copy buttons.

Same QSF + same model → same score, every time. The methods paragraph above captures everything a reader needs to reproduce the run; the model alias you pass (e.g. gpt-5.4-mini) is automatically pinned to a dated snapshot (gpt-5.4-mini-2026-03-17) so reports stay reproducible after the floating alias rotates.

60-second quickstart

pip install surveyshield-py
export OPENAI_API_KEY=sk-...
surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input — open it in a browser

That's the whole tool for most researchers. Cost: ~$0.05–$0.30 per review on the default model, depending on survey size. Wall-clock: 30–90 seconds.

What Survey Shield evaluates

Eight bot-resistance categories, grounded in Westwood et al. (PNAS 2025) and related work on detecting automated respondents:

Category	What it tests
Logic	Cognitive Reflection Test items, Sally-Anne theory-of-mind, syllogisms, impossible-event probes
Visual Reasoning	Image-based illusions, counting elements, perspective tasks
Traps	Attention checks, human-attestation oaths, invisible-text instructions
Open Ends	Knowledge-gap probes ("first paragraph of the Constitution"), reverse-shibboleths
Mouse and Keyboard Input	Map clicks, drag-and-drop, keystroke-timing tracking
Behavioral	reCAPTCHA v3, IAT latencies, total-survey-time gating
Context Awareness	"Is it raining where you are?" verified against a weather API
ECLAIR	Refusal probes — questions safety-tuned LLMs refuse but humans answer freely

Scope: what Survey Shield does not do

Survey Shield is scoped strictly to bot resistance. It does not critique your research design, theoretical framing, or question wording. Those remain your domain. Run the same QSF through Survey Shield twice and you'll get the same score; what neither run will tell you is whether the underlying questions are good research.

Install

pip install surveyshield-py            # review-only — small, no browser
pip install "surveyshield-py[live]"    # adds browser-use + Playwright (live mode)

The PyPI name is surveyshield-py; the import name is surveyshield.

Set an LLM provider key in your environment (or a .env file in the working directory — Survey Shield loads it via python-dotenv):

OPENAI_API_KEY=sk-...        # default
GOOGLE_API_KEY=...           # for Gemini models (model name starting with "gemini")

Usage

CLI

surveyshield review your_survey.qsf
# → writes your_survey.report.html next to the input

surveyshield review your_survey.qsf --output report.html --json review.json
surveyshield review your_survey.qsf --model gpt-4o --categories logical,content-traps

surveyshield serve --host 127.0.0.1 --port 8000
# → boots the FastAPI app + web UI at http://localhost:8000

surveyshield --help lists every command and flag.

Batch audit (many QSFs at once)

surveyshield audit runs review over a directory of QSFs (or a manifest CSV) and emits a long-format CSV joinable with manifest metadata (year, journal, study_id, …). One row per (paper × rubric × category), plus a per-paper HTML report + JSON under the output directory.

surveyshield audit path/to/jcr_qsfs/ \
    --rubrics traps,traps-behavioral,traps-behavioral-visual,full \
    --output-dir audit_output/

# Manifest form (preferred for the paper — extra columns pass through):
surveyshield audit manifest.csv --rubrics full
# manifest.csv columns: paper_id, qsf_path, year, journal, study_id, ...

The CSV is the analysis hand-off: load it into pandas / R for whatever figures the downstream paper or report needs.

Python

import asyncio
import surveyshield

review, _parsed = asyncio.run(
    surveyshield.review_qsf(
        "your_survey.qsf",
        model="gpt-5.4-mini",
        # categories=["content-traps", "eclaire"],   # default = all 8
    )
)

print(review.overall_score, review.overall_feedback.headline)
print(review.methods_statement)

with open("report.html", "w") as f:
    f.write(surveyshield.render_html(review))

The review object is a surveyshield.InstrumentReview Pydantic model with categories, recommendations, overall_feedback, parameters, and methods_statement fields. See surveyshield/__init__.py for the public surface.

Optional: live runtime

The live runtime drives a real browser through your survey URL and reports which categories actually blocked an AI agent (vs. just being present in the QSF). It costs ~$0.20 and 5–10 minutes per run, so the hosted demo doesn't expose it.

pip install "surveyshield-py[live]"
playwright install chromium

surveyshield take https://qualtrics.com/jfe/form/SV_xxx --max-steps 150

result = asyncio.run(surveyshield.take_survey(
    "https://qualtrics.com/jfe/form/SV_xxx",
    model="gpt-4o-mini",
    max_steps=150,
))
print(result.defense_score, result.bot_completion_likelihood)

Live and static reviews share the same 8-category rubric and produce the same shape of result — the difference is what they evaluate. Static review asks "is this defense implemented in the QSF?"; live review asks "did this defense actually work when an agent ran the survey?".

Self-host the web UI

git clone https://github.com/kiante-fernandez/survey-shield
cd survey-shield
./setup.sh                              # creates .conda env + .env stub
echo "OPENAI_API_KEY=sk-..." >> .env
surveyshield serve --reload             # → http://localhost:8000

Endpoints once it's running:

• Web UI: http://localhost:8000
• Interactive API docs: http://localhost:8000/docs
• Health: http://localhost:8000/health

HTTP API (for integration into other tools)

# Submit a QSF
curl -F "file=@your_survey.qsf" http://localhost:8000/api/v1/instrument/review
# → {"review_id": "<uuid>", "status": "queued", ...}

# Poll until complete (~30–90 s)
curl http://localhost:8000/api/v1/instrument/status/<uuid>

# Fetch the structured JSON result
curl http://localhost:8000/api/v1/instrument/results/<uuid>

# Or the HTML report
curl http://localhost:8000/api/v1/instrument/report/<uuid>
curl -OJ "http://localhost:8000/api/v1/instrument/report/<uuid>?download=1"

The live-runtime endpoint mirrors this shape at /api/v1/survey/* and is only enabled when an LLM API key is set in the env.

Citation

If you use Survey Shield in published work:

@misc{fernandez2026surveyshield,
  author = {Fernandez, K. and Low, A. and Bogard, J. and Fox, C. R.},
  title  = {Survey Shield: Static review of online survey instruments for resistance to non-human responses},
  year   = {2026},
  note   = {Manuscript in preparation},
}

Contributing

See CONTRIBUTING.md for the local dev setup, test suite layout, and pull-request workflow. Bug reports + category-rubric suggestions are very welcome.

License

MIT — see LICENSE.