DeepSafe

Do deepfake detectors actually work? We tested 24 of them.

The short answer: not as well as the papers say. Our ensemble of 19 published detectors catches 66% of fake media, and only 7% of Sora video. This repository has the code, the data, and the numbers, so you can check that yourself rather than take our word for it.

DeepSafe does three things:

Detect	Run 24 models over a file and get one calibrated verdict.
Evaluate	Score any detector across 411 AI generators and see where it breaks.
Adapt	Retrain on your own labeled data, then find out whether it actually generalized.

Free for research, education, and personal use.

Why this exists

Published deepfake detectors report near-perfect accuracy. Most of that accuracy does not survive contact with a generator the model has not seen.

Here is our own ensemble of 19 published detectors, on 15,499 held-out samples:

• Recall on fakes: 66.2%
• False positive rate on real media: 6.5%

A third of fake media walks straight through. And the failures are not evenly distributed:

Generator	Caught
Hunyuan (video)	1.7%
Sora (video)	7.0%
Veo (video)	8.0%
Kling (video)	12.0%
Stable Diffusion 1.4 (image)	100%
Midjourney v5 (image)	100%
GLIDE (image)	100%

The stack is near-perfect on the 2022-2023 diffusion models the literature was built around, and close to blind on the video generators people actually worry about today.

Ensemble AUC, held out: image 0.9466, audio 0.8290, video 0.6694. Those are lower than the cross-validated numbers from meta-learner training (0.9705 / 0.8748 / 0.8898), and that gap is itself the finding. Cross-validated scores on the training distribution overstate what a detector does in the wild.

Almost nobody publishes this. Papers compare against three or four baselines on one or two datasets because getting baselines to run is genuinely brutal, not because researchers are careless.

DeepSafe exists so that stops being the bottleneck. Full numbers, reproducible from the shipped prediction matrix, are in BENCHMARK.md.

What you get

pip install deepsafe-bench

That install has no dependencies. Two of the three verbs work immediately:

# Reproduce any published number, no media and no GPU required
deepsafe eval --baseline ensemble --modality video
deepsafe eval --list-baselines

# Score your own detector: any .py exposing predict(path) -> float
deepsafe eval --model my_detector.py --tier small --out ./results

# Adapt the ensemble to your labeled data, then find out if it generalized
deepsafe fit --tier 1

eval always reports in-distribution and held-out-generator performance separately. That separation is not configurable.

fit splits by generator, not by sample, and reports what happened on generators it never trained on. On our own data it correctly flags that the refit overfits: in-distribution AUC 0.9566, held-out 0.9097, against a single-best-detector baseline of 0.9221. It is built to catch your overfitting, not to flatter it.

deepsafe detect needs the models loaded, which means the [server] extra and roughly 20 GB of VRAM:

git clone https://github.com/deepsafehq/deepsafe-bench.git
cd deepsafe-bench
bash setup.sh          # pulls ~44 GB of weights and model code from HuggingFace

No HuggingFace token is needed; the weights are public. Then start the stack:

# Terminal 1: the models
cd apps/inference && PYTHONPATH=.:../../packages/shared python server.py

# Terminal 2: the API. Runs with no configuration at all, using SQLite and
# no auth, which is what you want locally.
cd apps/gateway && PYTHONPATH=.:../../packages/shared uvicorn main:app --port 8000

# Terminal 3: the web UI at http://localhost:3000
cd apps/web && pnpm install && pnpm dev

Interactive API docs are served at http://localhost:8000/docs.

Before exposing any of this to a network, set DEEPSAFE_REQUIRE_AUTH=true and DEEPSAFE_ENV=production. Auth is off by default so a local install works immediately; production mode requires both auth and a real DATABASE_URL and refuses to start without them.

Fine-tuning coverage

Tier 1 (ensemble refit, CPU) covers the whole stack. Tiers 2 and 3 need a GPU and are implemented per model; 9 of 19 detection models have a verified path, the rest raise a pointer to upstream training instructions:

from deepsafe.fit_gpu import support_table
print(support_table())

Adding an adapter for one of the remaining 10 is the best-scoped contribution available. See CONTRIBUTING.md.

Models

Modality	Count	Models
Image	7	NPR, Yermandy, Universal, AIDE, FSD, Effort, CO-SPY
Video	9	FakeSTormer, SBI, DFD-FCG, PwTF-DVD, LipFD, RECCE, MINTIME, NPR-Video, UnivFD-Video
Audio	3	ShiftySpeech, SafeEar, Nes2Net
Provenance	5	C2PA, SDXL Watermark, TrustMark, AudioSeal, VideoSeal

All 24 run in a single process with one CUDA context. Model code and weights are mirrored on HuggingFace so setup keeps working after upstream repositories and paper download links rot, which in this field they reliably do.

Every model is the work of its original authors under its own license. See THIRD_PARTY_NOTICES.md, which includes an unconditional takedown policy.

Setup

git clone https://github.com/deepsafehq/deepsafe-bench.git
cd deepsafe-bench
bash setup.sh

Requires a CUDA GPU with 24 GB or more of VRAM for the full lineup, NVIDIA driver 535+, CUDA 12.1+, and Python 3.11 or 3.12. Blackwell (sm_120) is not supported.

eval against a stub detector and the small dataset tier runs on CPU.

Limitations

Read these before trusting any output.

• Detection does not generalize. Expect a large drop on any generator absent from the training distribution. This is a property of the field, not a bug in this implementation.
• Not suitable for high-stakes decisions. Do not use this to accuse a person of anything. A confident score is not evidence.
• Audio is weakest outside English. See the multilingual numbers above.
• Provenance signals are stronger than detection, when present. A C2PA manifest or a watermark is real evidence; a model score is an estimate.

License

PolyForm Noncommercial 1.0.0. Free for research, education, personal projects, and use by nonprofits, educational institutions, and government bodies. Commercial use is not permitted.

This is a source-available license, not an OSI-approved open source license.

Citation

See CITATION.cff, or use GitHub's "Cite this repository" button. deepsafe eval also writes a CITATIONS.bib next to every report containing entries for DeepSafe and for every model it ran.