pygarble

A local first line of defence for text: secrets, PII, profanity and gibberish, with explainable findings and redaction. No LLM calls or model downloads; the base install has no runtime dependencies.

Documentation · Changelog · Issues

Why pygarble

• Local processing. Native checks use bundled rules, dictionaries and character statistics. Optional integrations run locally and are enabled explicitly.
• Deterministic and explainable. Every finding has a kind, span, confidence and reason. Finding and ScanReport objects never carry the matched text, so logging a report from Python cannot leak a secret. pygarble scan rows include the input line; use pygarble redact when output goes to logs.
• Separate APIs. Screen secrets, PII and profanity together, or check gibberish independently. The combined scanner covers all four categories.
• Redaction in three modes. Placeholders such as [EMAIL], length-preserving masks, or partial masks that keep the last four digits of a card or phone number.
• A document CLI. pygarble-screen scan emits findings-only JSON per document; pygarble-screen redact preserves document structure, including multiline private keys. A line-oriented pygarble CLI is also available.

Install and choose an API

python -m pip install pygarble

Upgrading from 0.8.0? This README describes 0.11.0, the first release since 0.8.0, and it is a breaking release: gibberish scores and the default ensemble's decisions change, and no setting restores 0.8.0 output exactly. 0.8.0 imports keep working, with a DeprecationWarning where a name moved. Read the upgrade guide before upgrading, then install it with pip install --upgrade pygarble.

Ten-second start

Use pygarble.screening for secrets, PII and profanity. Keep gibberish detection separate with pygarble.gibberish.GarbleDetector / pygarble.gibberish.EnsembleDetector. The top-level pygarble.Scanner runs all four categories.

from pygarble.screening import Scanner

scanner = Scanner(max_input_length=100_000)
assert scanner.redact("mail jane@example.com").text == "mail [EMAIL]"
assert not scanner.scan("qxzjkwpv bnmqwer zzxqv").flagged

The dedicated document CLI keeps source text out of scan reports and handles multiline private keys during redaction:

python -m pygarble.screening scan document.txt
python -m pygarble.screening redact document.txt

Optional local backends are explicitly enabled, and do not change the base installation. pip install 'pygarble[screening]' adds phonenumberslite, python-stdnum and detect-secrets. Gitleaks requires a separately installed executable. None requires LLM calls or model downloads.

After installing the optional dependencies and Gitleaks:

from pygarble.screening import Scanner

scanner = Scanner(
    backends=["phonenumbers", "stdnum", "detect-secrets", "gitleaks"],
    backend_options={"phonenumbers": {"region": "GB"}},
    max_input_length=100_000,
)

See the standalone screening guide for individual extras, backend options, custom detectors and coverage limits. Reuse a scanner across documents; Gitleaks starts a subprocess per document.

Combined API and line CLI

The combined scanner runs all four categories. Its line-oriented CLI echoes input text; use the document CLI above for findings-only logs and multiline redaction.

python -m pip install pygarble
printf 'mail jane@example.com\nhello\n' | pygarble scan
printf 'key AKIAIOSFODNN7EXAMPLE\n' | pygarble redact

$ printf 'mail jane@example.com\nhello\n' | pygarble scan
flagged	email	mail jane@example.com
clean		hello
$ printf 'key AKIAIOSFODNN7EXAMPLE\n' | pygarble redact
key [AWS_ACCESS_KEY_ID]

from pygarble import redact, scan

report = scan("mail jane@example.com, key AKIAIOSFODNN7EXAMPLE, damn")
assert report.kinds() == ("aws_access_key_id", "email", "profanity")
assert redact("mail jane@example.com").text == "mail [EMAIL]"

report.flagged is true when any finding reaches min_confidence (default 0.5) or the gibberish ensemble flags the text. Each Finding has category, kind, start, end, confidence and reason; report.to_dict() is JSON-ready.

What it catches

Confidence is 1.0 for checksum-verified or vendor-prefixed matches, down to 0.6 for keyword-plus-entropy secrets and ambiguous masking, and 0.5 for standalone high-entropy strings, which are opt-in (secrets_without_context=True); see the screening guide.

Coverage limits: native rules do not detect names, postal addresses, free-text dates of birth, hate speech beyond the word list, secrets without a recognisable shape, or non-English profanity. A clean result means the selected checks found nothing; choose any further review according to your application's requirements.

Redact

from pygarble.screening import Scanner

scanner = Scanner(categories=["secrets", "pii"])
text = "card 4111 1111 1111 1111, mail jane@example.com"
assert scanner.redact(text).text == "card [CREDIT_CARD], mail [EMAIL]"
assert scanner.redact(text, mode="partial").text == (
    "card ***************1111, mail ****************"
)

Overlapping findings merge into one region. placeholder templates accept {KIND}, {kind} and {category}; mask preserves length; partial keeps the last four characters when a region ends with a credit_card, phone, iban, ssn_us, nhs_number or aadhaar finding. Gibberish is never redacted.

Throughput

Measured on an Apple M2 (macOS arm64, Python 3.12.2) with python paper/regression/throughput.py --size-mb 2, on a synthetic corpus of five ASCII English paragraphs with 5% planted findings. MB is 10^6 UTF-8 bytes of scanned text; your numbers will differ.

These measurements use the native rules through the combined scanner and do not include optional backends. Use pygarble.screening.Scanner for the three rule categories without the gibberish ensemble. A single 1 MB line through the three rule categories took 0.11 s (repeated sentence) to 0.37 s (random dictionary words) in this benchmark. Reuse scanner instances; Gitleaks adds a subprocess invocation for each document.

Gibberish detection

The gibberish category is pygarble's English gibberish detector, also available on its own. It flags keyboard mashing, mojibake, control artifacts and degenerate model output.

from pygarble.gibberish import EnsembleDetector

detector = EnsembleDetector()
assert detector.predict("Hello world") is False
assert detector.predict("asdfghjkl") is True
assert detector.predict(["Hello world", "qxzjkwpv"]) == [False, True]

True means the selected checks flagged the text. score() returns the same heuristic value as a float in [0, 1]; it is not a calibrated probability.

Profiles and use cases

Named profiles default to any voting, except english_fusion, which defaults to fisher. With voting="fisher" each applicable member's score is read against that strategy's scores on a synthetic English null built from word frequencies, giving a coarse tail p-value; Fisher's method combines them and fisher_alpha (default 0.001) sets where the score crosses 0.5. Members are correlated and the null is synthetic, so fisher_alpha is not a guaranteed false-positive rate. Like every English-reference check, english_fusion can flag meaningful text in other languages written in Latin letters, and it gives no signal on other scripts: Russian or Chinese text reads as clean, which is not evidence that it is meaningful. pygarble check, score and analyze read one text per line (--field NAME for JSON lines); check exits 1 when any line is garbled. See the CLI guide and API guide.

$ printf 'hello world\nasdfghjkl\n' | pygarble check
clean	hello world
garbled	asdfghjkl
$ pygarble check --profile llm_output -t "the the the the the the the the"
garbled	the the the the the the the the

Calibrate, explain, configure

from pygarble.gibberish import EnsembleDetector, calibrate

garbled = ["qxzjkwpv bnmqwer", "asdfghjkl"]
clean = ["hello world", "please send the invoice"]
report = calibrate(EnsembleDetector(), garbled, clean)
detector = EnsembleDetector(threshold=report.recommended.threshold)
assert detector.predict(garbled) == [True, True]
assert detector.predict(clean) == [False, False]

calibrate() reports precision, recall, F1 and false-positive rate at every observed score and recommends a threshold; pygarble calibrate --garbled bad.txt --clean good.txt does the same from files. See the calibration guide.

import json
from dataclasses import asdict
from pygarble.gibberish import GarbleDetector, Strategy

controls = GarbleDetector(Strategy.CONTROL_CHARACTERS)
assert controls.predict("hello\x00world") is True
assert controls.predict("hello\nworld") is False

text = "Please review qxzjkwpvm before delivery."
result = GarbleDetector(Strategy.LOCAL_ANOMALY).analyze(text)
assert result.garbled is True
assert result.status == "garbled"
for span in result.spans:
    print(text[span.start:span.end], span.reason)
payload = json.dumps(asdict(result))

All 32 strategies are available through GarbleDetector and the Strategy enum; see the strategy guide. analyze() records the decision, score, status, per-strategy signals and spans (Python string offsets, exclusive end). Empty or wholly inapplicable input returns False with status insufficient_evidence.

from pygarble.gibberish import EnsembleDetector

detector = EnsembleDetector(
    allowlist=["syzygy", "myproductname"],
    max_input_length=100_000,
)
assert detector.predict("syzygy") is False

Allowlists apply to every strategy. Oversized input raises ValueError; invalid batch entries raise TypeError before processing begins.

English-specific behavior

Meaningful Hindi and other non-English text may score as gibberish. This is expected for English-specific checks. A False result does not prove the text is meaningful English; pygarble is not a language identifier or semantic nonsense detector. The corruption profile checks encoding and control artifacts without English plausibility scoring:

from pygarble.gibberish import EnsembleDetector

assert EnsembleDetector().predict("नमस्ते दुनिया") is True

corruption = EnsembleDetector(profile="corruption")
assert corruption.predict("नमस्ते दुनिया") is False
assert corruption.predict("Café au lait") is True
assert corruption.predict("hello\x00world") is True

Upgrading and evaluation

Relative to 0.8.0, 0.11.0 changes gibberish scores and the default ensemble; profile="legacy" selects the 0.8.0 members but does not reproduce their scores. Review the upgrade guide before changing versions.

The preprint (PDF also in paper/preprint/) measures the gibberish detector on two public English datasets, Survey Junk and Tweets vs. Random Strings, against two CPU baselines, with every method at its default decision. The english_fusion profile captures 89.5% of the survey junk and recalls 99.1% of the random strings at a precision of 0.906, from a 1.05 MB install. A Markov-chain baseline flags fewer tweets, a language identifier captures more survey junk, and the default english profile flags 33.2% of tweets. See the benchmark guide for every result, interval and limitation. PII, secrets and profanity are outside that benchmark.

Run make benchmark-verify to recompute every metric from the committed per-row predictions; the raw data is never committed. A seeded 600-row sample of the Apache-2.0 dataset is committed as the input of the golden corpus that CI checks.

• Changelog
• Preprint, benchmark and reproduction
• Engineering regression checks
• Golden corpus of frozen detector outputs for every profile on that sample and hand-written edge inputs, and a golden scan corpus for the scanner, both checked in CI
• Data provenance and curation

Citation

Cite the preprint for the benchmark results and the software entry for the library. The preprint is archived on Zenodo under DOI 10.5281/zenodo.23312785, which always resolves to the latest version.

@misc{rao2026pygarble,
  author       = {Rao, Ujjwal Singh and Singh, Ricky},
  title        = {How Much Junk Do Rules Catch? Evaluating and Tuning pygarble, a Dependency-Free Gibberish Screener, on Survey Responses and Social-Media Text},
  year         = {2026},
  howpublished = {Preprint, Zenodo},
  doi          = {10.5281/zenodo.23312785},
  url          = {https://doi.org/10.5281/zenodo.23312785}
}

@misc{pygarble,
  author       = {Rao, Ujjwal Singh and Singh, Ricky},
  title        = {pygarble: local screening of text for secrets, {PII}, profanity and gibberish},
  year         = {2026},
  howpublished = {\url{https://github.com/brightertiger/pygarble}},
  version      = {0.11.0},
  url          = {https://github.com/brightertiger/pygarble/releases/tag/v0.11.0},
  note         = {Version 0.11.0}
}

Repository layout and compatibility

Library code is in pygarble/, unit tests in tests/, and the website in docs/. Research and tooling are grouped under paper/: preprint/ contains the preprint, benchmark/ its protocol and results, scripts/benchmark/ the benchmark code, regression/ engineering checks, and the other files in scripts/ maintain package data and documentation.

pygarble/
  screening/
    pii/           # detector, patterns, checksums
    profanity/     # detector, word lists, normalization
    secrets/       # detector, patterns, entropy
    backends/      # phonenumbers, stdnum, detect-secrets, Gitleaks
    scanner.py     # standalone three-category scanner
  gibberish/
    strategies/    # individual heuristic strategies
    detector.py    # single-strategy API
    ensemble.py    # profiles and voting
    calibration.py # threshold selection
  data/            # shared tables and portable JSON copies
  scanner.py       # compatible four-category scanner

Old paths such as pygarble.core, pygarble.strategies, pygarble.pii and pygarble.profanity are compatibility pointers; 0.8.0 names that moved emit a DeprecationWarning naming their new home. Old and new imports share classes, enums, rule tables and caches. See the architecture guide for the full layout, import mappings and lazy-loading behavior.

Contributing

git clone https://github.com/brightertiger/pygarble.git
cd pygarble
python -m pip install -e ".[dev]"
python -m pytest -q

See the contributing guide for quality checks, documentation builds, and data regeneration. Please include your Python version, package version, selected profile, and a minimal input when reporting a problem.

Open source and publishing

The repository is public, and releases are distributed on PyPI. Contributions and synthetic bug reports are welcome; see CONTRIBUTING.md and SECURITY.md. The maintainer guide covers releases, Google Search Console and documentation discovery.

License

Library code is MIT licensed. The profanity word list is seeded from the LDNOOBW English list (Shutterstock), CC-BY-4.0, filtered and extended by the pygarble maintainers. Data provenance for the gibberish tables is recorded in the curation manifest linked above.