Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

datafog

datafog core

Find and transform sensitive data in your application.

PyPI version npm version crates.io version Rust CI status MIT license

Quickstart · Detection · Transformations · Local models · Documentation

datafog core is a Rust library with Python, Node.js, and browser WebAssembly bindings. Scan text or JSON for personal data and credentials, then redact, mask, remove, pseudonymize, or tokenize matched values. Findings use a consistent format across the bindings.

Rule-based scanning runs locally without model weights or downloads. Optional local model sessions add datafog and GLiNER2 support in Rust, Python, and Node.js.

Stable: 1.0.0 · Release notes · Public registry installation checks passed.

Release candidate: 1.1.0-rc.1 · Extension install and acceptance instructions. Install the candidate explicitly; npm latest remains on the stable release.

Core 1.0 freezes Workspace contract 1 across Rust, Python, Node and browser WASM, with optional native/browser FogVault integrations. Snapshot v3 writes and v1/v2/v3 reads keep their existing semantics. Exact published 0.5.0-beta.3 interchange remains tested; FogVault is independently versioned 0.1.0.

Quickstart

Choose the exact 1.0.0 package for your application:

SDK Install 1.0.0 Requirements
Python python -m pip install datafog-core==1.0.0 Python 3.10+
Node.js npm install --save-exact @datafog/node@1.0.0 Node.js 24+
Rust cargo add datafog-core@=1.0.0 Rust 1.88+
Browser / WASM npm install --save-exact @datafog/wasm@1.0.0 Browser with WebAssembly support

Python

from datafog_core import scan, scan_and_transform

findings = scan("Email jane@example.com")
print(findings[0].entity_type)   # EMAIL
print(findings[0].matched_text)  # jane@example.com

result = scan_and_transform(
    "Email jane@example.com",
    {"transform": {"default": {"strategy": "redact"}}},
)
print(result.text)  # Email [EMAIL]
Node.js example
import { scan, scanAndTransform } from "@datafog/node";

const findings = scan("Email jane@example.com");
console.log(findings[0].entityType); // EMAIL
console.log(findings[0].matchedText); // jane@example.com

const result = scanAndTransform("Email jane@example.com", {
  transform: { default: { strategy: "redact" } },
});
console.log(result.text); // Email [EMAIL]
Rust example
use datafog_core::{
    scan, scan_and_transform, PrivacyError, ScanAndTransformConfig,
    TransformationConfig, TransformationStrategy,
};

fn main() -> Result<(), PrivacyError> {
    let findings = scan("Email jane@example.com");
    println!("{}", findings[0].entity_type); // EMAIL
    println!("{}", findings[0].matched_text); // jane@example.com

    let result = scan_and_transform(
        "Email jane@example.com",
        &ScanAndTransformConfig::new(TransformationConfig::new(
            TransformationStrategy::Redact,
        )),
    )?;
    println!("{}", result.text); // Email [EMAIL]
    Ok(())
}
Browser / WASM example
import { init, scan, scanAndTransform } from "@datafog/wasm";

await init();
console.log(scan("Email jane@example.com"));

const result = scanAndTransform("Email jane@example.com", {
  transform: { default: { strategy: "redact" } },
});
console.log(result.text); // Email [EMAIL]

Python and Node.js packages include prebuilt native binaries for Linux x64 and ARM64 (GNU), macOS Intel and Apple Silicon, and Windows x64 (MSVC).

Detection coverage

Coverage Entity types Activation
Personal and financial data EMAIL, PHONE, SSN, CREDIT_CARD, IP_ADDRESS, DATE, ZIP_CODE On by default
Credentials API_KEY, BEARER_TOKEN, JWT, PRIVATE_KEY, CREDENTIAL_URI On by default
Labeled street addresses STREET_ADDRESS On by default; bounded English street-line rule
Labeled accounts ACCOUNT_NUMBER On by default; bounded lexical rule
Labeled US identifiers US_ROUTING_NUMBER, NPI On by default; requires a supported text label
German identifiers IBAN, VAT ID, tax ID, social security number, postal code, passport number, residence permit number Explicit German locale
UUIDs UUID Explicit detect_uuid setting
Names in JSON fields PERSON Structured scans, using name-field aliases or explicit JSON Pointer mappings

Use structured scans to find names by JSON field, or local models to detect names and unlabelled street addresses in prose.

Each finding includes its entity type, matched text, byte and code-point ranges, optional confidence, and detector name/version. Ranges are zero-based and end-exclusive. Node.js and WASM also provide UTF-16 ranges for JavaScript slice. Rule-based findings have no confidence score. Findings and ranges →

For scanning files from AI agents, use datafog MCP.

Detection settings
  • Select checks: scan(text, {"entities": ["EMAIL", "SSN"]}) in Python runs only the selected text detectors. Omit entities to use all enabled detectors. Selections must be nonempty, unique, exact entity names. Selecting a German type or UUID does not activate it; set locale or detect_uuid too.
  • German locale: {"locale": "de"} enables the seven German detectors. These use format/context rules; IBAN checksums and account existence are not verified. See the German reference for detector rules.
  • ENV and SQL: choose format: "env" or format: "sql" to handle email boundaries in assignments and quoted strings. Source-context rules →
  • Phone numbers: numbers without + use North American checks: ten digits, optionally preceded by 1, with area and exchange codes starting with 2–9. Numbers starting with + use a permissive 7–15 digit check, including shortened examples such as +1-555-1002. Country codes and assigned numbers are not validated; numeric identifiers can still match.
  • Decimal numbers and CSV: contiguous card digits after a digit and . or , are suppressed, so 3.4111111111111111 and 3,4111111111111111 do not produce card findings. Cards containing spaces or hyphens still undergo normal checks. Parse CSV/TSV into cells before scanning so delimiters and adjacent fields do not affect matching.

Use capabilities() in Rust or Python to discover supported entities and their activation rules.

Transformations

Strategy Result Application provider Browser / WASM
redact Replace with a label such as [EMAIL] None Supported
mask Replace characters, optionally revealing the first or last characters None Supported
remove Delete the matched span None Supported
pseudonymize Create a stable keyed HMAC-SHA-256 pseudonym Key provider Unsupported
tokenize Issue an opaque token that can be restored under the same scope Token provider Unsupported

Use transform with existing findings, or scan_and_transform / scanAndTransform to scan and transform together. Transformation records include source/output ranges and replacement metadata, without the original matched text.

Keep scan settings under scan and transformation policy under transform. A default strategy is required; entity selection, per-type overrides, and allowlists are optional. Configuration guide → · Strategy details →

Browser workspace and FogVault integration

The unreleased v1.1 browser extension flow adds a bundled, default-enabled name/address CNN with a disable control, plus SQLite3 Multiple Ciphers as a second encrypted Workspace backend. See the candidate integration guide.

Core 1.0 provides persistent Workspace pseudonyms, explicit approved saves, strict/progressive restoration, operation history and conversation-scoped recall. The optional browser adapter uses the maintained FogVault Worker/WASM owner and IndexedDB storage. Native Rust, Node and Python owners use the same portable vault core through the registered native adapter.

Install the browser integration:

npm install --save-exact @datafog/wasm@1.0.0 @datafog/fogvault-browser@0.1.0 @datafog/core-fogvault-browser@1.0.0

Integration guide and supported profiles → · Stable Workspace contract → · Model assets and runtime setup → · SDK release scope and separate extension acceptance →

Local models (beta)

Core 1.0 includes optional local sessions for the datafog PII model and GLiNER2 in Rust, Python and Node.js. Model files are provisioned separately; model quality measurements remain separate from SDK correctness.

Install the exact 1.0.0 SDK:

python -m pip install 'datafog-core==1.0.0'
npm install --save-exact @datafog/node@1.0.0
cargo add datafog-core@=1.0.0 --features models

For GLiNER2, install its optional dependencies in the Python environment used by the runner:

python -m pip install 'datafog-core[gliner2]==1.0.0'

GLiNER2 supports the PII checkpoint and compatible general-purpose span models with caller-defined labels. The GLiNER2 runner ships in the Python package and also serves Node.js and Rust callers. Download and verify model files separately, then use ModelSession.open, text scan, and close. Reuse sessions across calls. Both backends return consistent results and errors, with cancellation and cleanup support.

datafog setup and session API → · GLiNER2 setup →

Documentation and development

For existing datafog Python applications, follow the migration guide for changes to imports, results, and transformations.

For Rust development, run:

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features

See the development guide for the repository layout and Python, Node.js, and WASM package checks.

License

MIT. Built by datafog.

Metadata

Release files for datafog-core 1.1.0rc1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datafog-core 1.1.0rc1
File Size Uploaded Provenance Actions
datafog_core-1.1.0rc1.tar.gz 1.3 MB With provenance Details for datafog_core-1.1.0rc1.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for datafog-core 1.1.0rc1
File Actions
datafog_core-1.1.0rc1-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for datafog_core-1.1.0rc1-cp310-abi3-win_amd64.whl
datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_x86_64.whl
datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_aarch64.whl
datafog_core-1.1.0rc1-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for datafog_core-1.1.0rc1-cp310-abi3-macosx_11_0_arm64.whl
datafog_core-1.1.0rc1-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for datafog_core-1.1.0rc1-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 13.0 MB

Release files / datafog_core-1.1.0rc1.tar.gz

Download URL datafog_core-1.1.0rc1.tar.gz
Size 1.3 MB
Tags Source
SHA-256 checksum
How to use checksums
2dce458d362b7f7c365648813b2a48a656289d98f05f6d6d37a92201e3625bd2
BLAKE2b-256 checksum
How to use checksums
9625b317ef9313c85d6948cb50f23e89e10c0b19ec6d5b5d3a3025d3052805f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / datafog_core-1.1.0rc1-cp310-abi3-win_amd64.whl

Download URL datafog_core-1.1.0rc1-cp310-abi3-win_amd64.whl
Size 2.1 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
f51614941912a4d5df5dbf6d187e23ee4cd6d9d64275ce3f55d0bc9708e70b1a
BLAKE2b-256 checksum
How to use checksums
8bf65435359d9a422d72de4680dfb1c46e63eea6026847de5c140bbe8d601486
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_x86_64.whl
Size 2.6 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
97ae07747fe249c26b2db51fe2c13ecc4ab5dea5d724be0aaf2d578b4211b92b
BLAKE2b-256 checksum
How to use checksums
be1c0f0f125b8316c15a7aebcc06ae58548c8e003e3d6f03e3aa9c58d3404794
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL datafog_core-1.1.0rc1-cp310-abi3-manylinux_2_28_aarch64.whl
Size 2.4 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
7f4a5e2788c2568ff980b463b5977c2857482300199e47acd05e8ab31f07e7cd
BLAKE2b-256 checksum
How to use checksums
2b78999f9edec730065e88852ccef80a8ca13e6213155d79e850310c3312c2cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / datafog_core-1.1.0rc1-cp310-abi3-macosx_11_0_arm64.whl

Download URL datafog_core-1.1.0rc1-cp310-abi3-macosx_11_0_arm64.whl
Size 2.2 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
132e1938805352ad3fe87de7080d9debf9f43e209f66e09f0cd0507ecae71df7
BLAKE2b-256 checksum
How to use checksums
12272eef88b5f5b53a1c80c5e234ca0248f8bab4615b1087d489e5112fbe504c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Release files / datafog_core-1.1.0rc1-cp310-abi3-macosx_10_12_x86_64.whl

Download URL datafog_core-1.1.0rc1-cp310-abi3-macosx_10_12_x86_64.whl
Size 2.4 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
d7ecefe7b88f377387ea423b44089245e9446e9b4138bb53f39c12965f8efed3
BLAKE2b-256 checksum
How to use checksums
6234a404c829bf8f80bb0bd705a6ccea45bea9cc9b0449a1f80161893713868a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

Security for datafog-core 1.1.0rc1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for datafog_core-1.1.0rc1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page