DataFog Core

Find and transform sensitive data in your application.

Quickstart · Detection · Transformations · Local models · Documentation

DataFog Core is a Rust library with Python, Node.js, and browser WebAssembly bindings. Scan text or JSON for personal data and credentials, then redact, mask, remove, pseudonymize, or tokenize matched values. Findings use a consistent format across the bindings.

Rule-based scanning runs locally without model weights or downloads. Optional local model sessions add DataFog and GLiNER2 support in Rust, Python, and Node.js.

Stable: 0.4.4 · Model beta: 0.4.5-beta.4

Quickstart

Choose the package for your application:

Python

from datafog_core import scan, scan_and_transform

findings = scan("Email jane@example.com")
print(findings[0].entity_type)   # EMAIL
print(findings[0].matched_text)  # jane@example.com

result = scan_and_transform(
    "Email jane@example.com",
    {"transform": {"default": {"strategy": "redact"}}},
)
print(result.text)  # Email [EMAIL]

import { scan, scanAndTransform } from "@datafog/node";

const findings = scan("Email jane@example.com");
console.log(findings[0].entityType); // EMAIL
console.log(findings[0].matchedText); // jane@example.com

const result = scanAndTransform("Email jane@example.com", {
  transform: { default: { strategy: "redact" } },
});
console.log(result.text); // Email [EMAIL]

use datafog_core::{
    scan, scan_and_transform, PrivacyError, ScanAndTransformConfig,
    TransformationConfig, TransformationStrategy,
};

fn main() -> Result<(), PrivacyError> {
    let findings = scan("Email jane@example.com");
    println!("{}", findings[0].entity_type); // EMAIL
    println!("{}", findings[0].matched_text); // jane@example.com

    let result = scan_and_transform(
        "Email jane@example.com",
        &ScanAndTransformConfig::new(TransformationConfig::new(
            TransformationStrategy::Redact,
        )),
    )?;
    println!("{}", result.text); // Email [EMAIL]
    Ok(())
}

import { init, scan, scanAndTransform } from "@datafog/wasm";

await init();
console.log(scan("Email jane@example.com"));

const result = scanAndTransform("Email jane@example.com", {
  transform: { default: { strategy: "redact" } },
});
console.log(result.text); // Email [EMAIL]

Python and Node.js packages include prebuilt native binaries for Linux x64 and ARM64 (GNU), macOS Intel and Apple Silicon, and Windows x64 (MSVC).

Detection coverage

Use structured scans to find names by JSON field, or local models to detect names and street addresses in prose.

Each finding includes its entity type, matched text, byte and code-point ranges, optional confidence, and detector name/version. Ranges are zero-based and end-exclusive. Node.js and WASM also provide UTF-16 ranges for JavaScript slice. Rule-based findings have no confidence score. Findings and ranges →

For scanning files from AI agents, use DataFog MCP.

Transformations

Use transform with existing findings, or scan_and_transform / scanAndTransform to scan and transform together. Transformation records include source/output ranges and replacement metadata, without the original matched text.

Keep scan settings under scan and transformation policy under transform. A default strategy is required; entity selection, per-type overrides, and allowlists are optional. Configuration guide → · Strategy details →

Local models (beta)

0.4.5-beta.4 adds local sessions for the DataFog PII model and GLiNER2 in Rust, Python, and Node.js on all five native platforms.

Install the beta for your runtime:

python -m pip install 'datafog-core==0.4.5b4'
npm install @datafog/node@beta
cargo add datafog-core@0.4.5-beta.4 --features models

For GLiNER2, install its optional dependencies in the Python environment used by the runner:

python -m pip install 'datafog-core[gliner2]==0.4.5b4'

GLiNER2 supports the PII checkpoint and compatible general-purpose span models with caller-defined labels. The GLiNER2 runner ships in the Python package and also serves Node.js and Rust callers. Download and verify model files separately, then use ModelSession.open, text scan, and close. Reuse sessions across calls. Both backends return consistent results and errors, with cancellation and cleanup support.

DataFog setup and session API → · GLiNER2 setup →

Documentation and development

• Installation and runtime requirements
• Python · Node.js · Rust · Browser / WASM API references
• Structured JSON scans and name-field discovery
• Configuration and allowlists
• Errors and 0.4.x compatibility
• Build, test, and contribute
• Release notes

For existing datafog Python applications, follow the migration guide for changes to imports, results, and transformations.

For Rust development, run:

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features

See the development guide for the repository layout and Python, Node.js, and WASM package checks.

License

MIT. Built by DataFog.