ocsfkit 0.9.0

An OCSF workbench for normalizing, linting, explaining, diffing, and querying security events.

ocsfkit

ocsfkit is an OCSF workbench for security engineers who need to normalize, lint, explain, diff, and query security events without losing sight of mapping quality.

Most OCSF migration work fails in the gaps between "the JSON parsed" and "the event is trustworthy." ocsfkit is built for those gaps. It shows what class an event became, which fields mapped cleanly, which values were defaulted or guessed, which source fields were dropped, and what is still missing.

Why Use It?

• Make mappings reviewable. Every mapped value carries provenance: source, transform, default, or guess.
• Catch silent data loss. Dropped and unmapped source fields are visible in explain, coverage, and strict mode.
• Ship safer pipelines. Lint, coverage budgets, SARIF, JUnit, GitHub annotations, and GitHub summaries fit naturally into CI.
• Share samples safely. Scan and redact fixtures before they leave your environment.
• Compare semantic changes. Diff OCSF events by field meaning instead of raw formatting noise.
• Start small, grow later. The built-in registry covers practical OCSF Detection Finding, Authentication, Network Activity, and Process Activity workflows, and can import or sync upstream schema data.

Contents

• Install
• Five Minute Tour
• Example Output
• Common Workflows
• Command Reference
• Mapping Files
• Built-In OCSF Scope
• Fixtures and Examples
• Production Use
• Development
• Release Automation

Install

pip install ocsfkit

Homebrew:

brew tap pfrederiksen/tap
brew install ocsfkit

From a checkout:

python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"

Five Minute Tour

Parse JSON, YAML, NDJSON, or stdin:

ocsfkit parse fixtures/aws_guardduty_finding.json
ocsfkit parse fixtures/guardduty.ndjson --format ndjson
cat fixtures/aws_guardduty_finding.json | ocsfkit parse -

Map a GuardDuty finding into an OCSF Detection Finding:

ocsfkit map fixtures/aws_guardduty_finding.json \
  --mapping examples/guardduty-mapping.yaml

Installed releases include built-in mapping packs, so common sources can be mapped without locating an example YAML file:

ocsfkit map fixtures/aws_guardduty_finding.json --pack aws-guardduty
ocsfkit explain fixtures/aws_guardduty_finding.json --pack aws-guardduty

Explain whether that mapping is good enough to trust:

ocsfkit explain fixtures/aws_guardduty_finding.json \
  --mapping examples/guardduty-mapping.yaml

Gate a stream in CI:

ocsfkit scorecard fixtures/guardduty.ndjson \
  --mapping examples/guardduty-mapping.yaml \
  --min-confidence 0.70 \
  --max-unmapped 10 \
  --github-summary

ocsfkit gate fixtures/guardduty.ndjson \
  --pack aws-guardduty \
  --min-confidence 0.70 \
  --max-unmapped 10 \
  --no-strict \
  --sarif > ocsfkit-gate.sarif

Lint OCSF-looking events:

ocsfkit lint fixtures/ocsf_detection_finding.json
ocsfkit lint fixtures/broken_ocsf_event.json --sarif

Scan and redact sample telemetry before sharing it:

ocsfkit scan fixtures --warn-only
ocsfkit redact fixtures/aws_guardduty_finding.json --output redacted.json

Query common OCSF fields:

ocsfkit query fixtures/ocsf_detection_finding.json metadata.product.name

Example Output

Mapping output

ocsfkit map fixtures/aws_guardduty_finding.json \
  --mapping examples/guardduty-mapping.yaml

Abbreviated output:

{
  "class_uid": 2004,
  "class_name": "Detection Finding",
  "category_uid": 2,
  "category_name": "Findings",
  "severity_id": 4,
  "severity": "High",
  "time": 1768386104000,
  "message": "EC2 instance i-0123456789abcdef0 communicating with suspicious host",
  "cloud": {
    "account_uid": "111122223333",
    "region": "us-east-1"
  },
  "metadata": {
    "product": {
      "name": "Amazon GuardDuty"
    },
    "version": "1.7.0"
  }
}

Explain output

ocsfkit explain fixtures/aws_guardduty_finding.json \
  --mapping examples/guardduty-mapping.yaml

Abbreviated output:

Confidence: 0.462
Target class: Detection Finding (class_uid 2004)

Mapped fields
  time              $.eventTime     parse_timestamp       1768386104000
  severity_id       $.severity      severity_text_to_id   4
  severity          $.severity                            "High"
  message           $.title                               "EC2 instance..."
  cloud.account_uid $.accountId                           "111122223333"
  cloud.region      $.region                              "us-east-1"

Defaulted fields
  class_uid              2004
  class_name             "Detection Finding"
  category_uid           2
  metadata.version       "1.7.0"
  metadata.product.name  "Amazon GuardDuty"

Dropped fields
  $.debug
  $.rawPayload

Unmapped source fields
  $.id
  $.resource.instanceDetails.instanceType
  $.service.action.awsApiCallAction.remoteIpDetails.ipAddressV4

The real terminal output uses Rich tables. JSON, Markdown, HTML, and GitHub annotation modes are available when the report needs to feed another tool.

Lint output

ocsfkit lint fixtures/broken_ocsf_event.json

Event 1 lint
  warning  message                Missing recommended field
  warning  metadata.product.name  Missing recommended field
  error    time                   Expected int, got str
  error    class_name             Expected 'Detection Finding' for class_uid 2004
  error    severity_id            Invalid severity_id

lint exits non-zero on errors unless --warn-only is set.

Scan and redact output

ocsfkit scan fixtures/aws_guardduty_finding.json --warn-only

account_id event[1].$.accountId: 1111...3333
ipv4 event[1].$.service.action.awsApiCallAction.remoteIpDetails.ipAddressV4: 198....100.10

ocsfkit redact fixtures/aws_guardduty_finding.json

{
  "accountId": "<redacted>",
  "service": {
    "action": {
      "awsApiCallAction": {
        "remoteIpDetails": {
          "ipAddressV4": "<redacted>"
        }
      }
    }
  }
}

Coverage output

ocsfkit coverage fixtures/guardduty.ndjson \
  --mapping examples/guardduty-mapping.yaml \
  --markdown

Scorecard output

ocsfkit scorecard fixtures/guardduty.ndjson \
  --mapping examples/guardduty-mapping.yaml \
  --min-confidence 0.70 \
  --max-unmapped 10

Grade: C
Passed: yes
Events: 2
Average confidence: 0.734
Source field coverage: 0.889
Unmapped source fields: 4
Missing target fields: 0
Lint errors: 0

## ocsfkit Coverage

- Events: 2
- Average confidence: 0.734
- Source field coverage: 0.889

### Top Unmapped Source Fields

- `$.resource`: 2
- `$.resource.instanceDetails`: 2

Common Workflows

Review a New Vendor Mapping

ocsfkit init-mapping fixtures/aws_guardduty_finding.json \
  --product-name "Amazon GuardDuty" > mapping-draft.yaml

ocsfkit explain fixtures/aws_guardduty_finding.json \
  --mapping mapping-draft.yaml \
  --markdown

ocsfkit validate-mapping mapping-draft.yaml --strict

Use this when onboarding a new log source. The generated mapping is a worksheet, not a finished answer. Review unmapped fields before production use.

Prevent Regressions in CI

ocsfkit lint fixtures/ocsf_detection_finding.json --sarif
ocsfkit scan fixtures --sarif --warn-only
ocsfkit schema-drift examples/guardduty-mapping.yaml --sarif

ocsfkit coverage fixtures/guardduty.ndjson \
  --mapping examples/guardduty-mapping.yaml \
  --min-confidence 0.80 \
  --max-unmapped 25 \
  --github-summary

ocsfkit coverage fixtures/guardduty.ndjson \
  --pack aws-guardduty \
  --min-confidence 0.80 \
  --max-unmapped 25 \
  --sarif > ocsfkit-coverage.sarif

ocsfkit scorecard fixtures/guardduty.ndjson \
  --mapping examples/guardduty-mapping.yaml \
  --min-confidence 0.80 \
  --max-unmapped 25 \
  --github-summary

This catches missing OCSF fields, invalid types, falling confidence, and newly unmapped vendor fields.

For test-report dashboards, emit JUnit from golden mapping tests:

ocsfkit test-mapping tests/goldens --junit ocsfkit-mapping.xml

Compare Mapping Versions

ocsfkit map sample.json --mapping mapping-v1.yaml > before.json
ocsfkit map sample.json --mapping mapping-v2.yaml > after.json
ocsfkit diff before.json after.json

This highlights field-level OCSF changes, including class and severity changes that can affect routing, alerting, and dashboards.

Explore Supported Targets

ocsfkit targets search user
ocsfkit targets complete actor.user
ocsfkit targets show actor.user.name
ocsfkit pack list
ocsfkit pack validate

Use target discovery and mapping packs when building mappings for common source families such as AWS, identity, network, detections, and infrastructure. Pack aliases can be passed to commands that accept mappings:

ocsfkit map sample.json --pack aws-guardduty --format ndjson
ocsfkit gate sample.ndjson --pack aws-guardduty --no-strict --sarif > ocsfkit-gate.sarif

Command Reference

Command	Purpose
parse <input>	Load JSON, YAML, NDJSON, or stdin and emit normalized JSON.
map <input> --mapping mapping.yaml	Apply a mapping and emit OCSF JSON.
explain <input> --mapping mapping.yaml	Show mapping decisions, dropped fields, unmapped fields, missing targets, and confidence.
lint <input>	Validate OCSF-looking events against the bundled registry.
diff <before> <after>	Compare two OCSF events or same-length event streams.
diff-mapping <before> <after>	Compare two mapping YAML files semantically.
query <input> <path>	Extract common OCSF fields with dotted paths.
coverage <input> --mapping mapping.yaml	Summarize mapping quality across a stream and enforce quality budgets.
scorecard <input> --mapping mapping.yaml	Grade mapping readiness with coverage, lint, and strict checks.
gate <input> --mapping mapping.yaml	Run a stricter production-readiness gate with JSON or SARIF output.
validate-mapping mapping.yaml	Check mapping syntax, transforms, and likely schema issues.
schema-drift mapping.yaml	Compare a mapping against bundled or synced schema data.
scan <input>	Find likely secrets and sensitive identifiers in fixtures or reports.
redact <input>	Redact likely secrets while preserving event structure.
catalog	Generate Markdown or JSON docs from mapping YAML files.
doctor	Check local install health, schema support, and mapping pack validity.
benchmark <input> --mapping mapping.yaml	Measure mapping throughput on a representative corpus.
init-mapping <input>	Generate a starter mapping worksheet from a representative event.
test-mapping spec.yaml	Run fixture-based mapping regression tests.
test-transform spec.yaml	Run YAML-defined tests for built-in or custom transforms.
report <input> --mapping mapping.yaml	Write a standalone HTML mapping coverage report.
workshop <input>	Print a guided mapping worksheet and optional explanation report.
schema	Print the bundled minimal OCSF registry.
import-schema <path>	Convert upstream-style schema files into the compact registry format.
sync-schema --output schema.json	Download upstream OCSF schema data and import it.
targets list/search/show	Discover known OCSF target fields.
pack list/validate	Inspect and validate included mapping packs.

Useful output modes:

ocsfkit --version
ocsfkit explain sample.json --mapping mapping.yaml --json
ocsfkit explain sample.json --mapping mapping.yaml --markdown
ocsfkit explain sample.json --mapping mapping.yaml --html --output explanation.html
ocsfkit lint sample.json --github-annotations
ocsfkit scan fixtures --sarif --warn-only
ocsfkit coverage sample.ndjson --mapping mapping.yaml --sarif
ocsfkit scorecard sample.ndjson --mapping mapping.yaml --sarif
ocsfkit coverage sample.ndjson --mapping mapping.yaml --github-summary
ocsfkit scorecard sample.ndjson --mapping mapping.yaml --markdown
ocsfkit schema --format jsonschema > ocsfkit.schema.json
ocsfkit catalog --output docs/mapping-catalog.md
ocsfkit test-mapping tests/goldens --junit mapping-tests.xml

Strict mode is available on mapping-quality commands:

ocsfkit map sample.json --mapping mapping.yaml --strict
ocsfkit explain sample.json --mapping mapping.yaml --strict
ocsfkit coverage sample.ndjson --mapping mapping.yaml --strict
ocsfkit validate-mapping mapping.yaml --strict
ocsfkit schema-drift mapping.yaml

Strict mode fails on guessed fields, missing targets, and unmapped source fields. Python custom_transforms are blocked in strict mode unless --allow-unsafe-transforms is explicitly provided. Mappings that use Python transforms should also include custom_transforms_trusted: true after the transform module has been reviewed.

Mapping Files

Mappings are YAML. Source paths use a deliberate JSONPath subset, and target paths use dotted OCSF paths.

schema_version: 1.7.0
ocsf_version: 1.7.0
requires_ocsfkit: ">=0.7.0,<1.0.0"

target_class:
  class_uid: 2004
  class_name: Detection Finding
  category_uid: 2
  category_name: Findings

fields:
  time:
    from: $.eventTime
    transform: parse_timestamp
    required: true

  severity_id:
    from: $.severity
    transform: severity_text_to_id
    default: 1

  message:
    from: $.title

  cloud.account_uid:
    from: $.accountId

  actor.user.name:
    from: $.userIdentity.userName

  resources[]:
    foreach:
      from: $.resources
      fields:
        name:
          from: $.id
        type:
          from: $.type

drop:
  - $.debug
  - $.rawPayload

Supported source path examples:

• $.eventTime
• $.Resources[*].Id
• $.items[0].name
• $.items[?type==instance].id

The mapping engine tracks whether every target value came from a source field, a transform, a default, or a guess. Unknown source fields that are not mapped or explicitly dropped are reported as unmapped. Use foreach for repeated source objects that should become repeated OCSF objects such as resources[].

Built-in transforms include OCSF helpers and vendor-oriented transform packs:

• parse_timestamp
• severity_text_to_id
• aws.severity
• azure.status_id
• azure.status
• okta.status_id
• okta.status
• network.activity_id

Python packages can also expose reviewed transforms through the ocsfkit.transforms entry point group. Use entry points for reusable transforms and reserve custom_transforms.py for local migration work.

See the Mapping Guide for advanced examples, provenance details, custom transforms, and strict-mode guidance.

Built-In OCSF Scope

ocsfkit intentionally starts with a practical minimal registry. It currently focuses on these classes:

• Detection Finding (class_uid: 2004)
• Authentication (class_uid: 3002)
• Network Activity (class_uid: 4001)
• Process Activity (class_uid: 1007)

Common fields include:

• Base event fields: time, class_uid, class_name, category_uid, category_name, activity_id, activity_name, type_uid, type_name, severity_id, severity, message
• Metadata: metadata.version, metadata.product.name
• Identity and cloud: actor.user.name, actor.user.uid, cloud.account_uid, cloud.region
• Endpoint and process: device.hostname, src_endpoint.ip, src_endpoint.port, dst_endpoint.ip, dst_endpoint.port, process.name, process.pid
• Resources and status: resources[].name, resources[].type, status, status_id

Schema-version awareness currently supports 1.6.0 and 1.7.0, with 1.7.0 as the default expected version. Use ocsfkit schema --format jsonschema to export a JSON Schema for editors and external validators.

Fixtures and Examples

The repository includes fake but realistic fixtures. They are designed for tests, demos, mapping review, and documentation. No fixture contains real secrets or real account IDs.

Included source fixtures cover:

• AWS GuardDuty
• AWS Security Hub
• AWS CloudTrail
• AWS VPC Flow Logs
• Azure AD sign-in
• Azure Activity Logs
• Okta login
• GitHub Audit Log
• Google Cloud Audit Logs
• CrowdStrike detection
• Palo Alto traffic
• Zeek connection logs
• Splunk ES notable events
• Microsoft Sentinel alerts
• Microsoft Defender alerts
• Wiz findings
• Lacework alerts
• GCP Security Command Center findings
• Cloudflare logs
• Kubernetes audit events
• Microsoft Sysmon process events
• Windows Security authentication events

Important files:

• GuardDuty mapping
• Security Hub mapping
• CloudTrail console login mapping
• AWS VPC Flow Logs mapping
• Google Cloud Audit mapping
• Sysmon process mapping
• Windows Security authentication mapping
• Custom transform module
• OCSF Detection Finding fixture
• Broken OCSF fixture
• GuardDuty NDJSON fixture

More workflow documentation:

• Getting Started
• Install Guide
• CI Examples
• Mapping Guide
• Mapping Catalog
• Real-World Workflows

Production Use

ocsfkit includes supporting files for common production paths:

• Dockerfile for containerized CI or build-agent usage.
• .github/workflows/docker.yml for GHCR image builds on pushes and tagged releases.
• .pre-commit-hooks.yaml for validating mappings before commit.
• ocsfkit scorecard and ocsfkit gate for pass/fail readiness gates.
• ocsfkit catalog for generated mapping documentation.
• ocsfkit schema-drift for checking mappings against bundled or synced schema data.
• ocsfkit scan, ocsfkit redact, and SARIF output for fixture hygiene and code-scanning integrations.
• ocsfkit doctor and ocsfkit benchmark for release readiness and throughput checks.
• MkDocs configuration for publishing a navigable documentation site from docs/.

See the Install Guide for pipx, uvx, pip, Homebrew, Docker, GitHub Actions, and pre-commit examples.

Development

uv run --extra dev pytest
uv run --extra dev ruff check .
uv build

The CLI entry point is:

ocsfkit = "ocsfkit.cli:app"

Release Automation

The repository is configured for normal Python and Homebrew releases:

1. Tag a version, for example git tag v0.9.0 && git push --tags.
2. .github/workflows/release.yml builds source and wheel distributions.
3. PyPI publishing uses Trusted Publishing.
4. Homebrew tap updates run when HOMEBREW_TAP_ENABLED=true is set and HOMEBREW_TAP_TOKEN is available. The release workflow hashes the GitHub release archive before committing the formula update.
5. GitHub Actions are pinned to commit SHAs.
6. Release artifacts get GitHub provenance attestations.

Do not commit package index tokens. Use PyPI Trusted Publishing and scoped repository secrets for release automation.