Status: experimental, work in progress:

• Although cidrmatch is generated, you still need to provide corresponding function as UDF (I'll add example later)
• Requires more testing

pySigma Databricks Backend

This is the Databricks backend for pySigma. It provides the package sigma.backends.databricks with the DatabricksBackend class. Further, it contains the following processing pipelines in sigma.pipelines.databricks:

• snake_case: convert column names into snake case format

It supports the following output formats:

• default: plain Databricks/Apache Spark SQL queries
• dbsql: Databricks SQL queries with rules metadata (title, status) embedded as comment
• detection_yaml: Yaml markup for my own detection framework
• lakewatch: Lakewatch detection rules (kind: Rule YAML), batch by default

Unbound Keyword Search

The backend supports Sigma rules with unbound keywords (values without field names). These keywords search the raw log line.

Configuration

By default, the backend looks for keywords in a field named raw. You can customize this:

Command Line:

sigma convert -t databricks -O raw_log_field=message rule.yml

Programmatic:

from sigma.backends.databricks import DatabricksBackend

backend = DatabricksBackend(raw_log_field="event_data")

Examples

Simple Keywords (OR logic):

detection:
    keywords:
        - 'EVILSERVICE'
        - 'svchost.exe -n evil'
    condition: keywords

Generates: contains(lower(raw), lower('EVILSERVICE')) OR contains(lower(raw), lower('svchost.exe -n evil'))

Keywords with |all (AND logic):

detection:
    keywords:
        '|all':
            - 'Remove-MailboxExportRequest'
            - ' -Identity '
    condition: keywords

Generates: contains(lower(raw), lower('Remove-MailboxExportRequest')) AND contains(lower(raw), lower(' -Identity '))

Mixed with Field Conditions:

detection:
    selection:
        EventID: 4688
    keywords:
        - 'mimikatz'
    condition: selection and keywords

Generates: EventID = 4688 AND contains(lower(raw), lower('mimikatz'))

Wildcards in Keywords:

detection:
    keywords:
        - '*malware*'      # uses contains()
        - 'cmd.exe*'       # uses startswith()
        - '*.dll'          # uses endswith()
    condition: keywords

Regex Patterns:

detection:
    keywords:
        - '|re': '.*evil(cmd|powershell).*'
    condition: keywords

Generates: raw rlike '.*evil(cmd|powershell).*'

Lakewatch Output

Generate Lakewatch detection rules (kind: Rule YAML) from Sigma rules. Intended to run after the OCSF pipeline, which normalizes fields and injects the OCSF class, so the backend can pick the gold table.

sigma convert -t databricks -f lakewatch -p ocsf rules/ -o detections.yaml

Multiple rules are emitted as separate --- documents in one file.

Options (-O key=value)

sigma convert -t databricks -f lakewatch -p ocsf \
  -O table_name=main.security.authentication -O lookback="1 DAY" -O schedule=12h rule.yml

Sigma metadata maps to Lakewatch as: title→displayName, description→comment/objective, level→severity, status→fidelity, and attack.* tags→mitre. Only batch rules are generated (streaming is future work).

Maintainer

This backend is currently maintained by:

• Alex Ott