elasticsearch-pii-redacter 1.11.0

Redacting field data from your Elasticsearch indices and Searchable Snapshots

Did you find PII (Personally Identifiable Information) in your Elasticsearch indices that doesn’t belong there? This is the tool for you!

The elastic-pii-redacter can help you redact information from even Searchable Snapshot mounted indices. It works with deeply nested fields, too!

Client Configuration

The tool connects using the es_client Python module.

You can use command-line options, or a YAML configuration file to configure the client connection. If using a configuration file is desired, the configuration file structure requires elasticsearch at the root level as follows:

---
elasticsearch:
  client:
    hosts: https://10.11.12.13:9200
    cloud_id:
    request_timeout: 60
    verify_certs:
    ca_certs:
    client_cert:
    client_key:
  other_settings:
    username:
    password:
    api_key:
      id:
      api_key:
      token:

logging:
  loglevel: INFO
  logfile: /path/to/file.log
  logformat: default
  blacklist: []

REDACTIONS_FILE Configuration

NOTE: If, under forcemerge, only_expunge_deletes is True, any configured value for max_num_segments will be ignored, and only documents marked for delete will be cleared. It is important to note this distinction as the default behavior is to merge to 1 segment per shard.

---
redactions:
  - job_name_20240506_redact_hot:
      pattern: hot-*
      query: {'match': {'message': 'message1'}}
      fields: ['message']
      message: REDACTED
      expected_docs: 1
      restore_settings: {'index.routing.allocation.include._tier_preference': 'data_warm,data_hot,data_content'}
  - job_name_20240506_redact_cold:
      pattern: restored-cold-*
      query: {'match': {'nested.key': 'nested19'}}
      fields: ['nested.key']
      message: REDACTED
      expected_docs: 1
      restore_settings: {'index.routing.allocation.include._tier_preference': 'data_warm,data_hot,data_content'}
      forcemerge:
        max_num_segments: 1
  - job_name_20240506_redact_frozen:
      pattern: partial-frozen-*
      query: {'range': {'number': {'gte': 8, 'lte': 11}}}
      fields: ['deep.l1.l2.l3']
      message: REDACTED
      expected_docs: 4
      forcemerge:
        only_expunge_deletes: True