ls-keystore-utils

Wrapper library to make it easier to create and modify Logstash Secrets keystores for secure settings.

Problem Solved

Logstash keystores are encrypted PKCS#12 files used to securely store sensitive configuration values, such as passwords or API keys, for Logstash pipelines. Traditionally, managing these keystores requires manual use of the logstash-keystore CLI binary, which is slow because of the need to spin up a JVM, however small, and lacks programmatic integration. The LogstashKeystore class partly solves this by providing a Python-based interface for creating, reading, updating, and deleting keystore entries programmatically, while ensuring compatibility with Logstash's proprietary format.

How It Works

The LogstashKeystore class actually still leverages the official logstash-keystore binary (typically located at /usr/share/logstash/bin/logstash-keystore or best-guess auto-detected) for all write operations (adding, removing, or updating keys) to maintain full compatibility and security. For read operations, it uses custom cryptographic parsing via the cryptography and asn1crypto modules to efficiently retrieve key values without relying on the binary (which does not reveal key values at all). Unfortunately, these libraries do not appear to be able to write new keys to a logstash-keystore generated file. But it is still quite useful to be able to read the values and cache them in an AES-obfuscated memory store.

To reduce redundant calls to the logstash-keystore binary and minimize disk I/O, the class caches keystore data in memory, including obfuscated values and timestamps. Before each operation, it checks for external modifications by comparing cached timestamps with the keystore file's latest timestamp, and/or detecting additions, removals, or value changes. This caching mechanism ensures data consistency, prevents silent overwrites, and allows for efficient batch operations pass multiple keys in a single CLI call with serialized input. With value obfuscated value caching, it can reduce the need to overwrite existing values or execute a full Logstash pipeline just to see if the value is correct, or needs updating.

Installation

pip install -e .

or

pip install ls-keystore-utils

Usage

Instantiation

Create a new keystore or load an existing one. The automatic detection of the path.settings and location of the logstash-keystore binary path are attempted if not provided. If not specified, path_settings will be guessed as /etc/logstash, or /opt/homebrew/etc/logstash, and exepath will be guessed as /usr/share/logstash/bin/logstash-keystore or look for /opt/homebrew/Cellar/logstash/*/libexec/bin/logstash-keystore.

If you don't provide a value for salt_iv, one will be automatically generated to help obfuscate values in memory. If you provide an ObfuscatedValue object for obvpassword, you must provide a value for salt_iv that will de-obfuscate it. LogstashKeystore will continue to use that salt_iv value to obfuscate the values read from the Logstash Secret store file.

from ls_keystore_utils import LogstashKeystore, ObfuscatedValue, generate_salt_iv

# PREFERRED
# Create a new keystore with an obfuscated password
salt_iv = generate_salt_iv()
obv = ObfuscatedValue("mypassword", salt_iv)
ks = LogstashKeystore.create(
    path_settings="/path/to/logstash/config",
    exepath="/path/to/bin/logstash-keystore",
    salt_iv=salt_iv,
    obvpassword=obv
) 

# Create a new keystore with a plain text password
ks = LogstashKeystore.create(password="mypassword")

# PREFERRED
# Load an existing keystore using an obfuscated password
salt_iv = generate_salt_iv()
obv = ObfuscatedValue("mypassword", salt_iv)
ks = LogstashKeystore.load(
    path_settings="/path/to/logstash/config",
    exepath="/path/to/bin/logstash-keystore",
    salt_iv=salt_iv,
    obvpassword=obv
)

Adding Keys

Add single or multiple keys. Batch mode is more efficient as it uses one CLI call for all keys.

# Add a single key
ks.add_key("MY_KEY", "secret_value")

# Add multiple keys in batch mode (must be dict of key/value pairs)
ks.add_key({"KEY1": "value1", "KEY2": "value2"})

Removing Keys

Remove single or multiple keys. Batch mode removes all keys in one CLI call.

# Remove a single key
ks.remove_key("MY_KEY")

# Remove multiple keys in batch mode
ks.remove_key(["KEY1", "KEY2"])

Updating Keys

Update existing keys (overwrites values). Same as adding; batch mode supported.

# Update a single key
ks.update_key("EXISTING_KEY", "new_value")

# Update multiple keys in batch mode
ks.update_key({"KEY1": "new_val1", "KEY2": "new_val2"})

Deleting Keys

Delete keys (alias for removing). Supports single and batch modes like removal.

# Delete a single key
ks.delete_key("MY_KEY")

# Delete multiple keys in batch mode
ks.delete_key(["KEY1", "KEY2"])

Compare value to keystore value

More information on the ObfuscatedValue class may be found in the code in crypto.py or in future documentation.

The ObfuscatedValue class has an equality method __eq__() that can compare between other ObfuscatedValue class objects, or just encrypted similarly with the same salt_iv value. With the salt_iv value, it's also possible to reveal the unobfuscated value.

# Compare two ObfuscatedValue objects
>>> secret = "my_secret_value"
>>> salt_iv = b'\\x00' * 32
>>> ov1 = ObfuscatedValue(secret, salt_iv)
>>> ov2 = ObfuscatedValue(secret, salt_iv)
>>> assert ov1 == ov2
True

# Compare an ObfuscatedValue object with an obfuscated value
>>> from ls_keystore_utils.crypto import obfuscate_value
>>> secret = "my_secret_value"
>>> salt_iv = b'\\x00' * 32
>>> ov1 = ObfuscatedValue(secret, salt_iv)
>>> obfuscated = obfuscate_value(secret, salt_iv)
>>> assert ov1 == obfuscated
True

# Extract the unobfuscated value from the cache
# Assume that we've already instantiated ks as a LogstashKeystore object and that
# it has extracted MY_KEY with secret from an existing keystore file, and that the
# salt_iv value used to obfuscate it is 32 empty bytes.
>>> key = "MY_KEY"
>>> secret = "my_secret_value"
>>> salt_iv = b'\\x00' * 32
>>> assert secret == ks._current[key].obfuscated_value.reveal(salt_iv)
True

Features

• Create/Update/Delete: Full CRUD operations via logstash-keystore binary with batch support for efficiency.
• Read Operations: Retrieve individual or all key values using cached, parsed data.
• List Keys: Get a list of all key names in the keystore.
• Validation: Verify keystore integrity and detect external modifications.
• Backups: Create file backups of the keystore.
• Metadata: Track timestamps for change detection and age monitoring.
• Change Tracking: Automatically detects modifications and flags when Logstash restart is required.
• Caching: In-memory cache reduces redundant binary calls and ensures consistency.

Change Tracking

If a key has been updated with a new value, Logstash might need to be restarted to get that value since it is converted to an environment variable. If this is the case, then the LogstashKeystore class will set property needs_restart to True.

if ks.needs_restart is True:
    # do stuff here