Logprep allows to collect, process and forward log messages from various data sources.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for djkhl from gravatar.com djkhl Avatar for ekneg54 from gravatar.com ekneg54 Avatar for kaya-david from gravatar.com kaya-david Avatar for mhoff from gravatar.com mhoff Avatar for ppcad from gravatar.com ppcad

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: GNU Lesser General Public License v2 or later (LGPLv2+) (GNU LESSER GENERAL PUBLIC LICENSE)
  • Tags kafka , etl , sre , preprocessing , opensearch , soar , logdata
  • Requires: Python >=3.11
  • Provides-Extra: dev , doc
Classifiers
Report project as malware

Project description

Logprep

GitHub release (latest by date) GitHub Workflow Status (branch) Documentation Status GitHub contributors Coverage GitHub Repo stars

Introduction

Logprep allows to collect, process and forward log messages from various data sources. Log messages are being read and written by so-called connectors. Currently, connectors for Kafka, Opensearch, S3, HTTP and JSON(L) files exist.

The log messages are processed in serial by a pipeline of processors, where each processor modifies an event that is being passed through. The main idea is that each processor performs a simple task that is easy to carry out. Once the log message is passed through all processors in the pipeline the resulting message is sent to a configured output connector.

Logprep is primarily designed to process log messages. Generally, Logprep can handle JSON messages, allowing further applications besides log handling.

About Logprep

Pipelines

Logprep processes incoming log messages with a configured pipeline that can be spawned multiple times via multiprocessing. The following chart shows a basic setup that represents this behaviour. The pipeline consists of three processors: the Dissector, Geo-IP Enricher and the Dropper. Each pipeline runs concurrently and takes one event from it's Input Connector. Once the log messages is fully processed the result will be forwarded to the Output Connector, after which the pipeline will take the next message, repeating the processing cycle.

flowchart LR
A1[Input\nConnector] --> B
A2[Input\nConnector] --> C
A3[Input\nConnector] --> D
subgraph Pipeline 1
B[Dissector] --> E[Geo-IP Enricher]
E --> F[Dropper]
end
subgraph Pipeline 2
C[Dissector] --> G[Geo-IP Enricher]
G --> H[Dropper]
end
subgraph Pipeline n
D[Dissector] --> I[Geo-IP Enricher]
I --> J[Dropper]
end
F --> K1[Output\nConnector]
H --> K2[Output\nConnector]
J --> K3[Output\nConnector]

Processors

Every processor has one simple task to fulfill. For example, the Dissector can split up long message fields into multiple subfields to facilitate structural normalization. The Geo-IP Enricher, for example, takes an ip-address and adds the geolocation of it to the log message, based on a configured geo-ip database. Or the Dropper deletes fields from the log message.

As detailed overview of all processors can be found in the processor documentation.

To influence the behaviour of those processors, each can be configured with a set of rules. These rules define two things. Firstly, they specify when the processor should process a log message and secondly they specify how to process the message. For example which fields should be deleted or to which IP-address the geolocation should be retrieved.

Connectors

Connectors are responsible for reading the input and writing the result to a desired output. The main connectors that are currently used and implemented are a kafka-input-connector and a kafka-output-connector allowing to receive messages from a kafka-topic and write messages into a kafka-topic. Addionally, you can use the Opensearch or Opensearch output connectors to ship the messages directly to Opensearch or Opensearch after processing.

The details regarding the connectors can be found in the input connector documentation and output connector documentation.

Configuration

To run Logprep, certain configurations have to be provided. Because Logprep is designed to run in a containerized environment like Kubernetes, these configurations can be provided via the filesystem or http. By providing the configuration via http, it is possible to control the configuration change via a flexible http api. This enables Logprep to quickly adapt to changes in your environment.

First, a general configuration is given that describes the pipeline and the connectors, and lastly, the processors need rules in order to process messages correctly.

The following yaml configuration shows an example configuration for the pipeline shown in the graph above:

process_count: 3
timeout: 0.1

pipeline:
  - dissector:
      type: dissector
      rules:
        - https://your-api/dissector/
        - rules/01_dissector/rules/
  - geoip_enricher:
      type: geoip_enricher
      rules:
        - https://your-api/geoip/
        - rules/02_geoip_enricher/rules/
      tree_config: artifacts/tree_config.json
      db_path: artifacts/GeoDB.mmdb
  - dropper:
      type: dropper
      rules:
        - rules/03_dropper/rules/

input:
  mykafka:
    type: confluentkafka_input
    bootstrapservers: [127.0.0.1:9092]
    topic: consumer
    group: cgroup
    auto_commit: true
    session_timeout: 6000
    offset_reset_policy: smallest
output:
  opensearch:
    type: opensearch_output
    hosts:
        - 127.0.0.1:9200
    default_index: default_index
    error_index: error_index
    message_backlog_size: 10000
    timeout: 10000
    max_retries:
    user: the username
    secret: the passord
    cert: /path/to/cert.crt

The following yaml represents a dropper rule which according to the previous configuration should be in the rules/03_dropper/rules/ directory.

filter: "message"
drop:
  - message
description: "Drops the message field"

The condition of this rule would check if the field message exists in the log. If it does exist then the dropper would delete this field from the log message.

Details about the rule language and how to write rules for the processors can be found in the rule configuration documentation.

Documentation

The documentation for Logprep is online at https://logprep.readthedocs.io/en/latest/ or it can be built locally via:

sudo apt install pandoc
pip install -e .[doc]
cd ./doc/
make html

A HTML documentation can be then found in doc/_build/html/index.html.

Container signatures

From release 15 on, Logprep containers are signed using the cosign tool. To verify the container, you can copy the following public key into a file logprep.pub:

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEgkQXDi/N4TDFE2Ao0pulOFfbGm5g
kVtARE+LJfSFI25BanOG9jaxxRGVt+Sa1KtQbMcy7Glxu0s7XgD9VFGjTA==
-----END PUBLIC KEY-----

And use it to verify the signature:

cosign verify --key logprep.pub ghcr.io/fkie-cad/logprep:py3.11-latest

The output should look like:

Verification for ghcr.io/fkie-cad/logprep:py3.11-latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"ghcr.io/fkie-cad/logprep"}, ...

Container SBOM

From release 15 on, Logprep container images are shipped with a generated sbom. To verify the attestation and extract the SBOM use cosign with:

cosign verify-attestation --key logprep.pub ghcr.io/fkie-cad/logprep:py3.11-latest | jq '.payload | @base64d | fromjson | .predicate | .Data | fromjson' > sbom.json

The output should look like:

Verification for ghcr.io/fkie-cad/logprep:py3.11-latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

Finally, you can view the extracted sbom with:

cat sbom.json | jq

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for djkhl from gravatar.com djkhl Avatar for ekneg54 from gravatar.com ekneg54 Avatar for kaya-david from gravatar.com kaya-david Avatar for mhoff from gravatar.com mhoff Avatar for ppcad from gravatar.com ppcad

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: GNU Lesser General Public License v2 or later (LGPLv2+) (GNU LESSER GENERAL PUBLIC LICENSE)
  • Tags kafka , etl , sre , preprocessing , opensearch , soar , logdata
  • Requires: Python >=3.11
  • Provides-Extra: dev , doc
Classifiers

Release history Release notifications | RSS feed

This version

18.0.1

18.0.0

17.0.3

17.0.2

17.0.1

17.0.0

16.1.0

16.0.0

15.1.1

15.1.0

15.0.0

14.0.0

13.1.2

13.1.1

13.1.0

13.0.1

13.0.0

12.0.0

11.3.0

11.2.1

11.2.0

11.1.0

11.0.1

11.0.0

10.3 yanked

10.0.4

10.0.3

10.0.2

10.0.1

10.0.0

9.0.3

9.0.2

9.0.1

9.0.0

8.0.0

7.0.0

6.8.1

6.8.0

6.7.0

6.6.0

6.5.1

6.5.0

6.4.0

6.3.0

6.2.0

6.1.0

6.0.0

5.0.1

4.0.0

3.3.0

3.2.0

3.1.0

3.0.0

2.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

logprep-18.0.1.tar.gz (3.4 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

logprep-18.0.1-cp314-cp314-musllinux_1_2_x86_64.whl (908.6 kB view details)

Uploaded CPython 3.14musllinux: musl 1.2+ x86-64

logprep-18.0.1-cp314-cp314-manylinux_2_28_x86_64.whl (836.8 kB view details)

Uploaded CPython 3.14manylinux: glibc 2.28+ x86-64

logprep-18.0.1-cp313-cp313-musllinux_1_2_x86_64.whl (908.4 kB view details)

Uploaded CPython 3.13musllinux: musl 1.2+ x86-64

logprep-18.0.1-cp313-cp313-manylinux_2_28_x86_64.whl (836.9 kB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

logprep-18.0.1-cp312-cp312-musllinux_1_2_x86_64.whl (908.1 kB view details)

Uploaded CPython 3.12musllinux: musl 1.2+ x86-64

logprep-18.0.1-cp312-cp312-manylinux_2_28_x86_64.whl (836.8 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

logprep-18.0.1-cp311-cp311-musllinux_1_2_x86_64.whl (910.6 kB view details)

Uploaded CPython 3.11musllinux: musl 1.2+ x86-64

logprep-18.0.1-cp311-cp311-manylinux_2_28_x86_64.whl (839.7 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

File details

Details for the file logprep-18.0.1.tar.gz.

File metadata

  • Download URL: logprep-18.0.1.tar.gz
  • Upload date:
  • Size: 3.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for logprep-18.0.1.tar.gz
Algorithm Hash digest
SHA256 da99432df46e36e1d103f22aee7d08be6f9e1cb49d3b09dc012aa5f4229f69b3
MD5 008b4d8e22daa1e8a2bf1ba3eba8ae4c
BLAKE2b-256 b3877edd6722ff8e77261a3d89368c538fe77933320e87e9a92ff4601a387cf4

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1.tar.gz:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp314-cp314-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp314-cp314-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 20eda5dac768f8eee89ec0ed6fa0d2c67eb2fd38c56c17fa88fa8070b29300cc
MD5 c3d7f26eafa01657abf3216203cd6e25
BLAKE2b-256 c516a9fb4172de777e956ce2ba583687035c7cda7908cbdd38a99e5836ea077d

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp314-cp314-musllinux_1_2_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp314-cp314-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp314-cp314-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 339592946698dc43a54cf53bca24bdacdfaeabfbf62f50431ad778b3b09c86fa
MD5 66ddb5c1d9ad475e44354719cf816d32
BLAKE2b-256 d194524982f338e3a085b86239ea60903a79f0c575128b6f13e1d6dcfc98f7a5

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp314-cp314-manylinux_2_28_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp313-cp313-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp313-cp313-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 64a675042a63478a0cbae758bf27c48d0f0c059ce56049df7fde6920a17373ad
MD5 107cfb9a4dee416b0052197aa8c5349b
BLAKE2b-256 12735b6fed360e3fd00d101ed2e58a8370a8e075840d0471a26d1ddb6e1a6b9d

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp313-cp313-musllinux_1_2_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 3ef6e0883fe3de43c0b837543b72c40a746167bbe0aa0fdddff094687008a927
MD5 bb1b810528d37fcc01d05e9dbf8b3313
BLAKE2b-256 66faaf6d77b4de34c0f44cc0569da850c87f201af3c837e430d23d790be7351c

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp313-cp313-manylinux_2_28_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp312-cp312-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 cf08bcc8a87272ddc64896c269aea37ed8f9fef13e2c65f39251d90708454ddc
MD5 1b2b25d2e423d83fd7dd2877a215eb02
BLAKE2b-256 a6aa5c3ba998ed69f0a0369c235cc6ac38124e9f918da98ccc797e70824d7836

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp312-cp312-musllinux_1_2_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 a57daa59efdb845832c1019bbcefcd7a66478584a00795ec9e9309ae7153fa82
MD5 2c7871004b5001791b18c9dce27bb7f1
BLAKE2b-256 7509c6125e9f04a29678dd68ed65abe83eba7879ff90a29170cf642fbd6433b5

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp312-cp312-manylinux_2_28_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp311-cp311-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 01807c55fe6e06998b37160a43003528c7ad09e36434e38f8cf638c4d7a276e1
MD5 a8eede39f015bec3321ab530b271f8fd
BLAKE2b-256 3d16f4d5727d58b08b8ca21d5b05749e0b7117b3bd184d76f7caabf4c1920117

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp311-cp311-musllinux_1_2_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file logprep-18.0.1-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for logprep-18.0.1-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 77684e62dc6854e8f86bb3242618635ab29a082f3c9eee987a5b2d8ba1972e03
MD5 4c7a988bba95535c636ecba5fc9c4861
BLAKE2b-256 66bb101053ec544071a3fefc680cfc80964381029e4138e2099bc2bcef5d14b7

See more details on using hashes here.

Provenance

The following attestation bundles were made for logprep-18.0.1-cp311-cp311-manylinux_2_28_x86_64.whl:

Publisher: publish-release-to-pypi.yml on fkie-cad/Logprep

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page