mentat-gulp 1.6.52

gULP - (generic) Unified Log Processor.

the graphical, universal Log processor for incident response!

made with :heart: by Mentat.

Description - Architecture - Installation - Run & examples - GUI - Troubleshooting

description

Gulp is a powerful software tool designed to streamline incident response and analysis. Its core features includes:

• Data Ingestion Plugins: Gulp can ingest data from a variety of sources, thanks to its versatile plugin system.

• OpenSearch and ECS: Gulp is built on OpenSearch and uses the Elasticsearch Common Scheme (ECS) as its ingestion format, ensuring compatibility and ease of use.

• High-Speed Multiprocessing Engine: Gulp's engine is designed for speed, offering fast ingestion and querying capabilities through multiprocessing.

• Query using SIGMA rules: Gulp supports querying using Sigma Rules, allowing for easy, one-click queries with thousands of rules in parallel.

• Collaboration Platform: Gulp includes a collaboration platform, enabling teams to work together on the same incident. Features include note-taking, highlighting, and link adding.

• An innovative UI: Gulp's user interface includes multiple on-screen per-context(i.e. a log source) zoomable timelines for visualizing events, making it easier to understand and analyze incidents.

  

  

  

• Scalable: Gulp is designed with scalability in mind. As your data and team grow, you can simply add more gulp nodes, more cores to increase parallel ingestion and query capabilities, more OpenSearch and PostgreSQL nodes. This makes Gulp a flexible solution that can adapt to your evolving needs!

• Python based: Gulp is written in Python, leveraging open-source libraries whenever possible. This maximizes ease of adoption from the community, as Python is widely used and understood.

here is a detailed datasheet

architecture

• GULP architecture

plugins development

• plugins and mapping
• testing guidelines

integration with other applications

gulp can be of course integrated with other applications !

both websocket and REST API is available!

installation

install our pypi package and run the necessary services via the example docker-compose and you should be good to go with the backend:

WARNING: pip installation works only with python3.13 (tested) and 3.12, 3.14 is currently not supported because some dependencies have not yet released compatible versions.

set -e
curl -o .env https://raw.githubusercontent.com/mentat-is/gulp/develop/.env \
  && curl -o docker-compose.yml https://raw.githubusercontent.com/mentat-is/gulp/develop/docker-compose.yml \
  &&pip install mentat-gulp

then, start the docker compose to start the necessary services:

docker compose up -d

and finally start the backend creating a fresh collab database and operation:

gulp --reset-collab --create some_operation

gulp creates a default configuration in $HOME/.config/gulp/gulp_cfg.json if not already present and starts listening on http://localhost:8080

for advanced deployment, always check the following:

• install with docker
• install from sources/dev setup
• installing extra plugins

pypi/docker registry versions may be outdated... to use the bleeding edge version of gulp, install from sources and use the 'develop' branches.

clients

gulp web ui

environment variables

the following environment variables may be set to override configuration options.

• GULP_BIND_TO_ADDR, GULP_BIND_TO_PORT : if set, gulp will listen to this interface and port (either, the default 0.0.0.0, 8080 is used).

  • for the override to work, both GULP_BIND_TO_ADDR and GULP_BIND_TO_PORT must be specified, either the value of one alone is ignored.

• GULP_WORKING_DIR: this is the working directory for gulp (defaults to ~/.config/gulp), which contains:

  • gulp_cfg.json: the configuration, initialized with template if not present
  • plugins: optional extra plugins (have precedence over $INSTALLDIR/plugins)
  • mapping_files: optional extra mapping files (have precedence over $INSTALLDIR/mapping_files)
  • certs: optional SSL certificates for HTTPS
  • ingest_local directory to store big files for quick ingestion (ingest_local API)
  • tmp_upload folder to cache partial uploads during ingestion

• GULP_OPENSEARCH_URL: if set, overrides opensearch_url in the configuration to.

• GULP_POSTGRES_URL: if set, overrides postgres_url in the configuration.

• GULP_REDIS_URL: if set, overrides redis_url in the configuration.

• GULP_S3_URL: if set, overrides s3_url in the configuration.

• GULP_INTEGRATION_TEST: this may be set during tests to disable debug features if they're enabled

exposed services

using the default docker-compose.yml with no profile set. further profiles (i.e. dev, metrics) may expose additional services, check the compose file for details.

gulp

• gulp swagger page on http://localhost:8080/docs
• gulp web UI on http://localhost:3000
  • user/pwd: admin/admin (default gulp admin user)

postgreSQL

• postgreSQL on localhost:5432

  • user/pwd: postgres/Gulp1234!

• adminer on http://localhost:8001 to manage postgreSQL (use --profile dev)

  • server/user/pwd: postgres/postgres/Gulp1234!

OpenSearch

• opensearch on http://localhost:9200

  • user/pwd: admin/Gulp1234!

• elasticvue on http://localhost:8082 to visualize OpensSearch indexes (use --profile dev)

• opensearch dashboards on http://localhost:5001 for a more comprehensive OpenSearch management.

redis

• redis on localhost:6379

  • user/pwd: default/Gulp1234!

• redis insight on http://localhost:5540 to manage redis (use --profile dev)

MinIO

• minio on http://localhost:9000

  • user/pwd: admin/Gulp1234!
  • this is used as S3-compatible filestore for plugins which may need it.

SSL

to use SSL, the following configuration options and files variables may be provided:

OpenSearch

• Gulp configuration
  • opensearch_verify_certs: set to false to skip server verification
• certificate files
  • $GULP_WORKING_DIR/certs/opensearch-ca.pem: CA certificate for Gulp to connect to the Opensearch server
  • $GULP_WORKING_DIR/certs/opensearch.pem: client certificate for Gulp to connect to the Opensearch server
  • $GULP_WORKING_DIR/certs/opensearch.key: passwordless client certificate key

PostgreSQL

• Gulp configuration
  • postgres_ssl: use SSL for postgres connection, set to false to not use.
  • postgres_verify_certs: set to false to skip server verification
• certificate files
  • $GULP_WORKING_DIR/certs/postgres-ca.pem: CA certificate for Gulp to connect to the PostgreSQL server
  • $GULP_WORKING_DIR/certs/postgres.pem: client certificate for Gulp to connect to PostgreSQL server
  • $GULP_WORKING_DIR/certs/postgres.key: client certificate key

gulp

• Gulp configuration
  • https_enforce: set to true to enforce connection to Gulp only through HTTPS
  • https_enforce_client_certs: set to true to enforce check of client certificates signed by gulp-ca.pem CA
• certificate files
  • $GULP_WORKING_DIR/certs/gulp-ca.pem: Gulp CA
  • $GULP_WORKING_DIR/certs/gulp.pem: Gulp server certificate
  • $GULP_WORKING_DIR/certs/gulp.key: Gulp server certificate key

redis & minIO

check their specific documentation: basically you have to put certificates in $GULP_WORKING_DIR/certs, whcih is mounted in the containers, and tweak their configuration.

commandline examples

default startup, creates collab database with an operation names test_operation on the very first run.

gulp

to detect if gulp has already run once check for ~/.config/gulp/.first_run_done and delete it to revert to first run on the next run.

deletes data related to ALL existing operations, both on collaboration database and OpenSearch.

gulp --reset-collab

deletes data related to ALL existing operations, both on collaboration database and OpenSearch, in the end creates/recreates my_operation.

gulp --reset-collab --create my_operation

acts only on my_operation: creates/recreates operation, deletes all related data both on collaboration database and OpenSearch.

gulp --create my_operation