Explore Data Pipelines in Apache Kafka.
Project description
Streams Explorer
Explore Data Pipelines in Apache Kafka.
Contents
Features
- Visualization of streaming applications, topics, and connectors
- Monitor all or individual pipelines from multiple namespaces
- Inspection of Avro schema from schema registry
- Integration with streams-bootstrap and faust-bootstrap for deploying Kafka Streams applications
- Real-time metrics from Prometheus (consumer lag & read rate, replicas, topic size, messages in & out per second, connector tasks)
- Linking to external services for logging and analysis, such as Kibana, Grafana, AKHQ, Elasticsearch
- Customizable through Python plugins
Overview
Visit our introduction blogpost for a complete overview and demo of Streams Explorer.
Installation
Docker Compose
- Forward the ports to Prometheus. (Kafka Connect, Schema Registry, and other integrations are optional)
- Start the container
docker compose up
Once the container is started visit http://localhost:3000
Deploying to Kubernetes cluster
- Add the Helm chart repository
helm repo add streams-explorer https://raw.githubusercontent.com/bakdata/streams-explorer/master/helm-chart/
- Install
helm upgrade --install --values helm-chart/values.yaml streams-explorer
Standalone
Backend
- Install dependencies using Poetry
poetry install
- Forward the ports to Prometheus. (Kafka Connect, Schema Registry, and other integrations are optional)
- Configure the backend in settings.yaml.
- Start the backend server
uvicorn main:app
Frontend
- Install dependencies
npm ci
- Start the frontend server
npm start
Visit http://localhost:3000
Configuration
Depending on your type of installation set the configuration for the backend server in this file:
- Docker Compose: docker-compose.yaml
- Kubernetes: helm-chart/values.yaml
- standalone: backend/settings.yaml
In the helm-chart/values.yaml configuration is done either through the config
section using double underscore notation, e.g. K8S__consumer_group_annotation: consumerGroup
or the content of backend/settings.yaml can be pasted under the settings
section. Alternatively all configuration options can be written as environment variables using double underscore notation and the prefix SE
, e.g. SE_K8S__deployment__cluster=false
.
The following configuration options are available:
General
graph.update_interval
Update the graph every X seconds (integer, required, default:300
)graph.layout_arguments
Arguments passed to graphviz layout (string, required, default:-Grankdir=LR -Gnodesep=0.8 -Gpad=10
)graph.pipeline_distance
Increase/decrease vertical space between pipeline graphs by X pixels (int, required, default:500
)
Kafka
kafka.enable
Enable Kafka (bool, required, default:false
)kafka.config
librdkafka configuration properties (reference) (dict, required, default:{"bootstrap.servers": "localhost:9092"}
)kafka.displayed_information
Configuration options of Kafka topics displayed in the frontend (list of dict)
Kafka Connect
kafkaconnect.url
URL of Kafka Connect server (string, default: None)kafkaconnect.displayed_information
Configuration options of Kafka connectors displayed in the frontend (list of dict)
Kubernetes
k8s.deployment.cluster
Whether streams-explorer is deployed to Kubernetes cluster (bool, required, default:false
)k8s.deployment.context
Name of cluster (string, optional if running in cluster, default:kubernetes-cluster
)k8s.deployment.namespaces
Kubernetes namespaces (list of string, required, default:['kubernetes-namespace']
)k8s.containers.ignore
Name of containers that should be ignored/hidden (list of string, default:['prometheus-jmx-exporter']
)k8s.displayed_information
Details of pod that should be displayed (list of dict, default:[{'name': 'Labels', 'key': 'metadata.labels'}]
)k8s.labels
Labels used to set attributes of nodes (list of string, required, default:['pipeline']
)k8s.pipeline.label
Attribute of nodes the pipeline name should be extracted from (string, required, default:pipeline
)k8s.consumer_group_annotation
Annotation the consumer group name should be extracted from (string, required, default:consumerGroup
)
Schema Registry / Karapace
schemaregistry.url
URL of Confluent Schema Registry or Karapace (string, default: None)
Prometheus
prometheus.url
URL of Prometheus (string, required, default:http://localhost:9090
)
The following exporters are required to collect Kafka metrics for Prometheus:
AKHQ
akhq.enable
Enable AKHQ (bool, required, default:false
)akhq.url
URL of AKHQ (string, default:http://localhost:8080
)akhq.cluster
Name of cluster (string, default:kubernetes-cluster
)akhq.connect
Name of connect (string, default: None)
Kowl
Kowl can be used instead of AKHQ. (mutually exclusive)
kowl.enable
Enable Kowl (bool, required, default:false
)kowl.url
URL of Kowl (string, default:http://localhost:8080
)
Grafana
grafana.url
URL of Grafana (string, default:http://localhost:3000
)grafana.dashboards.topics
Path to topics dashboard (string), sample dashboards for topics and consumer groups are included in the./grafana
subfoldergrafana.dashboards.consumergroups
Path to consumer groups dashboard (string)
Kibana
kibanalogs.enable
Enable Kibana logs (bool, required, default:false
)kibanalogs.url
URL of Kibana logs (string, default:http://localhost:5601
)
Loki
Loki can be used instead of Kibana. (mutually exclusive)
loki.enable
Enable Loki logs (bool, required, default:false
)loki.url
URL of Loki logs (string, default:http://localhost:3000
)
Elasticsearch
for Kafka Connect Elasticsearch connector
esindex.url
URL of Elasticsearch index (string, default:http://localhost:5601/app/kibana#/dev_tools/console
)
Plugins
plugins.path
Path to folder containing plugins relative to backend (string, required, default:./plugins
)plugins.extractors.default
Whether to load default extractors (bool, required, default:true
)
Demo pipeline
ATM Fraud detection with streams-bootstrap
Plugin customization
It is possible to create your own linker, metric provider, and extractors in Python by implementing the LinkingService
, MetricProvider
, or Extractor
classes. This way you can customize it to your specific setup and services. As an example we provide the DefaultLinker
as LinkingService
. The default MetricProvider
supports Prometheus. Furthermore the following default Extractor
plugins are included:
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for streams_explorer-1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d05a791f70e82df0fcd3048cd9bd3bb533d0c85a86e870b55c07083d45423bea |
|
MD5 | 742124ddcd78e02e5f6fd8ca63f37865 |
|
BLAKE2b-256 | b9a32043a34d7feb026f4ad5c3081bf7ecb609d89f349ec59ccf48f67133e639 |