jps-observability-utils

Utilities for auditing Python and Node.js repositories for evidence of observability-related instrumentation, telemetry configuration, monitoring components, and integrations with common observability platforms.

Overview

jps-observability-utils is a Python CLI package that performs static repository scans and generates observability audit reports.

The initial version is designed for legacy codebase assessment, onboarding, and engineering due diligence. It helps identify evidence of technologies such as OpenTelemetry, Prometheus, Datadog, New Relic, Sentry, Elastic APM, collector configuration, metrics endpoints, and structured logging patterns.

The package is intentionally evidence-based. It does not claim runtime certainty. It reports what the repository contents suggest.

Initial Scope

The initial release includes two Typer-based CLI commands:

• audit-python — scan a Python repository
• audit-node — scan a Node.js repository

Each command generates a human-readable Markdown report and a machine-readable JSON report.

What the Tool Detects

The scanners look for evidence of observability-related instrumentation and integrations, including:

• telemetry instrumentation libraries
• OpenTelemetry SDKs, exporters, and environment variables
• Prometheus client libraries and /metrics patterns
• vendor-specific observability platforms such as Datadog, New Relic, Sentry, and Elastic APM
• collector / exporter configuration
• deployment and environment configuration relevant to telemetry
• structured logging patterns relevant to observability

What the Tool Does Not Do

This project does not, in its initial version:

• execute code
• validate runtime telemetry emission
• prove that observability is functioning in production
• modify the target repository
• auto-remediate missing instrumentation

Why This Tool Exists

Legacy repositories often contain partial, inconsistent, or undocumented observability setups. Engineers reviewing a codebase typically need fast answers to questions such as:

• Does this project appear to use OpenTelemetry?
• Is Prometheus instrumentation present?
• Is there evidence of Datadog or New Relic integration?
• Are telemetry environment variables configured?
• Is there collector or OTLP configuration in the repo?
• Are there signs of structured logging or metrics endpoints?

This tool is intended to reduce manual grep-heavy investigation.

Proposed CLI Usage

Examples:

jps-observability-utils audit-python /path/to/python-repo --format both --output-dir ./reports
jps-observability-utils audit-node /path/to/node-repo --format both --output-dir ./reports

Possible options may include:

• --output-dir
• --format [md|json|both]
• --ignore PATTERN
• --verbose

Expected Report Content

Each report should include:

• scan metadata
• repository path
• number of files scanned
• summary of detected technologies
• findings grouped by category
• confidence level for each finding
• file paths and evidence locations
• caveats explaining that the audit is static and heuristic-based

Confidence Model

A simple confidence model is recommended:

• High — strong evidence such as dependency + initialization code or env vars + exporter configuration
• Medium — partial but meaningful evidence such as dependency presence without clear initialization
• Low — weak or indirect evidence only

Suggested MVP Detection Targets

Python repositories

• OpenTelemetry
• Prometheus
• Datadog
• New Relic
• Sentry
• Elastic APM
• collector / OTLP config
• structured logging indicators

Node.js repositories

• OpenTelemetry
• Prometheus
• Datadog
• New Relic
• Sentry
• Elastic APM
• collector / OTLP config
• structured logging indicators

Recommended Package Structure

src/jps_observability_utils/
├── cli.py
├── constants.py
├── models.py
├── scanner.py
├── report_writer.py
├── matchers/
│   ├── common.py
│   ├── python_repo.py
│   └── node_repo.py
└── utils/
    ├── file_utils.py
    └── text_utils.py

Design Principles

• static evidence detection, not runtime proof
• clear and conservative language
• modular detection rules
• stable JSON output
• easy extensibility for additional technologies and languages

Example GitHub Project Description

Utilities for auditing Python and Node.js repositories for observability-related instrumentation and integrations.

Development Notes

Recommended implementation choices:

• Python 3.11+
• Typer for CLI
• pathlib for filesystem traversal
• dataclasses or Pydantic for report models
• pytest for testing

Testing Strategy

The test suite should include small fixture repositories representing:

• positive OpenTelemetry detection
• Prometheus-only detection
• vendor-specific APM detection
• no observability evidence
• mixed evidence across code and deployment files

Future Enhancements

Potential future additions:

• unified audit-repo command with language auto-detection
• HTML reports
• SARIF output
• maturity scoring
• custom rule packs
• support for additional languages

Status

This repository is intended to start with two focused audit utilities and expand over time as the detection catalog matures.