certmesh 3.0.32

Multi-provider TLS certificate lifecycle management — DigiCert, Venafi TPP, Vault PKI, ACM, Let's Encrypt

certmesh

Automated TLS certificate lifecycle management for Python 3.10+.

A unified CLI and Python API for managing certificates across DigiCert CertCentral, Venafi Trust Protection Platform, HashiCorp Vault PKI, and AWS Certificate Manager (public + private CA).

Features

• Multi-provider -- single tool for DigiCert, Venafi TPP, Vault PKI, AWS ACM/ACM-PCA, and Let's Encrypt
• Full lifecycle -- request, list, search, describe, download, renew, revoke, and export certificates
• REST API -- production-grade FastAPI service with OAuth2 (ADFS / Azure Entra ID), API key exchange, rate limiting, GZip compression, Prometheus metrics, and full CRUD endpoints for all providers
• Web UI Dashboard -- browser-based certificate inventory, renewal/revocation controls, expiry timeline, and live log streaming (SSE + xterm.js) with OAuth2 SSO and S3-backed state persistence
• Spec-compliant -- all provider API calls validated against official API reference documentation (DigiCert CertCentral v2, Venafi TPP v23/v25.3, AWS ACM)
• Credential security -- secrets come from Vault (KV v1/v2) or environment variables, never from config files
• Resilient -- circuit breakers with TOCTOU-safe HALF_OPEN probing, exponential-backoff retry, monotonic-clock polling, and configurable timeouts on all HTTP calls
• Automatic renewal -- scheduled certificate renewal engine with configurable policy (before-expiry threshold, per-provider dispatch) and Prometheus Pushgateway metrics for CronJob observability
• Configurable -- layered config: built-in defaults < YAML file < CM_* environment variables
• Cloud-native -- Docker image, Helm chart for EKS with IRSA, NLB, HPA, Vault PKI TLS, and JSON Schema input validation
• Typed -- fully typed with py.typed marker; dataclass models for all API responses

Installation

pip install certmesh

From source:

git clone https://github.com/SCGIS-Wales/certmesh.git
cd certmesh
pip install -e ".[dev]"

Requires Python 3.10, 3.11, 3.12, 3.13, or 3.14.

Quick Start

# Show effective config
certmesh config show

# Issue a certificate from Vault PKI
certmesh vault-pki issue --cn myservice.example.com --ttl 720h

# Request a public ACM certificate
certmesh acm request --cn myapp.example.com --validation DNS

# List DigiCert certificates
certmesh digicert list --status issued

# Renew a Venafi TPP certificate
certmesh venafi renew --guid "a1b2c3d4-e5f6-7890-abcd-ef1234567890"

Configuration

Configuration is layered (lowest to highest precedence):

1. Built-in defaults -- sensible defaults for all settings
2. YAML config file -- config/config.yaml or --config PATH
3. CM_* environment variables -- override any setting

# Use a .env file
certmesh --env-file .env digicert list

# Override log level
certmesh --log-level DEBUG acm list

See config/config.yaml for the full annotated reference and .env.example for all environment variables.

Authentication

Provider	Method
Vault	AppRole (default), LDAP, or AWS IAM
DigiCert	API key from CM_DIGICERT_API_KEY or Vault KV (keys do not expire but can be revoked)
Venafi	OAuth2 (TPP 20.1+, required 22.3+) or LDAP (pre-22.3 only); credentials from CM_VENAFI_USERNAME/CM_VENAFI_PASSWORD or Vault KV
AWS ACM	Standard boto3 credential chain (IAM role, env vars, ~/.aws/credentials)
Let's Encrypt	ACME account key (auto-generated or provided)

Credentials are resolved env-first, Vault-fallback. Vault is only contacted when needed.

CLI Reference

Exit codes: 0 = success, 1 = config/auth error, 2 = cert operation error, 3 = unexpected error.

DigiCert CertCentral

certmesh digicert list     [--status TEXT] [--limit INT]
certmesh digicert search   [--cn TEXT] [--serial TEXT] [--status TEXT] [--product TEXT]
certmesh digicert describe --cert-id INT
certmesh digicert order    --cn TEXT [--san TEXT ...]
certmesh digicert download --cert-id INT --key-file PATH
certmesh digicert revoke   --cert-id INT|--order-id INT [--reason CHOICE] [--comments TEXT]
certmesh digicert duplicate --order-id INT --csr-file PATH [--cn TEXT] [--san TEXT ...]

Venafi TPP

certmesh venafi list       [--limit INT] [--offset INT]
certmesh venafi search     [--cn TEXT] [--san TEXT]
certmesh venafi describe   --guid TEXT
certmesh venafi request    --policy-dn TEXT --cn TEXT [--san TEXT ...] [--client-csr]
certmesh venafi renew      --guid TEXT
certmesh venafi renew-bulk --guid-file PATH
certmesh venafi revoke     --dn TEXT|--thumbprint TEXT [--reason INT] [--disable]
certmesh venafi download   --guid TEXT

HashiCorp Vault PKI

certmesh vault-pki issue   --cn TEXT [--san TEXT ...] [--ip-san TEXT ...] [--ttl TEXT] [--output-dir PATH]
certmesh vault-pki sign    --cn TEXT --csr-file PATH [--san TEXT ...] [--ttl TEXT] [--output-dir PATH]
certmesh vault-pki list
certmesh vault-pki read    --serial TEXT
certmesh vault-pki revoke  --serial TEXT

AWS ACM (Public Certificates)

certmesh acm request           --cn TEXT [--san TEXT ...] [--validation DNS|EMAIL] [--key-algorithm TEXT] [--region TEXT]
certmesh acm list              [--status TEXT ...] [--region TEXT]
certmesh acm describe          --arn TEXT [--region TEXT]
certmesh acm export            --arn TEXT --passphrase [--output-dir PATH] [--region TEXT]
certmesh acm renew             --arn TEXT [--region TEXT]
certmesh acm delete            --arn TEXT [--region TEXT]
certmesh acm validation-records --arn TEXT [--region TEXT]
certmesh acm wait              --arn TEXT [--region TEXT]

AWS ACM Private CA

certmesh acm-pca issue   --ca-arn TEXT --csr-file PATH [--validity-days INT] [--signing-algorithm TEXT] [--region TEXT]
certmesh acm-pca get     --ca-arn TEXT --cert-arn TEXT [--region TEXT]
certmesh acm-pca revoke  --ca-arn TEXT --cert-arn TEXT --cert-serial TEXT [--reason CHOICE] [--region TEXT]
certmesh acm-pca list    --ca-arn TEXT [--region TEXT]

Config Management

certmesh config show       # Display effective merged config (secrets redacted)
certmesh config validate   # Validate config; exits 0 on success, 1 on failure

Remote Management

The CLI can manage a remote certmesh API instance running on Kubernetes. Every command works transparently in remote mode.

Connecting

# Direct HTTPS endpoint
certmesh --remote https://certmesh.prod.example.com venafi list

# Kubernetes port-forward (auto-manages kubectl subprocess)
certmesh --remote k8s://certmesh/certmesh-api:8000 venafi list

# TLS port-forward (service speaks HTTPS)
certmesh --remote k8s+tls://certmesh/certmesh-api:8443 venafi list

Authentication

# JWT token (required if server has OAuth2 enabled)
certmesh --remote https://certmesh.example.com --token "$JWT_TOKEN" venafi list

# With refresh token for automatic renewal on expiry
certmesh --remote https://certmesh.example.com \
  --token "$JWT_TOKEN" \
  --refresh-token "$REFRESH_TOKEN" \
  venafi renew --guid "a1b2c3d4-..."

# Environment variables (avoid tokens in shell history)
export CM_REMOTE_URL=https://certmesh.example.com
export CM_REMOTE_TOKEN="$JWT_TOKEN"
certmesh venafi list

If the server does not have OAuth2 enabled, no token is required — the CLI detects this automatically via GET /api/v1/info.

TLS & Proxy Configuration

# Custom CA certificate bundle
certmesh --remote https://certmesh.internal --remote-ca-cert /etc/pki/ca-bundle.pem venafi list

# System CA trust (automatically read from environment)
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt    # OpenSSL standard
export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt # Python requests
export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt    # curl convention

# Forward proxy support (httpx reads these automatically)
export HTTPS_PROXY=http://proxy.corp.example.com:8080
export NO_PROXY=localhost,127.0.0.1,.internal
certmesh --remote https://certmesh.example.com venafi list

Version Compatibility

The CLI enforces exact version match with the remote server. On connect, it calls GET /api/v1/info and compares versions:

$ certmesh --remote https://certmesh.example.com venafi list
Version mismatch: CLI version 3.0.15 does not match remote server version 3.1.0.
Install the matching version:
  pip install certmesh==3.1.0

Remote Health Checks

certmesh --remote https://certmesh.example.com remote status
certmesh --remote https://certmesh.example.com remote health
certmesh --remote https://certmesh.example.com remote ready

OpenAPI Specification

The certmesh API serves an OpenAPI 3.1 specification for validation and client generation:

# OpenAPI JSON schema (always available)
curl https://certmesh.example.com/openapi.json

# Swagger UI (when CM_DOCS_ENABLED=true)
# https://certmesh.example.com/docs

# ReDoc (when CM_DOCS_ENABLED=true)
# https://certmesh.example.com/redoc

Environment Variable	Description
CM_REMOTE_URL	Remote certmesh API URL
CM_REMOTE_TOKEN	JWT Bearer token
CM_REMOTE_REFRESH_TOKEN	OAuth2 refresh token
CM_REMOTE_TIMEOUT	Request timeout in seconds (default 30)
CM_REMOTE_CA_CERT	Path to CA certificate bundle
CM_REMOTE_TLS_VERIFY	Enable/disable TLS verification

REST API

The REST API provides the same operations as the CLI over HTTP. Start the API server with:

uvicorn certmesh.api.app:create_app --factory --host 0.0.0.0 --port 8000

Base path: /api/v1. Health endpoints at /healthz, /readyz, /livez. Prometheus metrics at /metrics.

DigiCert Endpoints

Method	Path	Description
GET	/api/v1/digicert/certificates	List issued certificates
POST	/api/v1/digicert/certificates/search	Search certificates by CN, status
GET	/api/v1/digicert/certificates/{id}	Describe a certificate
POST	/api/v1/digicert/orders	Order a new certificate
POST	/api/v1/digicert/certificates/{id}/revoke	Revoke a certificate

Venafi TPP Endpoints

Method	Path	Description
GET	/api/v1/venafi/certificates	List managed certificates
POST	/api/v1/venafi/certificates/search	Search by CN, SAN, thumbprint, serial, issuer
GET	/api/v1/venafi/certificates/{guid}	Describe a certificate by GUID
POST	/api/v1/venafi/certificates/{guid}/renew	Renew a certificate
POST	/api/v1/venafi/certificates/{guid}/revoke	Revoke a certificate

AWS ACM Endpoints

Method	Path	Description
GET	/api/v1/acm/certificates	List ACM certificates
POST	/api/v1/acm/certificates	Request a new certificate
GET	/api/v1/acm/certificates/{arn}/detail	Describe a certificate
GET	/api/v1/acm/certificates/{arn}/validation-records	Get DNS/email validation records
POST	/api/v1/acm/certificates/{arn}/export	Export certificate + private key (requires passphrase)
DELETE	/api/v1/acm/certificates/{arn}	Delete a certificate
POST	/api/v1/acm/route53/sync	Sync DNS validation records to Route53

Vault PKI Endpoints

Method	Path	Description
GET	/api/v1/vault-pki/certificates	List PKI certificates
POST	/api/v1/vault-pki/certificates	Issue a new certificate
GET	/api/v1/vault-pki/certificates/{serial}	Read a certificate by serial
POST	/api/v1/vault-pki/sign	Sign a CSR

Authentication Endpoints

Method	Path	Description
POST	/api/v1/auth/token	Exchange JWT for short-lived API key
POST	/api/v1/auth/token/refresh	Refresh API key
POST	/api/v1/auth/token/revoke	Revoke API key

Architecture

certmesh/
  cli.py               -- Click CLI (entry point: certmesh.cli:cli)
  settings.py           -- Layered config: defaults -> YAML -> env vars
  credentials.py        -- Env-first, Vault-fallback secret resolution
  certificate_utils.py  -- Key gen, CSR, PKCS#12, bundle assembly, persistence
  circuit_breaker.py    -- Thread-safe CLOSED/OPEN/HALF_OPEN state machine
  renewal.py            -- Automatic certificate renewal engine
  renewal_metrics.py    -- Prometheus Pushgateway push for CronJob observability
  exceptions.py         -- Full exception hierarchy
  providers/
    digicert_client.py  -- DigiCert CertCentral API v2 (spec-validated)
    venafi_client.py    -- Venafi TPP v23/v25.3 (OAuth2 + LDAP, spec-validated)
    acm_client.py       -- AWS ACM + ACM-PCA (boto3, spec-validated)
    letsencrypt_client.py -- Let's Encrypt / ACME (RFC 8555)
  backends/
    vault_client.py     -- Vault auth + KV v1/v2 + PKI engine
    secrets_manager_client.py -- AWS Secrets Manager
    route53_client.py   -- Route53 DNS record management
  api/
    app.py              -- FastAPI application factory
    auth.py             -- OAuth2 JWT Bearer validation (RS256 algorithm restriction)
    apikeys.py          -- Thread-safe API key exchange store
    rate_limiter.py     -- Configurable rate limiting (SlowAPI, RFC 7231)
    compression.py      -- GZip compression middleware
    routes/             -- REST API endpoints
    metrics.py          -- Prometheus metrics

Certificate Output

Issued certificates can be persisted to:

• Filesystem -- PEM files with private keys written mode 0600
• Vault KV v1/v2 -- certificate material stored as a KV secret (versioned or unversioned)
• AWS Secrets Manager -- certificate bundle stored as a JSON secret
• Multiple destinations -- any combination of the above

Configured per-provider via output.destination (list or legacy string: filesystem, vault, secrets_manager, or both).

Development

# Clone and install
git clone https://github.com/SCGIS-Wales/certmesh.git
cd certmesh
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest -v --cov=certmesh

# Lint and format
ruff check src/ tests/
ruff format src/ tests/

Docker

docker build -t certmesh .
docker run --rm certmesh --help

# Run the REST API
docker run -p 8000:8000 -e CM_CONFIG_FILE=/app/config.yaml certmesh

Helm Chart

helm install certmesh helm/certmesh \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/certmesh

See helm/certmesh/values.yaml for all configuration options. Helm values are validated at install time via values.schema.json.

Observability

API Server Metrics

The REST API exposes Prometheus metrics at /metrics (scraped by a Kubernetes ServiceMonitor when metrics.serviceMonitor.enabled=true):

Metric	Type	Labels
certmesh_http_requests_total	Counter	method, endpoint, status
certmesh_http_request_duration_seconds	Histogram	endpoint
certmesh_certificate_operations_total	Counter	provider, operation, status
certmesh_certificate_expiry_days	Gauge	provider, common_name, serial
certmesh_circuit_breaker_state	Gauge	name

Renewal CronJob Metrics (Pushgateway)

The renewal CronJob is ephemeral — once the pod exits, Prometheus has nothing to scrape. To keep renewal results visible, certmesh pushes metrics to a Prometheus Pushgateway at the end of each run.

Enable via Helm:

certRenewal:
  enabled: true
  pushgatewayUrl: "http://prometheus-pushgateway:9091"

Or via environment variable:

CERTMESH_PUSHGATEWAY_URL=http://prometheus-pushgateway:9091 certmesh renewal check

Metrics pushed after each run:

Metric	Type	Labels	Purpose
certmesh_cert_expiry_seconds	Gauge	provider, common_name	Per-cert expiry countdown — visible in Grafana even when renewal fails
certmesh_renewal_last_run_timestamp_seconds	Gauge	—	Unix timestamp of last run; detect a stalled CronJob
certmesh_renewal_last_success_timestamp_seconds	Gauge	—	Unix timestamp of last error-free run; 0 if the last run had errors
certmesh_renewal_certs_checked	Gauge	provider	Sanity check — unexpected drop signals a provider is unreachable
certmesh_renewal_certs_renewed	Gauge	provider	How many certs were renewed
certmesh_renewal_errors	Gauge	provider	Non-zero means a provider failed

A push failure is logged as a warning and never causes the renewal run to fail.

Useful PromQL (no AlertManager required):

# Any certificate expiring in less than 7 days
certmesh_cert_expiry_seconds < 7 * 86400

# Renewal check has not run in over 25 hours
time() - certmesh_renewal_last_run_timestamp_seconds > 90000

# Last run had errors in any provider
certmesh_renewal_errors > 0

AWS IAM Permissions

When running on AWS (EKS with IRSA, EC2, or Lambda), the IAM role needs the following permissions depending on which features are enabled:

AWS Certificate Manager (ACM)

{
  "Effect": "Allow",
  "Action": [
    "acm:RequestCertificate",
    "acm:DescribeCertificate",
    "acm:ListCertificates",
    "acm:DeleteCertificate",
    "acm:RenewCertificate",
    "acm:ExportCertificate",
    "acm:GetCertificate",
    "acm:ListTagsForCertificate",
    "acm:AddTagsToCertificate"
  ],
  "Resource": "*"
}

AWS ACM Private CA (ACM-PCA)

{
  "Effect": "Allow",
  "Action": [
    "acm-pca:IssueCertificate",
    "acm-pca:GetCertificate",
    "acm-pca:RevokeCertificate",
    "acm-pca:ListCertificateAuthorities",
    "acm-pca:DescribeCertificateAuthority",
    "acm-pca:GetCertificateAuthorityCertificate"
  ],
  "Resource": "*"
}

Route53 (ACM DNS Validation)

{
  "Effect": "Allow",
  "Action": [
    "route53:ChangeResourceRecordSets",
    "route53:ListHostedZones",
    "route53:GetHostedZone",
    "route53:ListResourceRecordSets"
  ],
  "Resource": [
    "arn:aws:route53:::hostedzone/*"
  ]
}

AWS Secrets Manager (Certificate Storage)

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:CreateSecret",
    "secretsmanager:PutSecretValue",
    "secretsmanager:GetSecretValue",
    "secretsmanager:DescribeSecret",
    "secretsmanager:UpdateSecret",
    "secretsmanager:TagResource"
  ],
  "Resource": "arn:aws:secretsmanager:*:*:secret:certmesh/*"
}

Vault AWS IAM Auth (STS)

{
  "Effect": "Allow",
  "Action": [
    "sts:AssumeRole",
    "sts:GetCallerIdentity"
  ],
  "Resource": "*"
}

Tip: For EKS deployments with IRSA, create a single IAM role with only the permissions you need, then reference it in your Helm values: serviceAccount.annotations."eks.amazonaws.com/role-arn".

Test Suite

• 800+ tests across the test suite (including provider route handler tests)
• 87%+ coverage (80% minimum enforced in CI)
• Tests use pytest, pytest-mock, responses, moto, and freezegun

CI

GitHub Actions runs on every push and PR:

Job	Matrix	Description
lint	Python 3.10 - 3.14	ruff check + ruff format --check
test	Python 3.10 - 3.14	pytest with coverage
build	Python 3.14	python -m build + twine check
Integration - Helm + kind	-	Full K8s deployment with Vault TLS, health probes, and capacity tests
Integration - Vault PKI	-	Vault PKI engine issue/sign/revoke against real Vault
Integration - Venafi (VCert)	-	VCert SDK tests against mock TPP
CodeQL	-	GitHub security analysis
auto-tag	-	Auto-version bump and tag on merge to main
publish-pypi	-	Publish to PyPI via trusted publisher

License

MIT -- see LICENSE for details.