Spark Logs

A tool for downloading and managing logs from Dataproc Spark clusters.

Installation

pip install .

Usage

The tool is fully interactive by default and will prompt for any missing information:

# Fully interactive - will prompt for project (with selection list), cluster, and application
spark-logs

# Partially interactive - only prompts for cluster and application
spark-logs --project my-gcp-project

# Partially interactive - only prompts for application
spark-logs --project my-gcp-project --cluster my-cluster

# No prompts - directly downloads logs when all parameters are specified
spark-logs --project my-gcp-project --cluster my-cluster --app-id application_1234567890_0001

You can also use the tool for listing resources:

# List all available Google Cloud projects
spark-logs --list-projects

# List all clusters in a project
spark-logs --project my-gcp-project --list-clusters

# List all Spark applications for a specific cluster
spark-logs --project my-gcp-project --cluster my-cluster --list-apps

Optional Arguments

• --region: Specify the GCP region (default: us-central1)
• --output-dir: Directory to save logs (default: ./logs)
• --service-account-json: Path to service account JSON key file

Authentication

The tool uses Google Cloud authentication. If no service account is specified, it will use the default authentication methods (gcloud auth, environment variables, etc.).

To use a service account:

spark-logs --service-account-json path/to/service-account.json