papaya-debugger 0.1.1

Intelligent debugging tool for Apache Spark jobs

---

Papaya is an intelligent debugging tool designed specifically for Apache Spark jobs.

It seamlessly integrates error monitoring, real-time notifications, and automated LLM-powered debugging to help debug your complicated Spark workflows.

• Real-time Monitoring: Continuously listens to your Apache Spark jobs and instantly detects errors and anomalies.
• Discord Alerts: Sends immediate notifications directly to your Discord channel whenever issues arise.
• Intelligent Debugging: Optionally analyzes your Spark errors and suggest automatic code fixes.
• Automated Pull Requests: Papaya automatically submits GitHub pull requests containing proposed code changes.

---

🛠️ Installation

Install Papaya via pip:

pip install papaya-debugger

---

🌟 Quick Start

Run Papaya by specifying the Spark UI URL:

papaya http://localhost:4040

To enable Discord notifications (requires DISCORD_TOKEN environment variable):

papaya http://localhost:4040 --discord-cid 123456789

To enable GitHub integration (requires GH_APP_TOKEN environment variable):

papaya http://localhost:4040 --github-repo myorg/myrepo

You can also adjust the polling interval (default is 0.5 seconds):

papaya http://localhost:4040 --poll 2.0

Note: You must set the GEMINI_API_KEY environment variable for Papaya to function.

---

📝 CLI Options & Environment Variables

Positional Arguments:

• SPARK_UI_URL (required): Spark UI URL of the active job to monitor (e.g., http://localhost:4040)

Optional Arguments:

• --discord-cid <int>: Discord channel ID to send messages to (requires DISCORD_TOKEN env var)
• --github-repo <OWNER/REPO>: GitHub repository of the Spark job (requires GH_APP_TOKEN env var)
• --poll <SECONDS>: Polling interval in seconds (default: 0.5)

Environment Variables:

• GEMINI_API_KEY: Required for Papaya to function (used for LLM-powered debugging).
• DISCORD_TOKEN: Required if using --discord-cid to send notifications to Discord.
• GH_APP_TOKEN: Required if using --github-repo to link to a GitHub repository.

---

📚 Documentation

Docs coming soon :)

---

🌐 Contributing

Contributions are welcome! Please open issues and pull requests on GitHub.

---