sparkgrep 0.1.0a1

Pre-commit hooks for Apache Spark development (Databricks, EMR, Dataproc, and more)

SparkGrep

Pre-commit hook that detects debugging leftovers in Apache Spark applications.

🎯 Purpose

SparkGrep helps maintain clean Apache Spark codebases by detecting common debugging leftovers and performance anti-patterns that developers often forget to remove before committing code.

🔍 What it Detects

• display() calls - Jupyter/Databricks debugging function
• .show() methods - DataFrame inspection calls
• .collect() without assignment - Potential performance issues
• .count() without assignment - Unnecessary computations
• Custom patterns - User-defined patterns via configuration

🚀 Installation

pip install sparkgrep

📋 Usage

As a Pre-commit Hook

Add to your .pre-commit-config.yaml:

repos:
  - repo: https://github.com/leandroasaservice/sparkgrep
    rev: v0.1.0a1  # Use this preview version.
    hooks:
      - id: sparkgrep

Command Line

# Check specific files
sparkgrep src/my_script.py notebook.ipynb

# Check with additional patterns
sparkgrep --additional-patterns "debug_print:Debug print statement" src/

# Disable default patterns and use only custom ones
sparkgrep --disable-default-patterns --additional-patterns "my_pattern:My description" src/

---

🛡️ Security & Quality

This project maintains high security and code quality standards:

🔒 Security Measures

• Daily security scans with Bandit, Safety, and GitGuardian
• Automated vulnerability detection and issue creation
• Admin-protected CI/CD pipelines
• Dependency vulnerability monitoring

📊 Code Quality

• 80% minimum code coverage enforced in CI
• SonarCloud integration for continuous code quality analysis
• Automated testing on every PR
• Code formatting with Ruff

---

📁 Project Structure

sparkgrep/
├── src/sparkgrep/          # Main package
│   ├── cli.py              # Command-line interface
│   ├── patterns.py         # Pattern definitions
│   ├── file_processors.py  # File processing logic
│   └── utils.py            # Utility functions
├── tests/                  # Test suite
│   ├── unit/               # Unit tests
│   └── integration/        # Integration tests
├── .github/                # GitHub configuration
│   ├── workflows/          # CI/CD pipelines
│   └── ISSUE_TEMPLATE/     # Issue templates
└── docs/                   # Documentation

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes with tests
4. Ensure all checks pass (task test, security scans)
5. Submit a pull request

Contribution Guidelines

• Tests required for all new features
• Security scans must pass
• Code coverage must remain ≥ 80%
• Admin approval required for all PRs to main
• Follow existing code style and patterns See CONTRIBUTING.md for details.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Project Docs

---

Made with ❤️ for the Apache Spark community