deprun 0.4.0

A modern Python task automation and multi-repository orchestration tool

Job Runner

Multi-repository task automation with intelligent dependency management.

Features

• Dependency Resolution: Automatically handles job dependencies with circular dependency protection
• Template System: Reusable job templates with inheritance
• Git Integration: Clone and manage multiple repositories
• Task-Level Execution: Run specific tasks within jobs
• Dependency Graphs: Visualize job dependencies with Mermaid
• Conditional Configuration: Use when clauses for dynamic configs
• Environment Management: Isolated environment variables per job

Installation

pip install job-runner

For development:

pip install job-runner[dev]

Quick Start

1. Create a jobs.yml file:

variables:
  build_dir: /tmp/builds
  user: ${USER}

jobs:
  build-frontend:
    type: build
    description: Build the frontend application
    repo:
      server: https://github.com/
      group: myorg/
      name: frontend
    directory: ${build_dir}
    script:
      - npm install
      - npm run build

  build-backend:
    type: build
    description: Build the backend service
    repo:
      server: https://github.com/
      group: myorg/
      name: backend
    directory: ${build_dir}
    script:
      - go mod download
      - go build -o server

  deploy:
    type: run
    description: Deploy all services
    dependencies:
      - build-frontend
      - build-backend
    script:
      - echo "Deploying services..."
      - ./deploy.sh

2. Run a job:

# Run a job with all its dependencies
job-runner run deploy

# Run a specific task within a job
job-runner run build-frontend test

# Run multiple tasks in sequence
job-runner run build-frontend clean,default,test

# Use "default" to refer to the main script
job-runner run build-frontend default    # Same as: job-runner run build-frontend

# Limit dependency depth
job-runner run deploy --depth 1

# Quiet mode: only show script output (useful for piping)
job-runner run build-frontend --quiet
job-runner run build-frontend -q | grep "Build complete"

# Fetch (clone) a repository without running scripts
job-runner fetch build-frontend

# Run all jobs (or filtered subset)
job-runner run-all                        # Run all jobs
job-runner run-all --type build           # Run all build jobs only
job-runner run-all --pattern "lib*"       # Run jobs matching pattern
job-runner run-all -t build -p "amx*"     # Combine filters
job-runner run-all --dry-run              # Preview without executing

3. List available jobs:

job-runner list

4. Visualize dependencies:

# Generate graph for a specific job
job-runner graph deploy

# Generate graph for all jobs (no duplicates)
job-runner graph

# Specify output file
job-runner graph --output my-deps.md
job-runner graph deploy -o deploy-deps.md

Configuration

Job Types

• build: Clone a repository and run build scripts
• run: Execute scripts in a specified directory

Templates

Create reusable job configurations:

templates:
  python-build:
    type: build
    script:
      - python -m pip install -r requirements.txt
      - python -m pytest
      - python -m build

jobs:
  my-python-app:
    template: python-build
    repo:
      server: https://github.com/
      group: myorg/
      name: my-app

Conditional Configuration

Use when clauses for dynamic configurations:

jobs:
  deploy:
    type: run
    script:
      - echo "Base script"
    when:
      - condition: "variables.get('ENVIRONMENT') == 'production'"
        data:
          script:
            - echo "Production deployment"
            - ./deploy-prod.sh

Tasks

Define multiple tasks within a job:

jobs:
  myapp:
    type: build
    repo:
      server: https://github.com/
      group: myorg/
      name: app
    script:
      - make build
    tasks:
      test:
        script:
          - make test
      clean:
        script:
          - make clean
      deploy:
        dependencies:
          - myapp:test
        script:
          - make deploy

Run specific tasks:

# Run a single task
job-runner run myapp test

# Run multiple tasks in sequence
job-runner run myapp clean,default,test

# The "default" task refers to the job's main script
job-runner run myapp default              # Runs: make build
job-runner run myapp clean,default        # Runs: make clean, then make build

# Run tasks with dependencies
job-runner run myapp deploy               # Runs test first, then deploy

CLI Commands

Command	Description
job-runner list	List all available jobs
job-runner run JOB [TASKS] [OPTIONS]	Run a job or specific tasks (comma-separated)
job-runner run-all [OPTIONS]	Run all jobs or filtered subset (see options below)
job-runner fetch JOB	Clone repository for build job without running scripts
job-runner info JOB [FIELD]	Show job information
job-runner validate	Validate configuration
job-runner dump JOB	Export job definition
job-runner graph [JOB]	Generate dependency graph (specific job or all jobs)

run Options

Option	Description
--depth	Maximum dependency depth
--quiet, -q	Suppress all output except script output (useful for piping)

run-all Options

Option	Description
--type, -t	Filter by job type: build or run
--pattern, -p	Filter by name pattern (wildcards: *, ?)
--depth	Maximum dependency depth
--dry-run	Preview execution without running

Advanced Features

Environment Variables

jobs:
  myapp:
    type: run
    env:
      NODE_ENV: production
      DEBUG: "false"
    script:
      - npm start

Variable Substitution

variables:
  version: "1.0.0"
  output: /tmp/builds

jobs:
  build:
    script:
      - echo "Building version {version}"
      - cp output {output}/app-{version}

Multiple Configuration Files

Organize jobs across multiple files:

# jobs.yml
jobs-dir: ./jobs

# jobs/frontend.yml
jobs:
  frontend:
    type: build
    # ...

# jobs/backend.yml
jobs:
  backend:
    type: build
    # ...

Development

# Clone the repository
git clone https://gitlab.com/proj_amx_01/tools/job-runner.git
cd job-runner

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black deprun tests

# Lint
ruff check deprun tests

# Type check
mypy deprun

# Build wheel package
python -m build

# Or use the Makefile
make build

# Clean build artifacts
make clean

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Credits

Created by Peter De Herdt

Support

• Documentation
• Issue Tracker