jentic-apitools-jobs 1.0.0a18

Jentic Apitools Jobs - Async job execution

Jentic API Tools - Jobs

Async job execution with pluggable backends for running pipeline operations in the background. Supports in-memory thread pool (default) and optional Redis/Celery distributed execution.

Key Features

The jobs package provides a TaskRunner abstraction with two implementations: ThreadPoolRunner for in-memory execution using a thread pool, and CeleryRunner for distributed execution via Redis and Celery. A create_task_runner factory selects the appropriate runner based on configuration. Job state is tracked through JobStorage (with MemoryJobStorage and Redis-backed implementations). Progress is reported through ProgressReporter objects that support SSE streaming to API clients. Periodic cleanup removes expired jobs and optionally their output directories. Supported JobOperation values are IMPORT, SCORE, IMPROVE, and QA; the corresponding processors (process_import, process_score, process_improve) live in jobs.processors and are dispatched by both the in-memory and Celery runners.

Dependencies

Internal: jentic.apitools.common, jentic.apitools.pipelines. Optional: redis, celery[redis].

Installation

pip install jentic-apitools-jobs

Quick Start

from jentic.apitools.jobs import create_task_runner
from jentic.apitools.common.models import JobOperation

runner = create_task_runner()
runner.storage.create_job("job-1", JobOperation.SCORE)
runner.submit_job("job-1", JobOperation.SCORE, {"spec_url": "https://example.com/api.json"})

Testing

pytest tests -v