jira-ingest

Async Jira data pipeline for Data Center and Cloud. Fetches projects, releases, boards, issues, and transitions; writes Parquet, CSV, or JSON Lines to local disk, S3, Azure Blob, or GCS; and optionally loads records into any SQLAlchemy-compatible database.

Features

• Dual-mode: Jira Data Center (Bearer PAT + optional mTLS) and Jira Cloud (Basic Auth)
• Async fetching with concurrent project processing, in-memory caching, and exponential-backoff retry
• Configurable custom fields extraction -- map any customfield_XXXXX to a logical name
• PII hashing for assignee and author fields
• Output formats: Parquet (Snappy), CSV, JSON Lines
• Output destinations: local filesystem, S3, Azure Blob, GCS via fsspec
• Pluggable database loader: PostgreSQL, Redshift (with S3 COPY fast path), Snowflake, DuckDB, SQLite
• Click CLI with run and validate commands
• Pydantic v2 settings and data schemas
• ruff + mypy strict + pre-commit + GitHub Actions CI

Quick start

pip install pipewell-jira-ingest
cp .env.example .env   # edit with your Jira URL and credentials
jira-ingest validate   # confirm connectivity
jira-ingest run        # fetch everything and write to ./output

For database loading, install the optional extra:

pip install "pipewell-jira-ingest[database]"   # PostgreSQL, SQLite, etc.
pip install "pipewell-jira-ingest[redshift]"   # Redshift with S3 COPY fast path

Documentation

Configuration reference

All settings are read from environment variables (or a .env file) with the prefix JIRA_.

CLI

jira-ingest run [OPTIONS]

  --env-file TEXT            Path to .env file  [default: .env]
  --start-date TEXT          Filter issues from date (YYYY-MM-DD)
  --end-date TEXT            Filter issues until date (YYYY-MM-DD)
  --date-suffix TEXT         Output file date suffix  [default: today]
  --database-url TEXT        SQLAlchemy URL to load into a database
  --db-schema TEXT           Target database schema
  --redshift-iam-role TEXT   IAM role ARN for Redshift S3 COPY
  --append                   Add to an existing --date-suffix's output
                              instead of replacing it (default: replace)

jira-ingest validate [OPTIONS]

  --env-file TEXT            Path to .env file  [default: .env]

Output layout

Each data type is written as a directory of part files, not one single file -- see Output sinks for why and for the --append / replace-on-rerun semantics:

{JIRA_SINK_URI}/
  issues/issues_{date}/part-<uuid>.parquet
  projects/projects_{date}/part-<uuid>.parquet
  releases/releases_{date}/part-<uuid>.parquet
  boards/boards_{date}/part-<uuid>.parquet
  transitions/transitions_{date}/part-<uuid>.parquet

Development

pip install -e ".[dev]"
pre-commit install
pytest