novapipe 1.0.0

NovaPipe: An advanced, plugin-driven ETL CLI tool

NovaPipe

---

NovaPipe — lightweight, plugin-driven ETL & orchestration for Python.

🎯 Quickstart

Install from PyPI:

pip install novapipe

Create pipeline.yaml:

tasks:
  - name: extract
    task: extract_data
    params:
      source: "s3://my-bucket/input.csv"

  - name: transform
    task: transform_data
    params:
      input_path: "{{ extract }}"
    depends_on:
      - extract

  - name: load
    task: load_data
    params:
      path: "{{ transform }}"
    depends_on:
      - transform

Run it:

novapipe run pipeline.yaml \
  --var environment=prod \
  --summary-json summary.json \
  --metrics-port 8000

View a human-friendly report:

novapipe report summary.json

---

🚀 Features

Category	Features
Core CLI	init, run, inspect, describe, tutorial, dag, report, playground
Plugin Mgmt.	Scaffold, list (with filters), CI-template, version-gating
Pipeline Modeling	run_if/run_unless, branches, skip-downstream, retries, timeouts, ignore-failures
Execution Engine	Async/sync, layered parallelism, resource tagging, rate-limit
Templating & Context	Jinja2 with built-ins, multi-output unpacking, CLI vars, interactive REPL
Resource & Env	CPU/memory caps, rate limits, env injection
Observability	Structured logging, JSON summary, human report, Prometheus metrics
Testing & CI/CD	Unit & integration tests, plugin + repo CI templates

---

📖 Documentation

Full docs and examples are hosted on ReadTheDocs:

https://novapipe.readthedocs.io/

---

💡 Getting Help

• Ask questions on Discussions

• Report bugs via Issues

---

🤝 Contributing

1. Fork & clone
2. poetry install (or pip install -e .[dev])
3. Create a feature branch, commit, and open a PR
4. Ensure tests pass: pytest

See CONTRIBUTING.md & CODE_OF_CONDUCT.md for details.

---

📜 License

MIT © Muqtar Shaikh

1. Replace badge URLs with your actual GitHub org/repo and PyPI project names.
2. Link the Quickstart to real example pipelines in your examples/ folder.
3. Populate the “Features” table with any additional items as you add them.