quantum-pipeline 2.2.0

How it fits together

graph LR
    M[molecules.json] --> HF

    subgraph sim["quantum-pipeline (pip install)"]
        HF[PySCF<br/>Hartree-Fock reference] --> VQE[VQE loop<br/>Qiskit ansatz, scipy optimizer<br/>Aer on CPU or GPU]
    end

    VQE --> Out[PDF report +<br/>convergence plots]

    subgraph platform["data platform (docker compose, optional)"]
        K[Kafka<br/>one Avro message per finished run] --> S[Spark jobs<br/>scheduled by Airflow] --> I[(Iceberg tables<br/>on Garage)]
        I --> ML["ML module<br/>predict convergence from the first K steps"]
    end

    VQE -->|result with all iterations| K
    VQE -->|metrics| G[Prometheus + Grafana]

Point quantum-pipeline at a JSON file of molecules and it runs VQE for each one. PySCF supplies the Hartree-Fock starting point, Qiskit builds the ansatz, a scipy optimizer drives it, and Aer simulates the circuit on CPU or on an NVIDIA GPU. A run ends with a PDF report and a convergence plot per molecule.

Everything to the right of the solver is optional. With Kafka on, the finished result of each molecule, with every iteration in it, leaves the solver as one Avro message, and Spark jobs scheduled by Airflow land those messages in Iceberg tables.

Prometheus metrics and a Grafana dashboard with alert rules watch the run. The ML module, which reads the tables to predict from the first iterations whether a run will converge, is in progress and not part of a normal run.

What it does

• Runs VQE for every molecule in a JSON file, in sto3g, 6-31g or cc-pvdz
• Lets you pick the classical optimizer (L-BFGS-B, COBYLA, SLSQP, Nelder-Mead, Powell, BFGS, CG, TNC), the ansatz depth, the iteration cap and the convergence threshold
• Simulates with Qiskit Aer on CPU, or on a GPU through cuStateVec with the :gpu image (Docker Hub · GHCR)
• Writes a PDF report and PNG convergence plots to gen/
• Streams each finished run, with all its iterations, to Kafka as Avro, with a schema registry
• Turns the stream into Iceberg tables with Spark and Airflow
• Exports Prometheus metrics and ships a Grafana dashboard
• Work in progress: models that predict convergence and final energy from the first iterations of a run

Quick Start

pip install quantum-pipeline
quantum-pipeline -f molecules.json -b sto3g --max-iterations 100 --optimizer L-BFGS-B --report

Or with Docker:

docker pull straightchlorine/quantum-pipeline:cpu
docker run --rm straightchlorine/quantum-pipeline:cpu -f data/molecules.json -b sto3g --max-iterations 100

Both images are on Docker Hub as straightchlorine/quantum-pipeline:cpu and :gpu, and on GHCR as ghcr.io/straightchlorine/quantum-pipeline with the same tags.

The GPU image is built for one CUDA architecture (Ampere by default). For another GPU, rebuild it with the CUDA_ARCH build argument - see the GPU acceleration page.

The installation guide covers GPU acceleration; the deployment guide brings up the full platform with docker compose.

Documentation

Full docs at https://docs.qp.piotrkrzysztof.dev. Build them locally with:

pdm run mkdocs serve

Started as an engineering thesis, continued through a master's. Source on GitHub, mirrored on Codeberg.

License

MIT License. See LICENSE for details.