Generate mermaid diagrams from Jenkins pipelines
Project description
jenkinsdiagrams
This script converts a folder of Jenkins files into a visual representation.
Whilst there are Jenkins plugins to view pipeline graphs, you have to enter a pipeline first. If you have many pipelines and don't know the top-level/entrypoint pipelines, you might like this tool. Visualise in the terminal, as textual diagrams, or render with your IDE/GitHub/Docusaurus etc.
Inputs
Specify a path to your project.
Jenkinsfile.*
and any files ending with .jenkinsfile
will be scanned
Options
With .jenkinsfile
s, the file isn't enough to state whether they are used as a top-level pipeline in Jenkins (vs being
called by a higher-level pipeline).
To create a more accurate visual representation of your pipelines, add this line to the top of your Jenkins files which are called directly from the Jenkins UI:
// jenkins2diagram:toplevel
Supported outputs
Here's a few options - I'm considering:
- Mermaid diagram
- Renders in GitHub .md previews
- Renders in Docusaurus (a static website tool)
Ascii art- Renders anywhere
Image files
I'll focus on mermaid first.
I think the app will return text to stdout, which can be piped >
by the user into a file if they wish.
Developing a POC (1/2)
This app is quite domain / keyword specific, so for this POC I aim to:
- Get the basic file reading, pipeline mappings, etc. correct with simplified syntax:
# Jenkinsfile
// jenkins2diagram:toplevel
build job: 'a'
build job: 'b'
// a.jenkinsfile
// b.jenkinsfile
build job: 'c'
// c.jenkinsfile
Should produce:
graph TD
Z[Jenkinsfile]
Z[Jenkinsfile] --> A[a]
Z[Jenkinsfile] --> B[b]
B[b] --> C[c]
Notes on GraphViz DOT and Mermaid
Note: To kickstart generating the tree, I used anytree. It can export to a Python dictionary, JSON, or "Dot" format. The DOT language is a grammar for Graphviz.
- IntelliJ
- recognised Mermaid in my code block, offered to install the plugin, and rendered correctly. Mermaid plays nice with GitHub and Docusaurus, as is already mentioned
- making graph.dot let the IDE suggest a plugin install. A sample graph renders nicely, and copy to clipboard/save to file work great
- GitHub markdown rendering
-
Including graphviz graphs is a little indirect (some html tags and custom marks seem to be necessary)
Apparently Graphviz is in C, and Mermaid is written in JS- and mermaid is a little prettier
-
- Docusaurus
- I only see documentation on Mermaid support - nothing for Graphviz
- Converting Graphviz DOT to Mermaid
- Kroki: A server that renders a bunch of different textual diagrams to images
Overall, I should probably aim to add a Mermaid exporter to anytree
. There's no issues/PRs for it so far.
anytree
's dotexporter.py
is ~ 400 lines, but more than half of it is examples.
I'll try a basic version in this POC first.
The mermaid tree generation is working nicely. Seeing that anytree can import/export trees as Python dictionaries helped me see I can move to a minimal dependency version, hopefully without too much challenge. That can happen later. I will try to raise a basic PR for a Mermaid exporter now. https://github.com/c0fec0de/anytree/pull/215 is the basic PR, with nice diagrams in the description!
Developing a POC (2/3)
- Change to support more proper syntax
- It seems about right for now
- Be more flexible with matches (
build(job: '...',...)
vsbuild job:'...'
, etc.))- Writing some unit tests for this
Developing a POC (3/3)
Create a CLI
As a user, I would like to:
- (Option)
--path PATH
to scan.- Default: current directory.
- Example:
jenkisdiagram --path ~/IdeaProjects/monorepo
.
- (Option)
--folders NAME
to scan, in addition to root.- Default: all folders.
- If it's too slow for large repositories, can change default to scan only root and
pipelines
.
- If it's too slow for large repositories, can change default to scan only root and
- Example:
jenkinsdiagram --folders infra/jenkins/pipelines-old --folders infra/jenkins/pipelines-new
- Default: all folders.
Typer is awesome
Calling the app with --help
will list the (all optional) arguments:
python .\main.py --help
Optional args, exceptions, etc. are all nicely formatted.
Arg parsing is done and the CLI feels nice. I want to publish to homebrew before
Pushing to HomeBrew
Packaging a Python CLI tool for Homebrew
pip install -r requirements.txt
PyPi publish first, then brew! https://medium.com/@joel.barmettler/how-to-upload-your-python-package-to-pypi-65edc5fe9c56 https://www.jetbrains.com/help/pycharm/creating-and-running-setup-py.html
python setup.py sdist twine upload dist/* Okay, that has published https://pypi.org/project/jenkinsdiagram/1.0.0/
it's just a .tar.gz of the repo :think: Can't seem to make it run (as an app) from terminalm, ah well let's jump to trying homebrew
Now we can try the homebrew stuff
Simon Willison's guide is good
https://github.com/IdiosApps/homebrew-jenkinsdiagram install instructions
Install wanted GitHub login for some reason. Fresh PAT wasn't working, but gh auth login
revalidate did the trick
brew list
shows jenkinsdiagram
but the command doesn't work, and neither does just jenkinsdiagram
install logs noted:
/opt/homebrew/Cellar/jenkinsdiagram/1.0.0/libexec/bin/pip
compare with brew tap simonw/datasette brew install simonw/datasette/datasette
/opt/homebrew/Cellar/jenkinsdiagram/1.0.0 has no bin
/opt/homebrew/Cellar/datasette/0.64.1 has a bin
> /opt/homebrew/Cellar/datasette/0.64.1/bin/datasette --version
datasette, version 0.64.1
Probably in the distribution side of things, but let's compare the homebrew repos: https://github.com/IdiosApps/homebrew-jenkinsdiagram/blob/main/Formula/jenkinsdiagram.rb https://github.com/simonw/homebrew-datasette/blob/main/Formula/datasette.rb Nothing crazy different
Let's compare one file we made to make a dist; https://github.com/simonw/datasette/blob/main/setup.py https://github.com/IdiosApps/jenkinsdiagram/blob/pypi-homebrew/setup.py
This looks interesting:
entry_points="""
[console_scripts]
datasette=datasette.cli:cli
""",
This looks like a reasonable thing to attempt!
Update setup.py with entry_point + version bump
sdist
push
pip install
poet -f
bump brew repo url + hash
...
now we have /opt/homebrew/Cellar/jenkinsdiagram/1.0.1/bin
!
Running it gives:
File "/opt/homebrew/Cellar/jenkinsdiagram/1.0.1/libexec/lib/python3.10/site-packages/jenkinsdiagram/main.py", line 8, in <module>
import app
ModuleNotFoundError: No module named 'app'
So we're getting closer, but still a bit more investigation to go.
need to declare imports in init?
may need a nice .main.py wrapping of the cli? https://github.com/simonw/datasette/pull/556. It seems fair:
➜ python -m datasette --version
python -m datasette, version 0.64.1
➜ datasette --version
datasette, version 0.64.1
Hmm, last couple commits had no impact - still get the same error, but at least I know I can get it in PyPi (earlier than brew)
python -m jenkinsdiagram --version
File "......./jenkinsdiagram/jenkinsdiagram/cli.py", line 8, in <module>
import app
ModuleNotFoundError: No module named 'app'
Test PyPi - we can push to an alternate repo to prevent cluttering the real PyPi project with bad dists
https://packaging.python.org/en/latest/guides/using-testpypi/
twine upload --repository testpypi dist/*
python3 -m pip install --index-url https://test.pypi.org/simple/ your-package
https://packaging.python.org/en/latest/tutorials/packaging-projects/
Other nice things
- CLI could output to SVG
- https://github.com/mermaid-js/mermaid-cli
- png and pdf work fine; svg needs some manual config setup (noted in CLI)
- user installs with
npm i -g
- and it's an official repo from the mermaid team
https://github.com/superj80820/mermaid-js-converter- sends out diagram to a URL
- https://github.com/mermaid-js/mermaid-cli
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for jenkinsdiagram-1.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82341a987910de20134be3d8519260b341f7d240944d73589938e480177c1712 |
|
MD5 | c2ff3288973436f88c88de5ee06786e9 |
|
BLAKE2b-256 | 3ed3105db77326591d23c6827f188ce546946dea1956966b97749e48e6cc721f |