flux-python 0.47.0rc1

Python bindings for the flux resource manager API

Flux Python Bindings

🐍️ You called me?

Hello! You've found the flux Python bindings, an experiment to build and deploy Flux to Pypi without needing to store code alongside Flux. The goal of this experiment is to test them separately. This was originally developed at vsoch/flux-python and has been ported here to automate more officially. The following guides might be useful to you:

• ⭐️ Flux Framework Documentation
• ⭐️ Flux Projects
• ⭐️ Tutorials
• ⭐️ Installing Flux Python on an LLNL System

Development

We provide a .devcontainer environment you can open in VSCode to have an environment ready to go with Flux (and Flux Security). You can follow the instructions in the DevContainer to build the Flux Python bindings. By default, this environment installs the latest flux-core. If you want to build a custom version with Flux core you can do:

rm -rf ~/flux-core
export FLUX_VERSION=0.48.0
wget https://github.com/flux-framework/flux-core/releases/download/v${FLUX_VERSION}/flux-core-${FLUX_VERSION}.tar.gz
tar -xzvf flux-core-${FLUX_VERSION}.tar.gz
sudo mv flux-core-${FLUX_VERSION} ~/flux-core
rm flux-core-${FLUX_VERSION}.tar.gz
cd ~/flux-core
./configure --prefix=/usr/local
make
sudo make install

And then copy over the Python source to "flux" and build your custom archives and/or wheels:

$ rm -rf /workspaces/flux-python/flux
$ cp -R src/bindings/python/flux /workspaces/flux-python/flux
$ cd /workspaces/flux-python
# Archive
$ python3 setup.py sdist

# Wheel (will build for the python version you've targeted)
# This isn't currently being done but is a future TODO!
$ python3 setup.py sdist bdist_wheel

And if you want to upload:

$ twine upload dist/*.tar.gz

Install on a System

Since we need to link to Flux libraries, you are advised to install flux and flux-security in the same location that will be discovered via the executable "flux," so typically /usr or /usr/local. You can do either:

# Find the flux version on your system
flux --version

# Install the bindings that match that version
pip install flux-python==0.48.0

And if you are having trouble with the WCI cache, you can also wget the file directly from pypi and install the .tar.gz directly.

wget https://files.pythonhosted.org/packages/25/fb/02951d80e44a19db291f0e7370d4e7d82c0c1b17709a37913881f958dff7/flux-python-0.48.0rc0.tar.gz
pip install flux-python-0.48.0rc0.tar.gz

If you install to a local (personal) location (e.g., $HOME/.local) you'll need to add this to your PYTHONPATH

# The directory "flux" is under the site-packages here
export PYTHONPATH=$HOME/.local/lib/python3.7/site-packages

Building Modules

We will need to build the tarball providing paths to the flux-core and flux-security sources. This can be improved upon to just be one path if all the dependencies are provided with the flux install (and we don't need the source). Note that we also provide a custom version for the pypi package:

# Generate the wheel (requires pip install wheel)
$ python3 setup.py sdist bdist_wheel --flux-root /home/vscode/flux-core --security-src /home/vscode/security --security-include /usr/local/include/flux/security --version 0.46.0-rc-0

You can then install the wheel (as a user or to the root)

$ pip install dist/flux-0.46.0-cp38-cp38-linux_x86_64.whl --user
$ sudo pip install dist/flux-0.46.0-cp38-cp38-linux_x86_64.whl 

Processing ./dist/flux-0.46.0-cp38-cp38-linux_x86_64.whl
Requirement already satisfied: cffi>=1.1 in /usr/lib/python3/dist-packages (from flux==0.46.0) (1.14.0)
Installing collected packages: flux
Successfully installed flux-0.46.0

If the development container you are using still installs the Python bindings, you'll want to do the sudo variant above to override. And then start a flux instance:

$ flux start --test-size=4

And import flux.

$ ipython

import flux
flux.Flux()

Automation

In order to (eventually) support a workflow to check for a release, get the release's flux assets and build, we have a workflow script that will:

• Get the latest flux releases
• Compare to the latest pypi releases
• If there is a new release (from Flux on GitHub) not in pypi, build and release

And the script also can take an optional version string that will find and build a version explicitly. You might (to test) do:

$ export GITHUB_TOKEN=xxxxxxxxxxxxx
$ python .github/scripts/check_releases.py flux-framework --version 0.46.0

That basically checks if there should be a build. It will use the container provided here to install a custom version of flux core that builds the release.

Work in Progress

@vsoch wants to try installing her test releases to a system somewhere, and figure out this issue with buidling a wheel (or maybe we don't want wheels after all):

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Binary wheel 'flux_python-0.46.0rc0-cp38-cp38-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.

Release

SPDX-License-Identifier: LGPL-3.0

LLNL-CODE-764420