julia-project 0.1.24

Manage Julia dependency in a Python module

julia_project

This package provides the class JuliaProject for managing a Julia project that lives inside a Python package. julia_project supports two libraries for calling Julia from Python, pyjulia (the Python module "julia") and juliacall.

julia_project is in pypi; it can be installed via pip install julia_project. It is meant to be used as a library in other projects.

julia_project is meant to provide some automation, hand holding, and error checking in managing a Julia dependency in a Python package.

For the user of a package that uses julia_project

Suppose the Python module mymodule uses julia_project to manage its Julia dependency. The user of mymodule can do the following to import mymodule and install and initialize the Julia project that mymodule depends on.

import mymodule
mymodule.project.ensure_init()

See the docstring for ensure_init for optional arguments. The author of mymodule may have already called ensure_init as step peformed when import mymodule is executed. In this case, calling ensure_init again is a no-op.

To compile, or recompile, the Julia project, the user calls mymodule.project.compile(). The compiled Julia system image will be used the next time mymodule is imported, speeding up both startup and the first execution of code.

Calling mymodule.project.clean() removes the compiled system image and some other files. This will force again resolving the Julia package requirements on the next import mymodule. Calling mymodule.project.clean_all() will remove the entire project tree. This is a kind of "factory reset". The next time you run project.ensure_init() a new directory will be created and populated with files from the installation directly.

Calling mymodule.project.update() checks for compatible updates of Julia packages that are direct or indirect dependencies of mymodule, and performs the update.

If you want to handle installation and initialization of the Julia project and packages yourself, you can do

import mymodule
mymodule.project.disable_init()

Then subsequent calls to ensure_init, explicit or otherwise will do nothing. project.enable_init() will enable initialization if it has been disabled.

If someone else has called mymodule.project.disable_init() and you want to override it, you can call mymodule.project.enable_init().

Choosing pyjulia or juliacall

Pass either "juliacall" or "pyjulia" as the argument calljulia to ensure_init. For example

import mymodule
myjuliamod.project.ensure_init(calljulia="juliacall")

Using julia_project to call Julia functions

A Python-package author can use find_julia to provide a custom interface to Julia resources. The author may provide an full-featured or thin interface. In any case it is sometimes useful to access the Julia/Python interface library directly. You can also get the imported Python library, either julia (i.e. pyjulia) or juliacall like this

myjuliamod.project.julia

For example, the Julia module Main may be accessed like this.

Main = myjuliamod.project.julia.Main
Main.sind(90) # 1.0

The semantics and syntax of Python modules julia and juliacall are quite different. But, julia_project provides a minimal common layer. For example,

Example = project.simple_import("Example")

imports the Julia module Example.

Some parts of managing the Julia project are particular to either pyjulia or juliacall. These are handled by the classes PyJulia and JuliaCall. And project.calljulia is an instance of one of these.

For the author of a package using julia_project

The intended use is as follows. You want to create a Python package that calls some Julia packages via pyjulia. You create a directory representing the top level of a Python package, with a setup.py and requirements.txt and the Python code in a directory mymodule. You create a file ./mymodule/Project.toml describing the Julia packages for the project. In a Python source file in ./mymodule/, you create an instance of julia_project.JuliaProject that manages the Julia project. Call this instance project and import it into mymodule. For example, in _julia_project.py, you might have project = julia_project.JuliaProject([args]). And in __init__.py of mymodule you have from _julia_project.py import project. (See the example directory).

What julia_project does

Then import mymodule; mymodule.project.ensure_init() will do the following

• Look for the Julia executable in various places using find_julia
• Offer to download and install Julia if it is not found.
• Optionally create a private Julia depot for mymodule to avoid possible issues with PyCall in different Python environments.
• Check that the julia (or juliacall) package is installed. I.e. check that PyCall, or PythonCall is installed and built, etc.
• Optionally download and install a Julia registry.
• Optionally load a custom Julia system image.
• Instantiate the Julia project.
• Provide a Python function that compiles a system image that will be found the next time mymodule is imported. The scripts and environment for compilation are found in a specified subdirectory of the Python project.
• Write info about all of the above to a log file

Using julia_project to create a project

Here is a brief example. See the example directory for a complete example.

• Include the following in a file loaded in ./mymodule/, that is, the directory found by import mymodule.

import os
mymodule_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# This just creates an object, but does none of the steps above.
project = JuliaProject(
    name="mymodule",
    package_path=mymodule_path,
    registries = {"MyModuleRegistry" : "[e-mail]:myuser/MyModuleRegistry.git"},
    logging_level = logging.INFO # or WARN, or ERROR,
    )

# If the following is omitted, the user of mymodule must call it explicitly.
project.ensure_init() # This exectutes all the management features listed above

• Create ./mymodule/Project.toml (or ./mymodule/JuliaProject.toml) for the Julia project.

Compiling

Make a folder ./mymodule/sys_image/. Add a file ./mymodule/sys_image/Project.toml (or ./mymodule/sys_image/JuliaProject.toml) This typically contains the same dependencies as the top-level Project.toml. Perhaps a few more or less. Add a script ./mymodule/sys_image/packages.jl containing an Array{Symbol} of packages to be included in the image.

[:APackage, :AnotherPackage]

• You don't need to include PyCall or PythonCall.

• Optionally include a file compile_exercise_script.jl that will passed as precompile_execution_file.

• After compiling, the system image file will be renamed from sys_julia_project.so (or dll, or dylib) to a name that includes the version of the julia exectuable that built it. The latter is the file name that will be searched for the next time you import mymodule.

• The project is compiled by calling the method JuliaProject.compile either explicitly or during the installation.

Arguments to JuliaProject

name,
package_path,
registries=None,
version_spec="^1.6",
strict_version=True,
sys_image_dir="sys_image",
sys_image_file_base=None,
calljulia="pyjulia",
env_prefix="JULIA_PROJECT_",
post_init_hook=None,
depot=None,
logging_level=None,
console_logging=False

• name -- the name of the module, e.g. "mymodule". Used only in the logger and the name of the system image.
• package_path -- path to the top level of mymodule.
• registry_url -- if None then no registry will be installed (other than the General registry, if not already installed.)
• version_spec -- A julia version compatibility specification. The julia executable must satisfy this specification.
• strict_version -- If True prerelease (development) versions of Julia are disallowed when applying version_spec.
• sys_image_dir -- the directory in which scripts for compiling a system image, and the system images, are found. This is relative to the top level of mymodule.
• sys_image_file_base -- the base name of the Julia system image. The system image file will be sys_image_file_base + "-" + a_julia_version_string + ".ext", where ext is the dynamic lib extension for your platform.
• calljulia -- The julia-from-python interface library. One of two Python packages "pyjulia" and "juliacall".
• env_prefix -- Prefix for environment variables to set project options
• depot -- If True, then a private depot in the mymodule installation directory will be used.
• post_init_hook -- A function that will be called immediately before ensure_init returns.
• logging_level -- if None then no logging will be done. if logging.INFO, then detailed info will be logged
• console_logging -- if True, then the log messages are echoed to the console.

Environment variables

• In the following, the prefix JULIA_PROJECT_ may be changed with the argument env_prefix described above. This allows you to set environment variables specific to each project that do not interfere.

• JULIA_PROJECT_JULIA_PATH may be set to the path to a Julia executable. This will override other possible paths to a Julia executable.

• JULIA_PROJECT_INSTALL_JULIA may be set to y or n. If set, then no interactive query is done to install Julia via jill.py. Instead the value y or n is used.

• JULIA_PROJECT_COMPILE may be set to y or n. If set, then no interactive query is done to compile a system image after installing packages. Instead the value y or n is used.

• JULIA_PROJECT_LOG_PATH may be set to the path to the log file.

• JULIA_PROJECT_DEPOT -- If set to y, then a private Julia depot will be created in a directory depot under the mymodule installation directory. The depot contains all downloaded registries, packages, precompiled packages, and many other data related to your julia installation. Set to n to use the standard depot. If it is unset, you may be prompted for your choice.

Location of julia executable

JuliaProject will look in the following locations, in order

• The environment variable JULIA_PROJECT_JULIA_PATH. With JULIA_PROJECT_ optionally replaced by env_prefix described above.

• In the package top level for the installation ./julia/

• A julia installation from jill.py, with preferred versions specified as above.

• Your system or shell PATH variable.

• A fresh installation of julia via jill.py after asking if you want to download and install.

Building PyCall

Installing and using PyCall is sometimes easy and sometimes confusing. The latter happens if you try to use PyCall with different Python environments. The whole issue can be avoided by using a private, or Python-package-specific Julia "depot". Any of the following will create and use such a depot.

• Enable a new depot by passing the argument depot=True when initializing your JuliaProject instance.

• The user can set the environment variable JULIA_PROJECT_DEPOT described above.

• If no PyCall.jl is found, the option to create the package-specific depot will be given.

• If there is a libpython conflict detected during installation you will be prompted to create a depot.

The new depot will be used each time mymodule is imported. Remove or rename the directory mymodule/depot to prevent this.

When using a new depot, registries, packges, cached precompiled files, and many other things are stored in the installation directory of the project, e.g. mymodule.

This is a heavy solution because it involves duplicating many files if you use Julia for other projects, with Python or not. But, it does not require that the end user understand anything about the status of your Julia installation, libpython, PyCall.jl, etc.

Using a private depot should also allow julia_project to work with conda environments.

Testing

TESTS ARE OUTDATED!

You can run tests like this:

pytest -p julia.pytestplugin  julia_project/tests

Tests that don't require a Julia installation may be run like this:

pytest --no-julia  -p julia.pytestplugin  julia_project/tests

Warning

This package is very new and is neither well tested nor documented.