MyST Libre

Following the REES, myst-libre streamlines building ✨MyST articles✨ in containers.

• A repository containing MyST sources
• A Docker image (built by binderhub) in a public (or private) registry, including:
  • Dependencies to execute notebooks/markdown files in the MyST repository
  • JupyterHub (typically part of images built by binderhub)
• Input data required by the executable content (optional)

Given these resources, myst-libre starts a Docker container, mounts the MyST repository and data (if available), and builds a MyST publication.

[!NOTE] This project was started to support publishing MyST articles as living preprints on NeuroLibre.

Installation

External dependencies

[!IMPORTANT] Ensure the following prerequisites are installed:

• Node.js (For MyST) installation guide
• Docker installation guide

Install myst-libre

pip install myst-libre

Set up environment variables:

If you are using a private image registry and/or Curvenote CLI features, create a .env file in the project root and add the following:

DOCKER_PRIVATE_REGISTRY_USERNAME=your_username
DOCKER_PRIVATE_REGISTRY_PASSWORD=your_password
CURVENOTE_TOKEN=your_curvenote_api_token

The CURVENOTE_TOKEN is required for operations like curvenote submit, curvenote deploy, curvenote pull, etc. You can generate an API token from your Curvenote profile settings.

Quick Start

Import libraries and define REES resources

Minimal example to create a rees object:

from myst_libre.tools import JupyterHubLocalSpawner, MystMD
from myst_libre.rees import REES
from myst_libre.builders import MystBuilder

rees = REES(dict(
                  registry_url="https://your-registry.io",
                  gh_user_repo_name = "owner/repository"
                  ))

Other optional parameters that can be passed to the REES constructor:

• gh_repo_commit_hash: Full SHA commit hash of the gh_user_repo_name repository (optional, default: latest commit)
• binder_image_tag: Full SHA commit hash at which a binder tag is available for the "found image name" (optional, default: latest)
• binder_image_name_override: Override the "found image name" whose container will be used to build the MyST article (optional, default: None)
• dotenv: Path to a directory containing the .env file for authentication credentials to pull images from registry_url (optional, default: None)
• bh_image_prefix: Binderhub names the images with a prefix, e.g., <prefix>agahkarakuzu-2dmriscope-7a73fb, typically set as binder-. This will be used in the regex pattern to find the "binderhub built image name" in the registry_url. See reference docs for more details.
• bh_project_name: See this issue (optional, default: [registry_url without http:// or https://])

Note that in this context what is meant by "prefix" is not the same as in the reference docs. (optional, default: binder-)

Image Selection Order

1. If the myst.yml file in the gh_user_repo_name repository contains project/thebe/binder/repo, this image is prioritized.
2. If project/thebe/binder/repo is not specified, the gh_user_repo_name is used as the image name.

Note that if (2) is the case, your build command probably should not be myst build, but you can still use other builders, e.g., jupyter-book build.

If you specify binder_image_name_override, it will be used as the repository name to locate the image.

This allows you to build the MyST article using a runtime from a different repository than the one specified in gh_user_repo_name, as defined in myst.yml or overridden by binder_image_name_override.

The binder_image_tag set to latest refers to the most recent successful build of an image that meets the specified conditions. The repository content might be more recent than the binder_image_tag (e.g., gh_repo_commit_hash), but the same binder image can be reused.

Fetch resources and spawn JupyterHub in the respective container

hub = JupyterHubLocalSpawner(rees_resources,
                             host_build_source_parent_dir = '/tmp/myst_repos',
                             container_build_source_mount_dir = '/home/jovyan', #default
                             host_data_parent_dir = "/tmp/myst_data", #optional
                             container_data_mount_dir = '/home/jovyan/data', #optional
                             )
hub.spawn_jupyter_hub()

• MyST repository will be cloned at:

tmp/
└── myst_repos/
    └── owner/
        └── repository/
            └── full_commit_SHA_A/
                ├── myst.yml
                ├── _toc.yml
                ├── binder/
                │   ├── requirements.txt (or other REES dependencies)
                │   └── data_requirement.json (optional)
                ├── content/
                │   ├── my_notebook.ipynb
                │   └── my_myst_markdown.md
                ├── paper.md
                └── paper.bib

Repository will be mounted to the container as /tmp/myst_repos/owner/repository/full_commit_SHA_A:/home/jovyan.

• If a repo2data manifest is found in the repository, the dataset it names is expected to be already staged at:

tmp/
└── myst_data/
    └── my-dataset

Data is never downloaded automatically. A repository's binder/data_requirement.json can point at arbitrary sources, so running repo2data on any submitted repository would let it pull unreviewed content onto the build host. A build uses whatever is already present: if the directory is missing or empty, the build proceeds without the data mount and warns, so content that reads the dataset will fail to execute.

To fetch data once you have reviewed its source and destination, opt a single run in:

hub = JupyterHubLocalSpawner(rees_resources,
                             host_build_source_parent_dir = '/tmp/myst_repos',
                             allow_repo2data_download = True,  # off by default
                             )

The dataset directory can also be named explicitly for data you staged yourself:

rees_resources.dataset_name = "my-dataset"

Dataset names are validated: projectName becomes a path component on both the host and in the container, so absolute paths and names containing .. are rejected, and a name that resolves outside host_data_parent_dir (via a symlink, say) is refused.

When the data is present it is mounted read-only as /tmp/myst_data/my-dataset:/home/jovyan/data/my-dataset. If no data is declared, this step is skipped.

Build your MyST article

MystBuilder(hub).build()

Check out the built document

In your terminal:

npx serve /tmp/myst_repos/owner/repository/full_commit_SHA_A/_build/html

Visit ✨http://localhost:3000✨.

Table of Contents

• Myst Libre
  • Table of Contents
  • Installation
  • Usage
    • Authentication
    • Docker Registry Client
    • Build Source Manager
    • JupyterHub Local Spawner
    • MyST Markdown Client
  • Module and Class Descriptions
  • Contributing
  • License

Usage

Authentication

The Authenticator class handles loading authentication credentials from environment variables.

from myst_libre.tools.authenticator import Authenticator

auth = Authenticator()
print(auth._auth)

Docker Registry Client

The DockerRegistryClient class provides methods to interact with a Docker registry.

from myst_libre.tools.docker_registry_client import DockerRegistryClient

client = DockerRegistryClient(registry_url='https://my-registry.example.com', gh_user_repo_name='user/repo')
token = client.get_token()
print(token)

Build Source Manager

The BuildSourceManager class manages source code repositories.

from myst_libre.tools.build_source_manager import BuildSourceManager

manager = BuildSourceManager(gh_user_repo_name='user/repo', gh_repo_commit_hash='commit_hash')
manager.git_clone_repo('/path/to/clone')
project_name = manager.get_project_name()
print(project_name)

Module and Class Descriptions

AbstractClass

Description: Provides basic logging functionality and colored printing capabilities.

Authenticator

Description: Handles authentication by loading credentials from environment variables.
Inherited from: AbstractClass
Inputs: Environment variables DOCKER_PRIVATE_REGISTRY_USERNAME and DOCKER_PRIVATE_REGISTRY_PASSWORD

RestClient

Description: Provides a client for making REST API calls.
Inherited from: Authenticator

DockerRegistryClient

Description: Manages interactions with a Docker registry.
Inherited from: Authenticator
Inputs:

• registry_url: URL of the Docker registry
• gh_user_repo_name: GitHub user/repository name
• auth: Authentication credentials

BuildSourceManager

Description: Manages source code repositories.
Inherited from: AbstractClass
Inputs:

• gh_user_repo_name: GitHub user/repository name
• gh_repo_commit_hash: Commit hash of the repository

JupyterHubLocalSpawner

Description: Manages JupyterHub instances locally.
Inherited from: AbstractClass
Inputs:

• rees: Instance of the REES class
• registry_url: URL of the Docker registry
• gh_user_repo_name: GitHub user/repository name
• auth: Authentication credentials
• binder_image_tag: Docker image tag
• build_src_commit_hash: Commit hash of the repository
• container_data_mount_dir: Directory to mount data in the container
• container_build_source_mount_dir: Directory to mount build source in the container
• host_data_parent_dir: Host directory for data
• host_build_source_parent_dir: Host directory for build source
• allow_repo2data_download: Permit this run to fetch the dataset declared in the repository's binder/data_requirement.json (default False; see the data section above)
• container_network: Name of an existing Docker network to attach the spawned container to (default: Docker's default bridge; see "Restricting network access" below)
• verify_metadata_blocked: Refuse to spawn if the instance metadata service is reachable from container_network (default: True when container_network is set)
• metadata_probe_image: Minimal image used for that probe (default: busybox:latest)

Restricting network access from build containers

For a full server preparation runbook, see docs/server-setup.md.

Notebook code from a submitted repository executes inside the spawned container with whatever network access that container has. On the default bridge that includes the cloud instance metadata service — on OpenStack, 169.254.169.254, which serves user-data and config-drive contents.

Docker has no per-container egress ACL, so the block is applied with host firewall rules. Put build containers on their own network so the rules can target only that traffic:

docker network create --driver bridge \
  --opt com.docker.network.bridge.name=br-mystbuild \
  mystbuild

hub = JupyterHubLocalSpawner(rees_resources,
                             container_network = 'mystbuild',
                             )

Then block the metadata service for that interface. Use DOCKER-USER, which Docker evaluates before its own rules and does not overwrite:

# Instance metadata (OpenStack, and the same address on EC2/GCP)
iptables -I DOCKER-USER -i br-mystbuild -d 169.254.0.0/16 -j REJECT
# If IPv6 is enabled on the instance
ip6tables -I DOCKER-USER -i br-mystbuild -d fe80::a9fe:a9fe -j REJECT

These rules match traffic forwarded from the container, so the host's own access to metadata (cloud-init, etc.) is unaffected. They do not survive a reboot on their own — persist them with netfilter-persistent save or the equivalent for your distribution.

To also keep build containers off internal networks, add rules for the ranges you use, for example:

iptables -I DOCKER-USER -i br-mystbuild -d 10.0.0.0/8 -j REJECT

Verify from inside a build container before relying on any of this:

docker run --rm --network mystbuild curlimages/curl \
  -s -m 3 http://169.254.169.254/openstack/ ; echo "exit=$?"

A non-zero exit (timeout or refused) means the rule is working.

myst-libre also checks this itself: with container_network set, the spawner probes the metadata service from that network before building and refuses to start if it answers. The result is cached per network per process. This exists because the Docker network survives a reboot while DOCKER-USER rules do not, so "the network exists" is not evidence that it is protected.

Note: be careful before blocking 172.16.0.0/12 or the range holding your other Docker networks. In Docker-in-Docker mode myst-libre reaches the spawned Jupyter container by container IP, so blanket-blocking inter-container traffic will break the build.

MystMD

Description: Manages MyST markdown operations such as building and converting files.
Inherited from: AbstractClass
Inputs:

• build_dir: Directory where the build will take place
• env_vars: Environment variables needed for the build process
• executable: Name of the MyST executable (default is 'myst')
• state_file: Path to the process state file used for orphan recovery (default: <tmpdir>/myst_libre_processes.json)

Reaping orphaned build processes

A myst build launches a tree — myst → npm run start → node ./server.js — that holds ports until it is torn down. Normal teardown kills the whole process group, but a worker crash or restart leaves no PID to signal and the tree survives as an orphan.

myst-libre records each launched process group at spawn time, along with the process that owns it. Under Celery, hook it to celeryd_after_setup — once per worker node, after setup, before children fork or any task is consumed:

from celery.signals import celeryd_after_setup
from myst_libre.tools import MystMD

@celeryd_after_setup.connect
def reap_myst_orphans(sender, instance, **kwargs):
    reaped = MystMD.reap_orphans()
    if reaped:
        logging.warning(f"Reaped {len(reaped)} orphaned myst process group(s)")

Use celeryd_after_setup, not worker_process_init — the latter runs in every prefork child, so N reaps would race.

Concurrency safety. A record is only reaped when the process that launched it is gone. A build running right now in a sibling worker has a live owner and is left untouched — without that rule, a worker restart during a long build would kill healthy work, since a live build and an orphan are indistinguishable from the myst process alone. Because of this, reap_orphans() is safe to call at any time, including from a periodic task; a long-lived worker accumulates orphans from crashed children that no startup hook will see.

Orphan records are additionally re-checked against the recorded start-time key, so a recycled PID is dropped rather than signalled. Stale and dead entries are pruned on every call.

Ports are deliberately not used as the key here. With myst build --execute, mystmd finishes the entire execution phase before starting either server, so on a long build no port exists for most of its duration and both appear only near the end. The pgid is known at launch and stays valid throughout.