A Flexible Database for Multi-Dimensional Serial Crystallography

Documentation: We have extensive documentation in the form of GitLab pages on https://amarcord.pages.desy.de/amarcord/

Source Code: https://gitlab.desy.de/amarcord/amarcord

AMARCORD is a free and open-source database application designed to gather metadata about chemicals, runs and analysis results of serial crystallography (SX) experiments. The software features an easy-to-use, web-based user interface and was used in numerous experiments in order to cope with both the amount of data collected, as well as different experimental groups working during the same beam time. It was further extended not only to collect metadata, but also trigger analysis jobs and ingest the resulting figures of merit, resulting in a complete framework for running an SX experiment.

Python setup

uv

AMARCORD uses uv for managing its dependencies. So either install that and run:

uv venv

to create a venv with all dependencies installed. Running programs is then simply

uv run amarcord-<program-name> <arguments>

Plain pip

Since we have requirements.txt files, as long as you don’t want to add new dependencies, you can just create a virtual environment and activte it to get up and running:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Notes for Microsoft Windows users

We’re using python-magic to determine the type of uploaded files. This depends on libmagic which is not available on Windows. You can either do everything using WSL or you can manually pip install python-magic-bin which solves the issue.

If you manually created your virtual environment, the way to activate it on Windows isn’t

source venv/bin/activate

but rather

source venv/Scripts/activate

this is for idiotic reasons.

How to start a backend server

To start a backend server, you have to create a database first. This isn't done implicitly when starting the web server, since this created problems. It's easy to create one, however, just do this (yes, there are four / in the URL):

python amarcord/cli/upgrade_db_to_latest.py --db-connection-url 'sqlite+aiosqlite:////tmp/test.db'

To start a web server with a “blank”, but usable SQLite database in /tmp/test.db (adapt if you're on Windows), run:

DB_URL='sqlite+aiosqlite:////tmp/test.db' uvicorn --port 5000 amarcord.cli.webserver:app

which will open a web server on port 5000.

You won't see much when you point your browser to http://localhost:5000 though, because we haven't built the front-end yet, and we don't have a prebuilt version of the front-end in the repository. But, read on.

How to build and start the frontend

The frontend is written in Elm. You can download the elm compiler binary at the Install Elm web site.

If you just want to build the necessary files for the frontend, without being an active developer, you can simply navigate to the frontend/ directory and execute the ./build-elm-manually.sh script. It will put all the assets into frontend/output/build, which is the default search path for the web server, which you can run, described above. Then go to http://localhost:5000/index.html in your browser.

To run a live development environment, you need elm-live which, unfortunately, needs node.js. But you can install that really easily on different platforms.

Assuming you’ve got it installed, run:

npm install

To install the dependencies (you can also use Yarn, which might be more performant). Then, start a development server via:

./run-live-dev-env.sh

And point your browser to http://localhost:8000.

Nix

To speed up CI builds and unify dependencies, AMARCORD uses the Nix package manager. It also uses flakes, so enable those in your ~/.config/nix/nix.conf:

experimental-features = nix-command flakes

If you have Nix installed, building the AMARCORD Python package is as simple as

nix build '.#amarcord-python-package'

To build a Docker container:

nix build '.#amarcord-docker-image

To get a development shell with the Python dependencies:

nix develop

To get a development shell with the Elm dependencies:

cd frontend
nix develop '..#frontend'

For the CI build, we have an instance of gitlab-runner on cfeld-vm04 using the shell executor right now.