Fetch metadata from EPU / SerialEM and launch on-the-fly pre-processing
Project description
A simple app to fetch acquisition metadata from a running EPU session or SerialEM. It parses the first found xml/mrc (EPU) or mdoc file (SerialEM) associated with a data collection session and launches Relion or Scipion pipeline. In case of SerialEM you need to enable saving mdoc file for each movie.
Dependencies
Dependencies are installed from pip automatically:
pyqt5 (GUI)
numpy (to parse MRC headers)
emtable (for some Relion schedules scripts)
watchdog (watch folder when running in daemon mode)
Installation
You can install either using pip or from sources.
from pypi (recommended): pip install MDCatch
from sources - you have two options:
create python virtualenv:
python3 -m venv mdcatch
source mdcatch/bin/activate
git clone https://github.com/azazellochg/MDCatch.git
cd MDCatch
pip install -e .
create conda virtualenv (requires conda installed):
conda create -n mdcatch python=3.8
conda activate mdcatch
git clone https://github.com/azazellochg/MDCatch.git
cd MDCatch
pip install -e .
Configuration
Relion 3.1 or Scipion 3.0 is in in your PATH
Preprocessing templates: Schedules folder for Relion, template.json for Scipion
Edit config.py to adjust it to your setup
NOTE: You will need to edit the code in schedule.py if you do not use default provided Schedules
Testing
python -m unittest mdcatch.tests
Running
To run with a GUI simply type mdcatch. If you want to run in daemon mode, run mdcatch –watch (or better setup a daily cron job)
Working principle
GUI mode (default)
The idea is to run the app on a processing server once EPU/SerialEM starts data collection and the first movie is acquired. The server has to have access to both EPU session folder and movies folder, or to SerialEM movie folder.
find and parse the first xml/mdoc file, getting all acquisition metadata
create a Relion/Scipion project folder username_microscope_date_time inside PROJECT_PATH (or inside Scipion default projects folder)
create symlink for movies folder; copy gain reference, defects file, MTF into the project folder
modify existing Relion Schedules/Scipion templates, copy them to the project folder then launch Relion/Scipion on-the-fly processing
ACL Linux commands are executed for the project folder (so that uid has rwx permissions), where uid is obtained from DEF_USER
Daemon mode
From version 0.9.7 onwards it’s possible to run the app in fully automatic mode. It will run in the background recursively watching for new directories (directory name should start with PREFIX, e.g. lmb_username_myEpuSession) inside METADATA_PATH. Once an xml (EPU) or a mdoc (SerialEM) file is created in such folder, the default pipeline will launch. All subsequent steps are equivalent to the GUI mode.
Make sure you have set in config.py: DEF_USER, DEF_SOFTWARE, DEF_PIPELINE, DEF_PREFIX, METATADA_PATH.
Though all 3 pickers can be run fully automatically, Topaz and LogPicker will most likely require particle size / threshold adjustment, so crYOLO is preferred over other pickers.
It’s probably useful to setup a daily cron job for mdcatch –watch to detect new EPU/SerialEM sessions.
Screenshots
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.