Heimdall
Installation
# Clone repository
git clone https://github.com/gkrieg/Heimdall && cd Heimdall
# Create conda env
conda create --name heimdall python=3.10 && conda activate heimdall
# Install dependencies
pip install torch==2.0.1+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
# Install Heimdall (in editable `-e` mode)
pip install -e .
Quickstart
train.py provides a clear overview of the inputs needed, how to prepare the data, model, optimizer, and run the trainer.
python train.py +experiments=cta_pancreas
Make sure to edit the global file config/global_vars.yaml based on your set up.
Sweeps
scripts/create_sweep.py has the arguments --experiment-name (the hydra experiment file name), --project-name (W&B project name), --fg and --fc which are the names of the hydra configs. It is a short script that will load in sweeps/base.yaml and updates it appropriately, and creates a sweep argument and returns it. This can work in tandem with deploy_sweep.sh to submit multiple sweeps on SLURM systems.
python scripts/create_sweep.py --experiment-name cta_pancreas --project-name Pancreas-Celltype-Classification
Dev Notes
Dev installation
pip install -r requirements.txt
Once the pre-commit command line tool is installed, every time you commit
some changes, it will perform several code-style checks and automatically
apply some fixes for you (if there is any issue). When auto-fixes
are applied, you need to recommit those changes. Note that this process can
take more than one round.
After you are done committing changes and are ready to push the commits to the
remote branch, run nox to perform a final quality check. Note that nox is
linting only and does not fix the issues for you. You need to address
the issues manually based on the instructions provided.
Cheatsheet
# Run cell type classification dev experiment with wandb disabled
WANDB_MODE=disabled python train.py +experiments=cta_pancreas
# Run cell type classification dev experiment with wandb offline mode
WANDB_MODE=offline python train.py +experiments=cta_pancreas
# Run cell cell interaction dev experiment with wandb disabled
WANDB_MODE=disabled python train.py +experiments=cta_pancreas
# Run cell cell interaction dev experiment with wandb disabled and overwrite epochs
WANDB_MODE=disabled python train.py +experiments=cta_pancreas tasks.args.epochs=2
# Run cell cell interaction dev experiment with user profile (dev has wandb disabled by default)
python train.py +experiments=cta_pancreas user=lane-remy-dev
Nox
Run code linting and unittests:
nox
Run dev experiments test on Lane compute node with CUDA (lane-shared-dev user profile):
nox -e test_experiments
Run fast dev experiments (only selected small datasets):
nox -e test_experiments -- quick_run
Run full dev experiments (including those datasets):
nox -e test_experiments -- full_run
Run dev experiments with a different user profile:
nox -e test_experiments -- user=box-remy-dev
Local tests
We use pytest to write local tests.
New test suites can be added under tests/test_{suite_name}.py.
Run a particular test suite with:
python -m pytest tests/test_{suite_name}.py
Run all tests but the integration test:
python -m pytest -m 'not integration'
Note: to run the integration test, you'll need to specify the Hydra user using a .env file.
The contents of the file should be like so:
HYDRA_USER=test
Turning off caching
To turn off dataset caching for dev purposes,
set cache_preprocessed_dataset_dir: null in config/global_vars.yaml.
Alternatively, pass cache_preprocessed_dataset_dir=null through the command
line, e.g.,
python train.py +experiments=cta_pancreas cache_preprocessed_dataset_dir=null
Distributed Pre-training
When training on the oven cluster if you don't set export NCCL_P2P_DISABLE=1 you can get the NCCL watchdog timeout errors.
Metadata
Release files for sc-heimdall 3.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sc_heimdall-3.0.0.tar.gz | 57.5 kB | Without provenance | Details for sc_heimdall-3.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sc_heimdall-3.0.0-py3-none-any.whl | Python 3 | none | any | Details for sc_heimdall-3.0.0-py3-none-any.whl |
Total release size: 221.1 kB
Release files / sc_heimdall-3.0.0.tar.gz
| Download URL | sc_heimdall-3.0.0.tar.gz |
|---|---|
| Size | 57.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5a59171c432a3b6f347a5a8961d6e405ec0b71848d0dac8f66d820350efb4627
|
|
BLAKE2b-256 checksum How to use checksums |
7b03ceca175dc7f976bc1350c4a1d608564300ae1a7947296b2174092bbcb785
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
python-httpx/0.27.2
|
Release files / sc_heimdall-3.0.0-py3-none-any.whl
| Download URL | sc_heimdall-3.0.0-py3-none-any.whl |
|---|---|
| Size | 163.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b5400ffce81da7213cc37c55b58f4fc0d36aeacec62d6422bbf5d330bf886e1f
|
|
BLAKE2b-256 checksum How to use checksums |
69bdb02590d4a8cbb6fff7e275220fb42a419b5db4a35d7e93843386b8a45cb0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
python-httpx/0.27.2
|
Security for sc-heimdall 3.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.