Skip to main content

Pipeline for producing galaxy scaling relation plots.

Project description

SWIFT Pipeline

This repository stores the new SWIFT/EAGLE/COLIBRE pipeline. The old one is available at this url.

Rationale

We already have a pipeline; why do we need a new one? The current pipeline suffers from a number of problems.

  1. Code fragmentation: everyone has their own version of the pipeline that they have made small changes to, and we have no idea which version is being run. This can then lead to issues with interoperability, in particular between the pipeline outputs and the comparison script.
  2. Extensibility: the current pipeline requires everything to be merged into one master repository, and there is only 'one copy' of these files. This was fine when only a single model needed to be analysed, but now that we have EAGLE-XL/COLIBRE/BAHAMAS all running on SWIFT, different configurations (and different figures!) are required.
  3. Comparisons: the current method for generating comparison figures is not ideal. There is no webpage generated automatically, and lots of issues persist.

This has all occurred because at the start of the pipeline project we didn't really know what the scope should be. Now this has become more clear, we are ready to move to a more usable and stable code-base.

Planned Improvements

To address the above problems, we will create a new 'pipeline' (that effectively borrows all of the code from the old one, just with new 'stitching'). This will aim to:

  1. Have separation of code and configuration. There will be one main program, swift-pipeline, that will take your configuration as arguments to produce output. This differs from the current situation where the code must be ran from scripts within one directory.
  2. Have a first-class solution for creating comparisons. The output files generated from swift-pipeline will contain all of the information required to entirely re-generate the figures (including ones from 'scripts' like the star formation history). The API for these additional plotting scripts will enable us to plot multiple simulation lines on one figure.

An example set of configuration files is available in example.

New Script API

Additional plotting scripts, such as the one used for the density-temperature figure, now should conform to the following API and be runnable as:

python3 my_script.py \
  -s snapshot_0000.hdf5 snapshot_0001.hdf5 ... \ # These may be from different sims
  -c halo_0000.hdf5 halo_0001.hdf5 ... \  # Again different sims
  -d input_directory_one input_directory_two ... \ # Again, different sims
  -n name_one name_two name_three ... \ # Names for different sims (for legend)
  -o output_directory \
  -C config \ # Config directory containing master config.yml (for obs data and stylesheet)

For an example of how to implement this, please see the example in example/config/scripts.

New Pipeline API

The pipeline now can be run in two modes:

  1. Produce all plots and, importantly, the output line data for, a single snapshot, with the output line data now being stored in the input directory (i.e. next to the snapshot).
  2. Produce all plots comparing multiple simulations (including the new scripts through the API defined above) using the output line data.

Both of these produce webpages automatically that include all of the required data.

To run the pipeline, you now need to use a configuration file and directory. As noted above, one of these is provided in example. This is passed to the pipeline, which now acts as an executable, in the following way:

swift-pipeline -C ~/config \ # Your configuration directory (customised for sim suite)
  -c example_0000.properties \ # Name of your catalogue file
  -s snapshot_0000.hdf5 \ # Name of your snapshot file
  -i /path/to/your/snapshot \ # Path to snap directory containing properties as well
  -o ~/plots/output \ # Output directory to store HTML, etc. in.

This will then create /path/to/your/snapshot/data_0000.yml. Once you have performed this for several simulations, you can create a comparison webpage for them using:

swift-pipeline -C ~/config \ # Your configuration directory (customised for sim suite)
  -c example_0000.properties example_0000.properties \ # Name of your catalogue files
  -s snapshot_0000.hdf5 snapshot_0000.hdf5 \ # Name of your snapshot files
  -i /path/to/your/A/snapshot /path/to/your/B/snapshot \ # Path to both directories
  -o ~/plots/output \ # Output directory to store HTML, etc. in.

This elevates the comparisons to being 'first class' citizens - they are treated in the same way as the creation of the 'real' data.

Installation

To install the pipeline, you can use the python packaging manager, pip,

pip3 install swiftpipeline

This will make the swift-pipeline executable available.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

swiftpipeline-0.3.5.tar.gz (198.4 kB view details)

Uploaded Source

Built Distribution

swiftpipeline-0.3.5-py3-none-any.whl (203.1 kB view details)

Uploaded Python 3

File details

Details for the file swiftpipeline-0.3.5.tar.gz.

File metadata

  • Download URL: swiftpipeline-0.3.5.tar.gz
  • Upload date:
  • Size: 198.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.8.14

File hashes

Hashes for swiftpipeline-0.3.5.tar.gz
Algorithm Hash digest
SHA256 80208f7ca4b2b14e3083a8efb4b2c414fc90dd43a2442d8b0915975a8c7fafb9
MD5 4b2b373b67b754238793cfcc4fb3ddd4
BLAKE2b-256 e661c7f5483d9685cc9a476d5c474caf5107d9ca7a03c3e37d363f96e10c67c1

See more details on using hashes here.

File details

Details for the file swiftpipeline-0.3.5-py3-none-any.whl.

File metadata

File hashes

Hashes for swiftpipeline-0.3.5-py3-none-any.whl
Algorithm Hash digest
SHA256 d209c9a77fd4f12cfdfe3cbd16633c7c38e5d8e441eac96849fd9608f3daf411
MD5 43dbfb9d9d923cbf927f068de6e9c2a4
BLAKE2b-256 f0776437a7717b25db89f87eb5cdf06c00b2b577bf4e66601937700c761a8512

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page