Skip to main content

Tools for Simons Observatory work with spt3g_software

Project description

GitHub Workflow Status (branch) Documentation Status https://coveralls.io/repos/github/simonsobs/so3g/badge.svg?branch=master https://img.shields.io/badge/dockerhub-latest-blue PyPI Package

Glue functions and new classes for SO work in the spt3g paradigm.

Environment Setup

Before installing SO software, make sure you know what python environment you will be using:

  • Are you working on Linux or MacOS?

  • Are you using a “python3” executable provided by your OS or one provided by another source (Anaconda, conda-forge / miniforge, homebrew, macports, etc)?

  • Are you going to be actively developing so3g or just installing and using it?

After you have determined the answers to these questions, you can set up your working environment.

Using Conda

If you already have a conda installation (a.k.a. conda “base” or “root” environment) that is recent, then you can use that to create an environment. First, verify some info about your installation:

which python3
python3 --version
which conda

Your python version should be at least 3.7.0. Does the location of python3 match the location of the conda command (are they in the same bin directory)? If so, then you are ready. If you do not have conda installed but would like to use it, you might consider installing the “miniforge” root environment (https://github.com/conda-forge/miniforge) which is configured to get packages from the conda-forge channel. Conda-forge provides a large number of software packages built from high quality recipes.

The next step is to create a dedicated conda environment for your SO work. This allows us to create and delete these environments as needed without messing up the root environment:

conda create -n simons # <- Only do this once
conda activate simons

Now install as many dependencies as possible from conda packages. These are listed in a text file in the top of this git repo:

conda install --file conda_deps.txt

Using a Virtualenv

If you are using a python3 provided by your OS or somewhere else, you can work inside a “virtualenv”. This is like a sandbox where you can install packages for working on one project and you can always just wipe the directory and start over if something gets messed up. In general, you should never pip install packages directly to your OS or package manager location (e.g. pip installing as root on linux, or pip installing directly to /usr/local with homebrew). Doing so will create problems for the future on your system. We will create a virtualenv in our home directory:

python3 -m venv ${HOME}/simons # <- Just do this once
source ${HOME}/simons/bin/activate

Now install some basic packages and then all of our requirements:

python3 -m pip install --upgrade pip setuptools wheel
python3 -m pip install -r requirements.txt

Other Python Packages

If you will be using the pointing code in so3g, install pixell and qpoint with pip (regardless of whether you are using a conda env or virtualenv):

pip install pixell
pip install qpoint

Installing Pre-Built Wheels

If you are just using so3g (not modifying or developing it), then you can install the latest release of the package with:

pip install so3g

This command should be used regardless of whether you are working in a conda env or a virtualenv.

Now any time you activate your virtualenv / conda env, you can use so3g.

Building From Source

If you will be developing so3g or want more control over the build, then you should build from source. You will need to install boost, FLAC, a BLAS/LAPACK library, and the spt3g_software package. For this discussion, we will assume that you have git checkouts of spt3g_software and so3g in:

${HOME}/git/spt3g_software
${HOME}/git/so3g

And that you will be installing both spt3g_software and so3g into:

${HOME}/so3g

Adjust the instructions below if your situation is different. We also need to be able to load this install prefix into our shell environment. There are several ways of doing that. The first is to edit your bash shell resource file and add a shell function like this:

load_so3g () {
  # Load the appropriate python environment (edit this!)
  source "${HOME}/simons/bin/activate"

  # Location of our installed spt3g / so3g
  dir="${HOME}/so3g"

  # Prepend our executable search path
  export PATH="${dir}/bin:${PATH}"

  # Get the python major / minor version
  pyver=$(python3 --version 2>&1 | awk '{print $2}' \
    | sed -e "s#\(.*\)\.\(.*\)\..*#\1.\2#")

  # Put our python module into our search path
  export PYTHONPATH="${dir}/lib/python${pyver}/site-packages"

  # Prepend our executable path and shared library search path.
  # These lines are for linux:
  if [ -z ${LD_LIBRARY_PATH} ]; then
    export LD_LIBRARY_PATH="${dir}/lib"
  else
    export LD_LIBRARY_PATH="${dir}/lib:${LD_LIBRARY_PATH}"
  fi
  # These lines are for MacOS:
  #if [ -z DYLD_LIBRARY_PATH ]; then
  #    export DYLD_LIBRARY_PATH="${dir}/lib"
  #else
  #    export DYLD_LIBRARY_PATH="${dir}/lib:${DYLD_LIBRARY_PATH}"
  #fi
}

From a new shell, you can now run “load_so3g” to load your python stack and put your install prefix into your environment. If you use environment modules, see the README and example in the modules/ directory.

Special Note on Conda

If you are building spt3g / so3g from source, it is highly recommended that you install dependencies from OS packages, use the python3 from your OS or package manager, and use a virtualenv as described in the first section. If you use a conda-provided python, then you have two choices:

  1. Build boost from source using that python (so that boost-python works correctly).

  2. Install the conda package for boost (and other dependencies) and use conda provided compilers to build spt3g / so3g.

Both of these choices are beyond the scope of this README. Below we assume that you are using a virtualenv created with the system (or homebrew / macports) python3.

Prerequisites on Linux

The easiest approach in this case is to use your OS package manager. For example:

apt install \
libboost-all-dev \
libopenblas-openmp-dev \
libflac-dev

Make sure your python virtualenv is activated. Next, download and install spt3g_software (https://github.com/CMB-S4/spt3g_software). Check the major / minor version of your python (e.g. 3.7, 3.8 or 3.9). We use that information to install spt3g into the correct site-packages directory. Below we assume an install prefix of “${HOME}/so3g” and that we are using python3.9:

cd ${HOME}/git/spt3g_software
mkdir -p build
cd build
cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="gcc" \
  -DCMAKE_CXX_COMPILER="g++" \
  -DCMAKE_C_FLAGS="-O3 -g -fPIC" \
  -DCMAKE_CXX_FLAGS="-O3 -g -fPIC -std=c++11" \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
  -DPython_EXECUTABLE:FILEPATH=$(which python3) \
  -DPYTHON_MODULE_DIR="${HOME}/so3g/lib/python3.9/site-packages" \
  -DCMAKE_INSTALL_PREFIX="${HOME}/so3g" \
  ..
make -j 2 install

Prerequisites on MacOS

The so3g / spt3g_software does not seem to run on MacOS when built with the clang++ compiler (unit tests fail with a cereal error). Instead, we will use homebrew to install our dependencies and the latest gcc compiler tools:

brew install \
flac \
bzip2 \
netcdf \
sqlite3 \
boost-python3 \
gcc

Next, download and install spt3g_software. Ensure that your virtualenv is activated. Check the major / minor version of your python (e.g. 3.7, 3.8 or 3.9). We use that information to install spt3g into our virtualenv or conda environment. Below we assume that our environment is in our home directory in a folder called “simons” and that we are using python3.9. We further assume that the homebrew gcc version is called “gcc-11”. Also, this assumes that homebrew is installing things to /usr/local:

cd ${HOME}/git/spt3g_software
mkdir -p build
cd build
cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="gcc-11" \
  -DCMAKE_CXX_COMPILER="g++-11" \
  -DCMAKE_C_FLAGS="-O3 -g -fPIC" \
  -DCMAKE_CXX_FLAGS="-O3 -g -fPIC -std=c++11" \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
  -DBOOST_ROOT="/usr/local" \
  -DPython_EXECUTABLE:FILEPATH=$(which python3) \
  -DPYTHON_MODULE_DIR="${HOME}/so3g/lib/python3.9/site-packages" \
  -DCMAKE_INSTALL_PREFIX="${HOME}/so3g" \
  ..
make -j 2 install

Compilation and Installation

To compile and install the so3g package (assuming our same install prefix of $HOME/so3g), we need to point it to the spt3g build directory that we used previously. For example:

cd ${HOME}/git/so3g
mkdir -p build
cd build
cmake \
  -DCMAKE_PREFIX_PATH=${HOME}/git/spt3g_software/build \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
  -DPYTHON_INSTALL_DEST="${HOME}/so3g" \
  -DCMAKE_INSTALL_PREFIX="${HOME}/so3g" \
  ..
make -j 2 install

The definition of CMAKE_PREFIX_PATH must point to the build directory for spt3g, because cmake output there will be used to generate best compilation and/or linking instructions for Boost and other dependencies of spt3g/so3g.

Now you can run your “load_so3g” (or similar) command whenever you want to load your python stack and also the so3g install prefix.

Local configuration through local.cmake

Optional, site-specific parameters may be set in the file local.cmake. Lines declaring set(VARIABLE, value) should have the same effect as passing -DVARIABLE=value to the cmake invocation.

To change the destination directory for the installation, add lines like this one:

set(CMAKE_INSTALL_PREFIX $ENV{HOME}/.local/)
set(PYTHON_INSTALL_DEST $ENV{HOME}/.local/lib/python3.7/site-packages/)

To point cmake to the spt3g build directory, add a line like this one:

set(CMAKE_PREFIX_PATH $ENV{HOME}/code/spt3g_software/build)

Testing

The unit tests are not installed with the so3g package, so in order to run them you must have a git checkout of so3g (even if you installed so3g from a pre-built wheel).

After installing the so3g package, you can run the unit tests by passing the path to the test directory to the pytest command:

pytest /path/to/so3g/test

You can run specific tests by calling them directly:

python3 -m unittest /path/to/so3g/test/test_indexed

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

so3g-0.1.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

so3g-0.1.15-cp311-cp311-macosx_13_0_x86_64.whl (16.1 MB view details)

Uploaded CPython 3.11 macOS 13.0+ x86-64

so3g-0.1.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

so3g-0.1.15-cp310-cp310-macosx_13_0_x86_64.whl (16.0 MB view details)

Uploaded CPython 3.10 macOS 13.0+ x86-64

so3g-0.1.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.1 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

so3g-0.1.15-cp39-cp39-macosx_13_0_x86_64.whl (16.0 MB view details)

Uploaded CPython 3.9 macOS 13.0+ x86-64

so3g-0.1.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.1 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

File details

Details for the file so3g-0.1.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 24524c44ba94091a72832ee9e62fe6c64cff7bc9abf8313763c530ae13a91ac9
MD5 04f4c182edc8f5093d26b3a7d60e4c51
BLAKE2b-256 68bac3d7224f4f944b6e91f2457ac4c24320467ded4adca96b34dfe662e1ae9b

See more details on using hashes here.

File details

Details for the file so3g-0.1.15-cp311-cp311-macosx_13_0_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp311-cp311-macosx_13_0_x86_64.whl
Algorithm Hash digest
SHA256 5f96fafd4acd2d57a0080808929548663ca66cc00bd618c8002ab124acff1f0d
MD5 25259c71be7a9cfdcf654eeb7b507479
BLAKE2b-256 cd39e1846155d8c8ae4ac178b30ecdae83205d9694cf9bd7b53d388acb31770d

See more details on using hashes here.

File details

Details for the file so3g-0.1.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2d8e6b4c02e76a51f5277f23253658e5cea7cb36e4b145fb1a3cd8376bd1b803
MD5 a21e3f5771c58ea3d5d148e9d14398da
BLAKE2b-256 01708f6ab3a011fcb63ea6117311a28b685c6f79ae5b6c1f037d1767cffb6db1

See more details on using hashes here.

File details

Details for the file so3g-0.1.15-cp310-cp310-macosx_13_0_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp310-cp310-macosx_13_0_x86_64.whl
Algorithm Hash digest
SHA256 64f5d42b35142c61a9b15c97ee4c75e3df2bb8ce819fdd2acb72e7469f1be50e
MD5 f12af7b8d8682e6059f4f1cef62c810a
BLAKE2b-256 bd2fa79571589fc5e3a7001ee199cce41a7c8845485c20593a2e377e19044e3a

See more details on using hashes here.

File details

Details for the file so3g-0.1.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7be88f13afff2a175bc9ea4f50235dc16e6d69430dce7563b31190390d63917d
MD5 656f817707a9f1137c11d882375717aa
BLAKE2b-256 faa21b4336ba5e020cc8069bccaa13ea6edba02e272f5519bed0a5b9fede58d5

See more details on using hashes here.

File details

Details for the file so3g-0.1.15-cp39-cp39-macosx_13_0_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp39-cp39-macosx_13_0_x86_64.whl
Algorithm Hash digest
SHA256 72796d9c731aef233da6eb5779136b4211921b9f144179adec41ced3713590e3
MD5 9b3eb567a23c0047c4b84a54abeab3f9
BLAKE2b-256 87225279b2c14fd643bd24b527f76bf087ead13ab8575d489437599878a79986

See more details on using hashes here.

File details

Details for the file so3g-0.1.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for so3g-0.1.15-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b94b4bb891f9b88cd6e45f355dde3d338f8dd7d926cab6db469c4c10223905b7
MD5 9e2164120a7952b64bf101bf2ef86979
BLAKE2b-256 298d59a5c1fdfc0b04101e839d89a892b17b503f95f1a9d53495c7d4a29ed805

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page