casatools 6.6.3.22

Astronomy data reduction library

CASAtools

If you are looking for the place to check out CASA with Git, this is not the right place. The repository to check out CASA can be found here. At some point in the future, this may form the underlying kernel for a future version of CASA, but at this point, this package is still in gestation.

CASAtools is a self-contained python module that provides the tools from the CASA project. This module only contains the non-GUI tools which are the SWIG bound C++ functionality from CASA.

CASAtools can be built and used with either Python 3.4 or 3.6. However, CASAtasks can only be built with Python 3. So if you plan to build CASAtasks, please ensure that you have a Python 3 interpreter available.

Maturity

This is alpha software in every sense of the word. Currently it builds on RedHat Enterprise Linux and OSX (RHEL6, RHEL7 and OSX 10.12) using the standard CASA development environment.

Building CASAtools

Install Dependencies on Red Hat

Those already working on CASA 5 can install all of the CASAtools dependencies with YUM:

-bash-4.2$ yum install casa-toolset-3

After doing this, you may need to add RedHat's Python 3 to your $PATH with:

-bash-4.2$ source /opt/rh/rh-python36/enable

In addition, Java 8 or greater is required. You can check this by looking for 1.8 in the Java version output:

-bash-4.2$ /usr/bin/java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
-bash-4.2$

Notable Dependencies

1. GNU Scientific Library version 2.2 or greater
2. SWIG version 3.0 or greater
3. OpenMPI version 1.10 required when parallelization is enabled

Install Dependencies on Macos

Install Casa 5 dependencies as described here: https://casa.nrao.edu/casadocs-devel/stable/casa-development-team/development-resources/build-configuration

Install Python 3

sudo port install python36
sudo port select --set python python36
sudo port select --set python3 python36
# Pip is not technically required but you will probably want it eventually
sudo port install py36-pip
sudo port select --set pip pip36
sudo port select --set pip3 pip36
sudo port select --set python python36

Checkout

Checkout the casa6 source code:

-bash-4.2$ git clone -q --recursive https://open-bitbucket.nrao.edu/scm/casa/casa6.git

Build

While CASAtools can be built with Python 2, CASAtasks requires Python 3 so it is probably best to build with Python 3 from the start. Make sure that -bash-4.2$ which python returns a path to the Python 3 executable you want to use because CASAtasks builds with this executable.

With the CASA build environment set up, the CASAtools module can be built like:

Linux:

-bash-4.2$ cd casa6/casatools
-bash-4.2$ autoconf
-bash-4.2$ ./configure
-bash-4.2$ ./setup.py build

Macos

cd casa6/casatools
autoconf && CC=/usr/bin/cc CXX=/usr/bin/c++ ./configure && ./setup.py build

A particular version of Python can be selected at configure time like:

-bash-4.2$ PYTHON_VERSION=3.4 ./configure

but as noted above, CASAtasks will use whatever version of python is in your path so it is best to configure your PATH to get a particular version of Python.

Alternate Developer Build For Linux

On Linux, one can alternately use make, which supports incremental and parallel builds that are useful for development. In this case, the steps are

-bash-4.2$ cd casatools
-bash-4.2$ autoconf
-bash-4.2$ ./configure
-bash-4.2$ ./setup.py genmake
-bash-4.2$ make [debug]

The ./setup.py genmake command creates a file named makefile with the make commands. In general, this file should only be modified if one knows what they are doing. Once the makefile has been generated, one can make changes to most source files (eg, .cc and .h files) and simply rerun the make command to generate an updated build. However, there are cases when earlier commands will have to be rerun. For example, if ac/templates/setup.py.in is modified, commands starting with autoconf will have to be rerun to generate a new setup.py file.

A non-exhaustive list of when ./setup.py genmake followed by make will need to be rerun includes:

• modification of cerberus files. These are copied by ./setup.py genmake; make does not deal with them
• changes in tool dependencies when modifying <tool>.xml files. If one does not change tool dependencies when modifying <tool>.xml files, they need only rerun make. However, if a new tool dependency is introduced or an existing dependency removed (eg via a function return value), one will have to start by rerunning ./setup.py genmake so the new dependency can propagate to the the makefile.

The debug parameter of make is optional. If provided, the resulting build is configured to use the -g option when compiling C and C++ code. Otherwise, the build is configured to be optimized using the -O2 option when compiling C and C++ files. The make command optionally takes the -j option with the number of parallel threads to use while building which can significantly improve build times on multi-core machines, e.g.:

-bash-4.2$ make -j8

To execute a parallel build with eight threads.

See If It Works

If the build completes successfully, try loading the CASAtools module:

-bash-4.2$ PYTHONPATH=build/lib.linux-x86_64-3.6 python3
Python 3.6.3 (default, Jan  9 2018, 10:19:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from casatools import image
>>> ia = image( )
>>> ia.fromshape("mytest.im",[20,20])
2019-03-25 16:15:08     INFO    ImageFactory::createImage       Created Paged image 'mytest.im' of shape [20, 20] with float valued pixels.
True
>>>
-bash-4.2$

CASAtools can be configured using ~/.casa/config.py. For example, extra data search paths can be added like:

-bash-4.2$ cat ~/.casa/config.py
datapath=[ "~/develop/casa/data/unittests" ]
-bash-4.2$

Run Available Tests

A number of tests have been brought into CASAtools from CASA. To run the tests, you need to check out the CASA test data repository needed for CASAtools unit tests. The full data repository will use about 100GB of space. You can check out the full data repository in the following way. Note that the --depth 1 will avoid downloading the past history of the repository and save a few Gbytes of space:

-bash-4.2$ git clone --depth 1 https://open-bitbucket.nrao.edu/scm/casa/casatestdata.git

After the checkout is complete, you must update your CASAtools RC file to indicate where the unit test data can be found. In my case it looks like:

-bash-4.2$ cat ~/.casa/config.py
datapath=[ "~/develop/casa/casatestdata/" ]
-bash-4.2$

The datapath list specifies directories where CASAtools should look for data files, and should include the directory we just checked out.

A few of the casatools tests use the casatestutils module. That is built independently from casatools. The path to a casatestutils build can be provided via PYTHONPATH prior to running the full suite of tests.

After all of this is set, the full suite of tests can be run with:

-bash-4.2$ PYTHONPATH=../casatestutils/build/lib ./setup.py test

There are about 28 tests in total. Running all of them takes about 48 minutes. The test results can be found in subdirectory of the casatools/build directory (e.g. casatools/build/testing.linux-x86_64-3.6). A separate directory is use for each tool test script.

To run a single set of test, you can just execute the test file:

-bash-4.2$ PYTHONPATH=build/lib.macosx-10.12-x86_64-3.6 python tests/tools/coordsys/test_coordsys.py

If the casatools module is already available in your PYTHONPATH, you would not need to explicitly specify it on the command line. This would be the case, for example, if you installed CASAtools via a binary PyPI wheel. If you want to run a single test from the set, you can specify it on the command line like:

-bash-4.2$ PYTHONPATH=build/lib.linux-x86_64-3.6 python tests/tools/coordsys/test_coordsys.py coordsys_test.test_constructor

Here, test_constructor is one test within the coordsys_test TestCase specified in test_coordsys.py.

Available Tools

Tool Name	Description
agentflagger	Tool for manual and automated flagging
atcafiller	Filler for ATNF/ATCA RPFITS data
atmosphere	Atmosphere model
calanalysis	Get and fit data from a calibration table (CASA 3.4 and later).
calibrater	Synthesis calibration (self- and cross-)
componentlist	A tool for the manipulation of groups of components
coordsys	Operations on CoordinateSystems
functional	Functionals handling
imagemetadata	Operations on imagemetadata
image	Operations on images
imagepol	Polarimetric analysis of images
imager	tool for synthesis imaging
iterbotsink	tool for synthesis imaging
logsink	tool for logsink
measures	measures tool
miriadfiller	Tool for the importmiriad task
msmetadata	Operations to retrieve metadata from a measurment set
ms	Operations on measurement sets
mstransformer	Tool to apply spw and frame transformations in MS
quanta	quanta tool handles units and quantities
regionmanager	Create and manipulate regions of interest
sakura	New single dish tool interface using sakura
sdm	Manipulate or examine SDM datasets
sidebandseparator	Tool for sideband separation
simulator	Tool for simulation
singledishms	New single dish tool interface to process an MS
spectralline	spectral line tool
synthesisdeconvolver	tool for synthesis imaging
synthesisimager	tool for synthesis imaging
synthesisimstore	tool for synthesis imaging
synthesismaskhandler	tool for mask handling in sysnthesis imaging
synthesisnormalizer	tool for synthesis imaging
synthesisutils	tool for synthesis imaging
table	Access tables from casapy
vlafiller	tool for VLA filler tasks
vpmanager	Tool for specifying voltage patterns and primary beams

Tool Initialization

The user initalization and customization file for the CASAtools module is ~/.casa/config.py. If this file does not exist, then an attempt is mad to import the RC values from casaconfig (i.e. from casatoolrc import *). If the casatoolrc module does not exist in PYTHONPATH, then the default values for all initialization state is used.

Changes from Standard CASA

While the goal was to simply reconstitute the CASA tools within a unencumbered python module, deviations were required as work progressed. These deviations are divided into categories based upon whether the change relates to the way the tools behave or the way the XML files are structured.

Changes to Behavior

1. Parameter Order --- with standard CASA, the SWIG binding is exposed directly to the user. While this is often fine, for CASAtools more control was desired over how parameter checking happens and how coersion happens. Because of this, a wrapper is created over the SWIG binding. This wrapper explicitly specifies the order of the parameters. The input parameters come first, followed by the output parameters. Within the input and output parameters, the order is determined by the XML file. With standard CASA, all parameters are largely unordered with binding controlled by the way python argument lists are handled.

2. Utils Tool --- the standard CASA uttool (AKA utils or just ut), has been converted from a toolbox/grab bag tool to a singleton object called ctsys in CASAtools, e.g.:

   In [1]: from casatools import ctsys
   

3. ctsys.resolve( ) --- a new member function was added to resolve the path to a data file based upon CASADATA path (as is done for <type mustexist="true">path</type>)

4. rc file --- the rc file, which is evaluated at startup to configure CASAtools, is ~/.casa/config.py; if this file is not found, then an attempt is mad to import the RC values from casatoolrc (i.e. from casatoolrc import *)

Xml Changes

1. String Constants --- (developer) default values for strings, should not include quotes. The standard CASA XML processing would strip out opening and closing quotes, but CASAtools XML processing does not.

2. Path Type Added --- (developer/automatic) a new parameter type called path was added to CASAtools. This path type evolved from the standard CASA convention of adding the mustexist attribute to string typed parameters. The automatic XML translation converts string parameters which include the mustexist attribute to path parameters. C++ receives the path parameters as a string, but when the mustexist attribut of the path parameter is set to true, the CASAtools binding layer will expand the required path using the CASADATA (colon separated) path list to locate the required file or directory. In the cases where the mustexist attribute was not used, developers must make any changes to convert a string parameter to a path parameter.

3. Type Element Added --- (developer) CASAtools adds a new <type> XML element which has precedence over the type attribute, e.g. <param type="...">. This addition allows types that can be passed to a any/variant param to be enumerated. This allows path strings to be expanded by the CASAtools binding layer when the path is part of an any/variant parameter. When more than one <type> is specified for a parameter, the parameter is assumed to be an any/variant parameter.

4. Bool --- (automatic) The majority of standard CASA XML files indicate boolean type parameters with bool. However, a small number of them use boolean. With CASAtools, only bool is accepted.

5. StringArray --- (automatic) The majority of standard CASA XML files indicate the vector of strings type parameters with stringArray. However, a small number of them use stringarray. With CASAtools, only stringArray is accepted.

6. Array Values --- (developer) With CASAtools, the behavior <value> elements for vector initialization has been rationalized:

   • <value/> --- empty vector (zero elements)
   • <value><value/></value> --- vector with one element initalized to the default initialization for the vector element type
   • <value><value>0.0</value></value> --- vector with one element initialized as specified, more internal <value> elements can be used to increase the default size of the vector

   It is not possible to automatically adjust these because with the old XML translation <value/> will sometimes result in a one element vector and other times a zero element vector.

7. Array Value Types --- There are now two Array types <type>Array and <type>Vec, e.g. doubleArray and doubleVec. Vec implies a one dimensional value, e.g. Python list, a one dimensonal numpy array, whereas Array implies potentially an N dimensional numpy array.

XML

Tool Specification

    <tool name="tool-name">
        <shortdescription>one-line description</shortdescription>
        <description>paragraph description</description>

        <needs>tool-name</needs>

        <code>
            <include>path to headerfile</include>
            <private>
                <include>path to headerfile</include>
            </private>
        </code>

        <method name="method-name">
            <shortdescription>one-line description</shortdescription>
            <description>paragraph description</description>
            <input>
                <param name="param-name">
                    <description>short description</description>
                    <type units="unit-name" mustexist="true">type-name</type>
                    <value>value-content</value>
                </param>
            </input>
            <output>
                <param name="param-name">
                    <description>short description</description>
                    <type units="unit-name" mustexist="true">type-name</type>
                    <value>value-content</value>
                </param>
            </output>

            <returns type="type-name"/>
        </method>
    </tool>

• Red text indicate unique names. They may be developer specified (e.g. tool-name), they may be predefined enumerations (e.g. unit-name), or they may be both (e.g. type-name).
• Blue XML elements or attributes indicates optional items (zero or one).
• Green XML elements can be repeated.
• Black XML elements or attributes must be supplied (once)

testing links

more information for testing