dran 0.0.9.dev0

Data reduction and analysis of HartRAO 26m telescope drift scans

DRAN README

DRAN is a data reduction and analysis software pipeline developed to systematically reduce and analyze HartRAO's <http://www.hartrao.ac.za>_ 26m telescope <http://www.hartrao.ac.za/hh26m_factsfile.html>_ drift scan data. The software is a new development intended to replace the old LINES continuum data reduction and analysis program previously used at HartRAO. The program began as part of the PhD projects for Pfesesani Van Zyl and is now mantained by the author who is currently a postdoc at SARAO (the South African Radio AStronomy Observatory) <http://www.sarao.ac.za/>_.

Acknowledging DRAN

To acknowledge DRAN in a publication please cite van Zyl P. 2023 <https://ui.adsabs.harvard.edu/abs/2023arXiv230600764V/abstract>_`.

Program structure

This program is built using Python <http://www.python.org>_ 3.8 and consists of two interfaces:

• CLI - A command-line interface and,
• GUI - A graphical user interface.

The CLI handles the automated and semi-automated data reduction and analysis of drift scan files, whereas the GUI is more aligned with handling single file inspection and fitting. The GUI also has features that allow for simple data analysis of timeseries data generated by the CLI.

Requirements

All required packages are in the requirements.txt file supplied with the software.

.. note:: There is an issue with the installation of pyqt5 (the library required to run the GUI) on some machines while loading the requirements file during program installation. Thus it is highly recommended that the user utilizes an anaconda installation or a virtual environment of their choice when installing the program. Please also note that the software has only been tested on anaconda so using a different virtual enviroment server might not work.

Installation

The DRAN program was developed using the anaconda virtual environment. Running the code on a virtual environment is highly recommended to avoid package clashes, and the following installation shows how to do this using anaconda <https://www.anaconda.com/products/individual>_. Anaconda is a data science toolkit with a wide variety of open-source packages and and libraries under one hub for building powerful projects.

First you will need to download anaconda <https://www.anaconda.com/products/individual>_ and install it on your machine. Once installed you will create the dran virtual environment and activate it using the following commands

.. code:: bash

$ mkdir dran $ cd dran $ conda create --name dran --file requests.txt $ conda activate dran

Alternatively, you can use the pypi package manager

.. code:: bash

$ pip install dran

.. note:: The program has not been tested on pyenv or env yet. This is planned for future releases.

Data files

• File format The program assumes that all raw files to be processed are strictly in FITS <https://fits.gsfc.nasa.gov/fits_documentation.html> file format. After processing the results are then stored in an SQLite database.

• File reading A file can be read in through 2 main ways

1. The user explicilty gives the full path of the file / folder they want to process.
2. The user selects the file / folder using the available options through the use of a dialog box (e.g. in the GUI).

Quickstart

After the successful installation of all the required libraries you can now start with processing the data. To dive right in you can type the following command

.. code:: bash

$ dran -h

The above command prints out the set of options listed below,

.. code-block:: python

usage: dran [-h] [-db DB] [-f F] [-force FORCE] [-c C] [-b B] [-delete_db DELETE_DB] [-mfp MFP] [-keep KEEP] [-delete_from DELETE_FROM] [-conv CONV]

Begin processing HartRAO drift scan data

optional arguments: -h, --help show this help message and exit -db DB turn debugging on or off. e.g. -db on, by default debug is off -f F process file or folder at given path e.g. -f data/Hydr aA_13NB/2019d133_16h12m15s_Cont_mike_HYDRA_A.fits or -f data/HydraA_13NB or -f data/ -force FORCE force fit all drift scans y/n e.g. -force y. Default is set to n -c C initiate the command to run program. e.g. -c gui or -c run_auto_analysis or -c cmdl -b B initiate browser. e.g. -b docs or -b dash -delete_db DELETE_DB delete database on program run. e.g. -delete_db y -mfp MFP multi-file processing of data between two dates. e.g. -mfp fileList.txt -keep KEEP keep original plots while processing data. e.g. -keep y -delete_from DELETE_FROM in coordination eith the filename, use delete_from to delete a row from a database e.g. delete_from CALDB -conv CONV convert the database tables to csv. e.g. conv CALDB.db -saveLocs SAVELOCS Save fit locations. e.g. -saveLocs y , default is no -fitTheoretical FITTHEORETICAL Fit theoretical locations of the baseline, i.e. the FNBW locs. e.g. -fitTheoretical y , default is no. -applyRFIremoval APPLYRFIREMOVAL Apply or don't apply RFI removal e.g. -applyRFIremoval n, default is yes. -fitLocs FITLOCS Fit data at sepcified locations e.g. -fitLocs pathToFile -analyze ANALYZE ANALYZE ANALYZE Analyze results from database e.g. -analyze CALDB.db Jupiter_22ghz all/no -quickview QUICKVIEW get quickview of data e.g. -quickview y -db_rows DB_ROWS use all rows in the database, including special case data, for analysis., e.g. -db_rows all --version show program's version number and exit

Depending on the process you want to run, you can select one or more of the available options. If there are more options you would like implemented please email the author, details are listed below.

Data processing examples

to perform an automated data reduction process on a single file, use the following command

.. code:: bash

$ dran-auto -f path-to-file/filename.fits

if you want to set debuggin on

.. code:: bash

$ dran-auto -f path-to-file/filename.fits -db on

to delete both a database

.. code:: bash

$ dran-auto -f path-to-file/filename.fits -delete_db y

to process multiple chunks of data within seperate periods of the year you use the following command

.. code:: bash

$ dran -mfp file_list.txt

NB Please note that this assumes that there exists a file called fileList.txt in the current directory that has the full path to the folder containing the source fits files you want to process. This file also has a start and end date in "YYYYdDDD" format stipulating the data range you want to process.

to process all the data located in your directory, NB this is the directory that contains all the folders containing your fits files. For example

Jupiter_22GHz 2020d139 2020d139

processes files located in the Jupiter_22GHz directory starting with the 2020d139 to 2020d139 dates.

To view the web documentation guide of the software

.. code:: bash

$ dran-docs

To delete an entry made in a database

.. code:: bash

$ dran-docs -delete_from CALDB

To keep the plots of the raw data

.. code:: bash

$ dran-auto -f path-to-file/filename.fits -keep y

To convert the database tables to csv

.. code:: bash

$ dran-auto -conv y

Starting the program will bring up the menu shown below with options to select how you want to proceeed with your data reduction or analysis.

.. code-block:: bash

##################################################################

###### ###### ###### # #

# # # # # # # # #

# # ##### ###### # # #

# # # # # # # ##

###### # # # # # #

##################################################################

PROGRAM STARTED

SELECT OPTION:

---

1. Data reduction
2. Open GUI
3. Run command line analysis
4. Open documentation
5. Open dashboard
6. Exit program

Please select an option

Option Selected:

To get a quick view of the source file

.. code:: bash

$ dran-auto -quickview y

Caveats

Although the program tries to fit the data appropriately there are a few things the user needs to be aware of. The program tries its best to fit all the data (good/bad). However the onus is still on the user to check/insure the scans have been fit in a satisfactory manner by checking the results in the plots folder.

Author contacts

If you have any problems, questions, ideas or suggestions, please OPEN AN ISSUE <http://www.hartrao.ac.za/hh26m_factsfile.html>_