Data reduction and analysis of HartRAO 26m telescope drift scans
Project description
DRAN README
DRAN is a data reduction and analysis software pipeline developed to systematically reduce and analyze
HartRAO's <http://www.hartrao.ac.za>
_ 26m telescope <http://www.hartrao.ac.za/hh26m_factsfile.html>
_ drift scan data. The software is a new development intended to replace the old LINES continuum data reduction and analysis program previously used at HartRAO. The program began
as part of the PhD projects for Pfesesani Van Zyl and is now mantained by the author
who is currently a postdoc at SARAO (the South African Radio AStronomy Observatory) <http://www.sarao.ac.za/>
_.
Acknowledging DRAN
To acknowledge DRAN in a publication please cite van Zyl P. 2023 <https://ui.adsabs.harvard.edu/abs/2023arXiv230600764V/abstract>
_`.
Program structure
This program is built using Python <http://www.python.org>
_ 3.8 and consists of two interfaces:
- CLI - A command-line interface and,
- GUI - A graphical user interface.
The CLI handles the automated and semi-automated data reduction and analysis of drift scan files, whereas the GUI is more aligned with handling single file inspection and fitting. The GUI also has features that allow for simple data analysis of timeseries data generated by the CLI.
Requirements
All required packages are in the requirements.txt file supplied with the software.
.. note:: There is an issue with the installation of pyqt5 (the library required to run the GUI) on some machines while loading the requirements file during program installation. Thus it is highly recommended that the user utilizes an anaconda installation or a virtual environment of their choice when installing the program. Please also note that the software has only been tested on anaconda so using a different virtual enviroment server might not work.
Installation
The DRAN program was developed using the anaconda virtual environment.
Running the code on a virtual environment is highly recommended to avoid
package clashes,
and the following installation shows how to do this using
anaconda <https://www.anaconda.com/products/individual>
_.
Anaconda is a data science toolkit with a wide variety of open-source packages and
and libraries under one hub for building powerful projects.
First you will need to download anaconda <https://www.anaconda.com/products/individual>
_ and install it on your machine.
Once installed you will create the dran virtual environment and activate it
using the following commands
.. code:: bash
$ mkdir dran $ cd dran $ conda create --name dran --file requests.txt $ conda activate dran
Alternatively, you can use the pypi package manager
.. code:: bash
$ pip install dran
.. note:: The program has not been tested on pyenv or env yet. This is planned for future releases.
Data files
-
File format The program assumes that all raw files to be processed are strictly in
FITS <https://fits.gsfc.nasa.gov/fits_documentation.html>
file format. After processing the results are then stored in an SQLite database. -
File reading A file can be read in through 2 main ways
- The user explicilty gives the full path of the file / folder they want to process.
- The user selects the file / folder using the available options through the use of a dialog box (e.g. in the GUI).
Quickstart
After the successful installation of all the required libraries you can now start with processing the data. To dive right in you can type the following command
.. code:: bash
$ dran -h
The above command prints out the set of options listed below,
.. code-block:: python
usage: dran [-h] [-db DB] [-f F] [-force FORCE] [-c C] [-b B] [-delete_db DELETE_DB] [-mfp MFP] [-keep KEEP] [-delete_from DELETE_FROM] [-conv CONV]
Begin processing HartRAO drift scan data
optional arguments: -h, --help show this help message and exit -db DB turn debugging on or off. e.g. -db on, by default debug is off -f F process file or folder at given path e.g. -f data/Hydr aA_13NB/2019d133_16h12m15s_Cont_mike_HYDRA_A.fits or -f data/HydraA_13NB or -f data/ -force FORCE force fit all drift scans y/n e.g. -force y. Default is set to n -c C initiate the command to run program. e.g. -c gui or -c run_auto_analysis or -c cmdl -b B initiate browser. e.g. -b docs or -b dash -delete_db DELETE_DB delete database on program run. e.g. -delete_db y -mfp MFP multi-file processing of data between two dates. e.g. -mfp fileList.txt -keep KEEP keep original plots while processing data. e.g. -keep y -delete_from DELETE_FROM in coordination eith the filename, use delete_from to delete a row from a database e.g. delete_from CALDB -conv CONV convert the database tables to csv. e.g. conv CALDB.db -saveLocs SAVELOCS Save fit locations. e.g. -saveLocs y , default is no -fitTheoretical FITTHEORETICAL Fit theoretical locations of the baseline, i.e. the FNBW locs. e.g. -fitTheoretical y , default is no. -applyRFIremoval APPLYRFIREMOVAL Apply or don't apply RFI removal e.g. -applyRFIremoval n, default is yes. -fitLocs FITLOCS Fit data at sepcified locations e.g. -fitLocs pathToFile -analyze ANALYZE ANALYZE ANALYZE Analyze results from database e.g. -analyze CALDB.db Jupiter_22ghz all/no -quickview QUICKVIEW get quickview of data e.g. -quickview y -db_rows DB_ROWS use all rows in the database, including special case data, for analysis., e.g. -db_rows all --version show program's version number and exit
Depending on the process you want to run, you can select one or more of the available options. If there are more options you would like implemented please email the author, details are listed below.
Data processing examples
to perform an automated data reduction process on a single file, use the following command
.. code:: bash
$ dran-auto -f path-to-file/filename.fits
if you want to set debuggin on
.. code:: bash
$ dran-auto -f path-to-file/filename.fits -db on
to delete both a database
.. code:: bash
$ dran-auto -f path-to-file/filename.fits -delete_db y
to process multiple chunks of data within seperate periods of the year you use the following command
.. code:: bash
$ dran -mfp file_list.txt
NB Please note that this assumes that there exists a file called fileList.txt in the current directory that has the full path to the folder containing the source fits files you want to process. This file also has a start and end date in "YYYYdDDD" format stipulating the data range you want to process.
to process all the data located in your directory, NB this is the directory that contains all the folders containing your fits files. For example
Jupiter_22GHz 2020d139 2020d139
processes files located in the Jupiter_22GHz directory starting with the 2020d139 to 2020d139 dates.
To view the web documentation guide of the software
.. code:: bash
$ dran-docs
To delete an entry made in a database
.. code:: bash
$ dran-docs -delete_from CALDB
To keep the plots of the raw data
.. code:: bash
$ dran-auto -f path-to-file/filename.fits -keep y
To convert the database tables to csv
.. code:: bash
$ dran-auto -conv y
Starting the program will bring up the menu shown below with options to select how you want to proceeed with your data reduction or analysis.
.. code-block:: bash
##################################################################
###### ###### ###### # #
# # # # # # # # #
# # ##### ###### # # #
# # # # # # # ##
###### # # # # # #
##################################################################
PROGRAM STARTED
SELECT OPTION:
- Data reduction
- Open GUI
- Run command line analysis
- Open documentation
- Open dashboard
- Exit program
Please select an option
Option Selected:
To get a quick view of the source file
.. code:: bash
$ dran-auto -quickview y
Caveats
Although the program tries to fit the data appropriately there are a few things the user needs to be aware of. The program tries its best to fit all the data (good/bad). However the onus is still on the user to check/insure the scans have been fit in a satisfactory manner by checking the results in the plots folder.
Author contacts
If you have any problems, questions, ideas or suggestions, please
OPEN AN ISSUE <http://www.hartrao.ac.za/hh26m_factsfile.html>
_
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for dran-0.0.10.dev0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18dde7c27958b87f0e5a9369e42a58b11a800ad4f4b21430fdfdb1b959c8bdb7 |
|
MD5 | dcc634f823b5a705de7b101e39e12a26 |
|
BLAKE2b-256 | d77e6c684c1fe6113e115e630093b43148414cb2ec698e320b2679607ba37f82 |