fesh2 2.3.2

Geodetic VLBI schedule management and processing

Fesh2: Geodetic VLBI schedule file management and processing

What is it?

Fesh2 is a Python package that provides automated schedule file preparation. It requires an installation of the NASA Field System.

Fesh2 runs in a terminal window and regularly checks IVS schedule repositories for new or updated versions of Master files and schedule files for one or more specified stations. A check for the latest version of the Master file(s) is done first, but skipped if the time since the last check is less than a specified amount. Similarly, checks on schedule files are only done if the time since the last check exceeds a specified time. If new or updated schedules are found, they are optionally processed with Drudg to produce snp, prc and lst files. By default, once the files have been checked, fesh2 will provide a summary and then go into a wait state before carrying out another check. Fesh2 can also be run once for a single check or status report and not go into a wait state. Fesh2 can also be run in a monit mode that gives a continually updating status report but does not process any files. Multiple instances can be run simultaneously. If drudg output (snp or prc files) have been modified by the user and a new schedule becomes available , fesh2 will download the file but not overwrite Drudg output, but it will warn the user.

Fesh2 can be run as a foreground application or as a service in the background.

Compatibility

Fesh2 code is compatible with Python versions 3.8 and above. Note that fesh2 is not compatible with Python version 2.

Note that if you're using an old version of Debian (Jessie or earlier) then you will need to install python 3.8 or higher by hand. This could be done inside a virtual environment by a user or on the system as root.

Installation

Fesh2 is a stand-alone add-on to the Field System so installation must be carried out separately.

PycURL dependency

Fesh2 depends on the python CURL library (PycURL) which should be installed automatically when you install fesh2. However, PycURL depends on the Python development libraries, and it won't install if they're not there. On a Debian machine,

apt-get install python3-dev

should do the trick. You may also need these:

apt-get install libcurl4-openssl-dev libssl-dev

Please make sure the above dependencies are satisfied before proceeding.

Installation should be carried out under the prog account. You may wish to install fesh2 within a python virtual environment, but regardless, the installation commands are the same:

Installing with pip:

Run the following:

pip install fesh2   

You will then need to edit the fesh2 configuration file for your station(s). More information on configuration is provided below.

A template configuration file fesh2.config is provided in the fesh2 GitHub repository and a copy of the contents is in the Appendix at the end of this document. fesh2.config should be placed in /usr2/control and edited to suit your site.

Configuration

Fesh2 will take its configuration in the following priority order from lowest to highest:

• Two config files, in this order:
  • /usr2/control/skedf.ctl
  • /usr2/control/fesh2.config
• Environment variables
• Command-line parameters

Note that Fesh2 will only run Drudg if it DoDrudg is set to True and it is configured appropriately. See the Drudg notes below.

If you have the configuration files in a different location to /usr2/control, use the command-line options --ConfigFile and --SkedConfigFile to set them appropriately.

Environment Variables

The following environment variables are recognised by fesh2:

• NETRC_FILE and COOKIES_FILE: the locations of the .netrc and .urs_cookies files, used by CURL for the https protocol. Curl sets these to files in the home directory by default. More information is given below on Curl configuration.
• LIST_DIR : the directory where drudg puts .lst output files. If this is not set, then the LstDir parameter in fesh2.config is used if set. Otherwise drudg defaults to the $schedules parameter in skedf.ctl

A number of environment variables used by drudg are also recognised by fesh2. See the drudg section below. The following environment variables are NOT recognised by fesh2:

• STATION is used in the Field System but not recognised by fesh2 as it is possible to configure fesh2 to manage schedules for multiple stations . The Stations parameter must be set in fesh2.config

The fesh2 configuration file

On startup, fesh2 looks for a configuration file called fesh2.config. This will need to be set up for your station before running fesh2 for the first time. Use your favourite text editor to modify the file. The comments in the file describe the parameters but here are a few points to note:

Station settings

You will want to edit options in the [Station] section to configure which stations you want to process, what types of Master and schedule files you want to obtain, how far into the future you want to search for schedule files and how often you want to look for them.

Email

If enabled, notification can be sent via email when:

• a new schedule has been downloaded
• a schedule file has been updated on the server, downloaded and needs manual processing

Drudg

After finding and downloading a new or updated schedule, Fesh2 can optionally run Drudg to produce new snp, prc and lst files. The [Drudg] section allows you to configure this behaviour. If you don't want fesh2 to automatically process your schedules, this feature con be turned off by setting DodDrudg to False.

Because fesh2 manages schedule files and uses drudg to process them, and to keep things consistent across the Field System, it will use settings in skedf.ctl. However, there are some caveats regarding skedf.ctl for the case where there is not an overriding definition:

• Fesh2 requires that $schedules is defined in skedf.ctl. The default of '.' (i.e. the directory the software is started in) is too arbitrary and could cause fesh2 to lose track of files.
• If $snap or $proc are not defined, then they will be set to the same as $schedules

Drudg may have been configured such that the user is prompted for a response, in which case it cannot be automated. If this is the case, then Drudg will not be run on SKD files.

If you want Fesh2 to automatically process SKD files, then some configuration is required. Fesh2 will get it's configuration information as follows, in this priority order (lowest to highest):

1. Fesh2 will first look in /usr2/control/skedf.ctl. Unless parameters are changed, Fesh2 will not run Drudg if the following parameters are set as shown:

   • tpicd YES
   • vsi_align ASK
   • cont_cal ASK
   • cont_cal_polarity ASK

2. Fesh2 will then look for any configuration parameters in the [Drudg] secoion of fesh2.config. These parameters can be different to skedf.ctl in case you want Drudg to behave differently when Fesh2 is running it for you

3. There are some FESH environment variables which can be set and these will override skedf.ctl or this file if they are. These environment variables are as follows with the lowest supported version of Drudge shown in brackets (text from FS documentation):

   • FESH_GEO_TPICD (10.0.0-beta3, December 2020) Possible values are non-negative integers. If set, the value is provided as the answer for the drudg prompt for the tpicd interval for geodesy schedules.
   • FESH_GEO_CONT_CAL (10.0.0-beta3, December 2020) Possible values are on or off. If set, the value is provided as the answer for the drudg prompt for continuous cal use for geodesy schedules.
   • FESH_GEO_CONT_CAL_POLARITY (10.0.0-beta3, December 2020) Possible values are 0, 1, 2, 3, or none. If set, the value is provided as the answer for the drudg prompt for the continuous cal polarity for geodesy schedules.
   • FESH_GEO_VSI_ALIGN (10.0.0-beta3, December 2020) Possible values are 0, 1, or none. If set, the value is provided as the answer for the drudg prompt for using vsi_align for geodesy schedules.
   • FESH_GEO_USE_SETUP_PROC (10.1.0, 2021) Possible values are yes or no. If set, the value is provided as the answer for the drudg prompt for the "use setup_proc" for geodesy schedules, an option to write a .snp file that skips setup on scans when the mode hasn’t changed.
   • FESH_GEO_VDIF_SINGLE_THREAD_PER_FILE (10.1.0, 2021) Possible values are yes or no. If set, the value is provided as the answer for the drudg prompt for the "VDIF single thread per file" for geodesy schedules.

4. Lastly, if any Drudg-related command-line parameters are set, they will override everything else

If any Drudg settings recieved by Fesh2 require manual intervention, Drudg will not be run on the SKD files.

Servers

The [Servers] section allows you to list all the IVS schedule file servers you want to search and the protocols to use for each. Specify the location of the top directory (i.e. the vlbi directory). Protocols known to work are https ( secure HTTP), ftp (anonymous FTP) and ftps (secure (SSL) anonymous FTP).

Curl

Fesh2 uses curl to access files on the servers. If https is being used (e.g. to access the CDDIS repository), then fesh2 needs to know the location of your .netrc and ( optionally) .urs_cookies files. Curl puts these in the user's home directory by default but they can be placed elsewhere if desired. This can be set on the command line or via the NETRC_FILE and COOKIES_FILE environment variables ( see above ) or in the fesh2 config file by setting the NetrcFile and CookiesFile parameters.

Usage

Fesh2 can be started by just typing

fesh2

which starts it in its default mode. fesh2 can also be run in monitoring mode in a terminal window by typing:

fesh2 --monit

Note: If you run fesh2 as multiple instances but with different configuration (e.g. running it in the background with SnapDir set on the command line, then running it as a monitor with the default SNAP file directory) then you may see some unexpected behaviour. In the above example, a schedule could be generated by drudg in the non-default SNAP directory but fesh2 --monit would think that the schedule hadn't been processed.

Command-line options exist to change many of the configuration parameters, allow for a one-off checks (no wait mode), forcing of file downloads etc. Some common usages are:

• Only consider the current or next experiment

fesh2 -n

• Just run once then exit, don't go into a wait loop

fesh2 --once

• Just get a schedule for a specified session, then exit. e.g.:

fesh2 --once -g r1456

• Force an update to the schedule file when there's a new one available to replace the old one. The default behaviour is to give the new file the name <code>.skd.new and prompt the user to take action. The file will also be drudged if the DoDrudg option is True

fesh2 --update

• Obtain schedule files, but don't process them:

fesh2 --DoDrudg False

• Get a status report. Shows the status of schedule files and when schedule servers were last queried.

fesh2 --check

• Run fesh2 with all terminal output suppressed. Useful when running fesh2 as a service.

fesh2 --quiet

All command-line parameters are as follows:

usage:       fesh2 [-h] [-c CONFIGFILE] [-k SKEDCONFIGFILE] [-g GET] [-m] [-u]
                   [-n] [-a] [-o] [--update] [--SchedDir SCHEDDIR]
                   [--ProcDir PROCDIR] [--SnapDir SNAPDIR] [--LstDir LSTDIR]
                   [--LogDir LOGDIR] [--Stations [STATIONS [STATIONS ...]]]
                   [--EmailNotifications [EMAILNOTIFICATIONS]]
                   [--GetMaster [GETMASTER]]
                   [--GetMasterIntensive [GETMASTERINTENSIVE]]
                   [--SchedTypes [SCHEDTYPES [SCHEDTYPES ...]]]
                   [-t MASTERCHECKTIME] [-s SCHEDULECHECKTIME]
                   [-l LOOKAHEADTIMEDAYS] [-d [DODRUDG]]
                   [--DrudgBinary DRUDGBINARY] [--TpiPeriod TPIPERIOD]
                   [--VsiAlign VSIALIGN] [--ContCalAction CONTCALACTION]
                   [--ContCalPolarity CONTCALPOLARITY] [--SetupProc SETUPPROC]
                   [--VdifSingleThreadPerFile VDIFSINGLETHREADPERFILE]
                   [--Servers [SERVERS [SERVERS ...]]] [--NetrcFile NETRCFILE]
                   [--CookiesFile COOKIESFILE]
                   [--CurlSecLevel1 [CURLSECLEVEL1]] [-y YEAR] [-e] [--monit]
                   [-q]

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIGFILE, --ConfigFile CONFIGFILE
                        The fesh2 configuration file to use. e.g.
                        /usr2/control/fesh2.config
  -k SKEDCONFIGFILE, --SkedConfigFile SKEDCONFIGFILE
                        The location of the skedf.cfg configuration file
  -g GET, --get GET     Just get a schedule for this specified session. Give
                        the name of the session (e.g. r4951).
  -m, --master-update   Force a download of the Master Schedule (default =
                        False), but just on the first check cycle.
  -u, --sched-update    Force a download of the Schedules (default = False),
                        but just on the first check cycle.
  -n, --current, --now  Only process the current or next experiment
  -a, --all             Find the experiments with all "Stations" in it
  -o, --once            Just run once then exit, don't go into a wait loop
                        (default = False)
  --update              Force an update to the schedule file when there's a
                        new one available to replace the old one. The default
                        behaviour is to give the new file the name
                        <code>.skd.new and prompt the user to take action. The
                        file will also be drudged if the DoDrudg option is
                        True
  --SchedDir SCHEDDIR   Schedule file directory
  --ProcDir PROCDIR     Procedure (PRC) file directory
  --SnapDir SNAPDIR     SNAP file directory
  --LstDir LSTDIR       LST file directory [env var: LIST_DIR]
  --LogDir LOGDIR       Log file directory
  --Stations [STATIONS [STATIONS ...]]
                        Stations to consider (e.g. "hb ke yg ho", "mg")
  --EmailNotifications [EMAILNOTIFICATIONS]
                        Send notifications by email. The fesh2 config file
                        will be read for details on mail server, recipients
                        etc
  --GetMaster [GETMASTER]
                        Maintain a local copy of the main Multi-Agency
                        schedule, i.e. mostly 24h sessions (default = True)
  --GetMasterIntensive [GETMASTERINTENSIVE]
                        Maintain a local copy of the main Multi-Agency
                        Intensive schedule (default = True)
  --SchedTypes [SCHEDTYPES [SCHEDTYPES ...]]
                        Schedule file formats to be obtained? This is a
                        prioritised list with the highest priority first. Use
                        the file name suffix ("vex" and/or "skd") and comma-
                        separated.
  -t MASTERCHECKTIME, --MasterCheckTime MASTERCHECKTIME
                        Only check for a new master file if the last check was
                        more than this number of hours ago. The default is set
                        in the configuration file.
  -s SCHEDULECHECKTIME, --ScheduleCheckTime SCHEDULECHECKTIME
                        Only check for a new schedule file (SKD or VEX) if the
                        last check was more than this number of hours ago. The
                        default is set in the configuration file.
  -l LOOKAHEADTIMEDAYS, --LookAheadTimeDays LOOKAHEADTIMEDAYS
                        Only look for schedules less than this number of days
                        away (default is 7)
  -d [DODRUDG], --DoDrudg [DODRUDG]
                        Run Drudg on the downloaded/updated schedules (default
                        = True)
  --DrudgBinary DRUDGBINARY
                        Location of Drudg executable (default =
                        /usr2/fs/bin/drudg)
  --TpiPeriod TPIPERIOD
                        Drudg config: TPI period in centiseconds. 0 = don't
                        use the TPI daemon (default) [env var: FESH_GEO_TPICD]
  --VsiAlign VSIALIGN   Drudg config: Applicable only for PFB DBBCs, none =
                        never use dbbc=vsi_align=... (default) 0 = always use
                        dbbc=vsi_align=0 1 = always use dbbc=vsi_align=1 [env
                        var: FESH_GEO_VSI_ALIGN]
  --ContCalAction CONTCALACTION
                        Drudg config: Continuous cal option. Either 'on' or
                        'off'. Default is 'off' [env var: FESH_GEO_CONT_CAL]
  --ContCalPolarity CONTCALPOLARITY
                        Drudg config: If continuous cal is in use, what is the
                        polarity? Options are 0-3 or 'none'. [env var:
                        FESH_GEO_CONT_CAL_POLARITY]
  --SetupProc SETUPPROC
                        Drudg config: the answer for the drudg prompt for the
                        use setup_proc for geodesy schedules, an option to
                        write a `.snp` file that skips setup on scans when the
                        mode hasn’t changed. [env var:
                        FESH_GEO_USE_SETUP_PROC]
  --VdifSingleThreadPerFile VDIFSINGLETHREADPERFILE
                        VDIF single thread per file: only applies to Mark5C or
                        Flexbuff recorders. Can be 'yes' or 'no' [env var:
                        FESH_GEO_VDIF_SINGLE_THREAD_PER_FILE]
  --Servers [SERVERS [SERVERS ...]]
                        Schedule file server URLs. Each of these will be
                        checked for the most recent files. Use comma-separated
                        URLs and specify the top directory (i.e. the 'vlbi'
                        directory). Use protocols https (Curl), ftp (anonymous
                        FTP) or sftp (anonymous secure FTP)
  --NetrcFile NETRCFILE
                        The location of the .netrc file, needed by CURL for
                        the https protocol. (CURL puts this in ~/.netrc by
                        default) [env var: NETRC_FILE]
  --CookiesFile COOKIESFILE
                        The location of the .urs_cookies files used by CURL.
                        (CURL puts this in ~/.urs_cookies by default) [env
                        var: COOKIES_FILE]
  --CurlSecLevel1 [CURLSECLEVEL1]
                        Workaround for CDDIS https access in some Debian
                        distributions. See the documentation (default = False)
  -y YEAR, --year YEAR  The year of the Master Schedule (default is this year)
  -e, --check           Check the current fesh2 status. Shows the status of
                        schedule files and when schedule servers were last
                        queried.
  --monit               Similar to --check but text output format is intended
                        for a FS monit interface
  -q, --quiet           Runs fesh2 with all terminal output suppressed. Useful
                        when running fesh2 as a service.

Logging

As well as writing information to the screen on activity, fesh2 also keeps a log of activity in the Field System log directory (by default) at /usr2/log/fesh2.log. The log file location can be configured in fesh2.config

Running fesh2 as a service

Fesh2 can be run in the background as a systemd service. All output is suppressed and status is available by examining the log file or using the --check or -e flag. Here's how to set it up from the superuser account for Debian Jessie or later:

1. Type the following command to add a systemd service:

   systemctl edit --force --full fesh2.service
   

   This should open a text editor. Paste in the following:

   [Unit]
   Description=Fesh2 Service
   After=network-online.target
   Wants=network-online.target
   
   [Service]
   ExecStart=/usr/bin/sudo -H -u oper /usr/local/bin/fesh2 --quiet
   
   [Install]
   WantedBy=multi-user.target
   

   Save and exit. This will configure systemd to start fesh2, running as user oper and suppress all output.
2. Enable the service:

   sudo systemctl enable fesh2.service
   

3. Check the status of the service:

   sudo systemctl status fesh2.service
   

4. You can stop, start and query the service:

   sudo systemctl stop fesh2.service          # Stop running the service 
   sudo systemctl start fesh2.service         # Start running the service 
   sudo systemctl restart fesh2.service       # Restart the service 
   

5. To see the current schedule file status (as user oper):

   fesh2 --check
   

If you would prefer to set this up as a user service, some notes are here:

• https://www.unixsysadmin.com/systemd-user-services/

For versions of Debian older thean Jessie (e.g. Wheezy), systemd can be enabled and some notes are here:

• https://scottlinux.com/2014/10/20/how-to-switch-to-systemd-on-debian-wheezy/

Appendix A: fesh2.config

# Fesh2 configuration file.
#
# Note: all non-essential parameters (i.e. those with defaults) are commented out
#
# Field System directories
[FS]
  # Log directory. This is where the fesh2 log file will be written.
  # Default is /usr2/log
  #LogDir = /usr2/log

# Default settings for this station/facility
[Station]
  # station name(s) to process by default. Just a two-letter code for a single
  # station or comma separated two-letter codes inside square brackets for
  # multiple stations. e.g.:
  # Stations = [Hb, Ho, Ke, Yg]
  # Stations = Ke
  Stations =  Mg
  #
  ##################
  # Which master files should be processed? At least one of these must be selected
  # Get the main Multi-Agency schedule, i.e. mostly 24h sessions. Default is True
  #GetMaster = True
  # Get the Multi-Agency Intensives schedule. Default is True
  #GetMasterIntensive = True
  ##################
  # What schedule file formats should be obtained? This is a prioritised list with the highest one first
  # Use the file name suffix (vex and/or skd) and comma-separated. Default is skd
  #SchedTypes = skd
  #SchedTypes = [skd, vex]
  # What is the minimum time between checks for a new Master file on the server (hours)?
  MasterCheckTime = 12
  # What is the minimum time between checks for a new Schedule (SKD or VEX) file on the server (hours)?
  ScheduleCheckTime = 1
  # How far ahead in time should we look for schedule files (days)?
  LookAheadTimeDays = 14

[Email]
  # If Fesh2 finds a problem with a schedule or if a schedule needs manual processing,
  # a notification message can be sent via email. To turn this feature on, set email_notifications to True,
  # specify a recipient(s)  and configure with the address of the SMTP server.
  EmailNotifications = True
  # an array containing one or more email addresses to receive notifications
  EmailRecipients = [[e-mail]]
  EmailSender = [e-mail]
  # The SMTP server
  EmailServer = myserver.thing.com.au
  # SMTP Port
  #EmailPort = 587
  # Server Password. Leave commented out if you don't need one
  #EmailPassword = "VerySecret"

[Drudg]
  # Drudg-related config
  # Run drudg on downloaded schedules? If False then files are downloaded but not processed
  DoDrudg = True
  # Location of the drudg executable
  DrudgBinary = /usr2/fs/bin/drudg
  # Directory for the output SNP file
  # Default is /usr2/sched
  SnapDir = /usr2/sched
  # Directory for the output LST (schedule summary) file
  # Default is /usr2/sched
  LstDir = /usr2/sched
  #-------------------------------------------------------------------------------------------------
  # If DoDrudg is True, then Fesh2 will run Drudg on the SKD files it downloads. It will drudg all new
  # SKD files and all updated SKD files provided they have not been changed locally in the meantime.
  #
  # However, Drudg may have been configured such that the user is prompted for a response, in which
  # case it cannot be automated. If this is the case, then Drudg will not be run on SKD files.
  #
  # If you want Fesh2 to automatically process SKD files, then some configuration of Drudg is required.
  # Fesh2 will get it's configuration information as follows, in this priority order (lowest to highest):
  # config file, env var, command line
  # 1. Fesh2 will first look in /usr2/control/skedf.ctl
  # 2. It will then look for any configuration parameters in this file (just below these comments)
  #    These parameters can be different to skedf.ctl in case you want Drudg to behave differently
  #    when Fesh2 is running Drudg for you
  # 3. There are some FESH environment variables which can be set and these will override skedf.ctl
  #    or this file if they are.
  # 4. Lastly, any Drudg-related command-line parameters are set, they will override everything else
  #
  # If any Drudg settings recieved by Fesh2 require manual intervention, Drudg will not be run on
  # the SKD files.
  #
  # If set, these parameters will override the corresponding settings in skedf.ctl but will be overridden
  # by environment variables and command-line parameters. Uncomment and set them if you wish, otherwise they are
  # ignored:
  #
  #  1. TPI period in centiseconds
  #     0 = don't use the TPI daemon
  #TpiPeriod = 0
  #
  #  2. Continuous cal option. Either "on" or "off". Default is 'off'
  #ContCalAction = off
  #
  #  3. If continuous cal is in use, what is the polarity? Options are 0-3 or "none". Default is none
  #ContCalPolarity = none
  #
  #  4. For PFB DBBCs only, vsi_align setup
  #     Applicable only for PFB DBBCs,
  #     none = never use dbbc=vsi_align=... (default)
  #     0 = always use dbbc=vsi_align=0
  #     1 = always use dbbc=vsi_align=1
  #VsiAlign = none
  #
  #  5. If set, the value is provided as the answer for the drudg prompt for the "use setup_proc" for geodesy
  #     schedules, an option to write a `.snp` file that skips setup on scans when the mode hasn’t changed.
  #     Possible values are `yes` or `no`.
  #SetupProc = yes
  #
  #  6. If set, the value is provided as the answer for the drudg prompt for the *"VDIF single thread per file"*
  #     for geodesy schedules.
  #     Possible values are `yes` or `no`.
  #VdifSingleThreadPerFile = yes
  #-------------------------------------------------------------------------------------------------

[Servers]
  # Schedule file server URLs. Each of these will be checked for the most recent files.
  # Use comma-separated URLs inside square brackets, and specify the top directory (i.e. the 'vlbi' directory).
  # Use protocols https (Curl), ftp (anonymous FTP) or sftp (anonymous secure FTP)
  Servers = [https://cddis.nasa.gov/archive/vlbi, ftp://ivs.bkg.bund.de/pub/vlbi, ftp://ivsopar.obspm.fr/pub/vlbi]
  # Alternative CDDIS (anonymous secure FTP):
  # Servers = ftps://cddis.nasa.gov/archive/vlbi

[Curl]
  # Curl configuration files.
  # A .netrc and a .urs_cookies file are needed by Curl for the https protocol. Curl puts these in the user's
  # home directory by default but they can be placed elsewhere if desired. This can be set
  # on the command line or via the NETRC_FILE and COOKIES_FILE environment variables. The command-line overrides
  # the environment variable which overrides the config file
  NetrcFile = /usr2/control/netrc_fesh2
  # The cookies file is optional and the default is /dev/null
  CookiesFile = /dev/null
  # In some recent Debian distributions, a connection to CDDIS will fail with error 35 (see
  # item 17 in the "CDDIS https Archive Access/File Download FAQ" https://cddis.nasa.gov/About/CDDIS_File_Download_FAQ
  # .html). You will see a WARNING message if this occurs and downloads from CDDIS will not be possible.
  # There is a workaround until the CDDIS server is changed to fix the problem: Set the following
  # parameter to True
  CurlSecLevel1 = False