krillscan 0.2.8

A python module for automatic analysis of backscatter data from vessels of opportunity

Krillscan - A python module for automatic analysis of backscatter data from fishing vessels to implement feedback management of the Antarctic krill fishery

• Automatically processes and analyzees EK60 and EK80 echosounder data
• stores echograms and NASC tables as NetCDF, HDF or .csv files
• detects krill swarms and can send near-real-time acoustic registrations via email

Installation

Krillscan was written in python 3.9 and can be run from the source code, from inside a docker container or as compiled executable. We added the compiled krillscan.exe to the latest distribution in this repository. To download the krillscan module with python try:

pip install krillscan

You will also need to install this library manually and store it into your curent working folder

https://github.com/CI-CMG/pyEcholab

Operation

Automatic processing

Here I describe how to operate krillscan from inside python, instructions for the docker container are found below. After opening a python prompt and installing krillscan, you need to create or download a settings.ini file that tells krillscan where to look for incoming files and where to store processed data and contains processing and data transfer parameters. The file should be a text files following this pattern:

[GENERAL]
source_folder = E:\Endurance\2022\raw
target_folder = E:\Endurance\2022\krillscan
scrutinization_frequency = 200000.0
vessel_name = Antarctic Endurance

[CALIBRATION]
gain = None
sa_correction = None
beam_width_alongship = None
beam_width_athwartship = None
angle_offset_alongship = None
angle_offset_athwartship = None

[PROCESSING]
sv_SNR_threshold_db = 3
swarm_sv_threshold_db = -70
dbdiff_sv_threshold_db = 3
seafloor_offset_m= 10
seafloor_sv_threshold_db= -38
surface_exclusion_depth_m = 20
maximum_depth_m = 250

[EMAIL]
email_send = 0
email_from = *Enter a mail address here*
email_to = *Enter a mail address here*
pw = *Enter IMAP password address here*
files_per_email = 6
send_echograms = 1
echogram_resolution_in_seconds = 1

This settings.ini file must be located in the same working directory as your python prompt. To send emails set email_send=1 (or True).

To start the processing type:

import krillscan as ks
ks.krillscan.ks.start()
# to stop the processing: 
# ks.krillscan.ks.stop()

This starts two threads, one that looks for new data, processes and stores it and another that scans for recently processed data and sends them out via email. This is an ad-hoc solution so far, future versions of krillscan will transfer data directly to a cloud storage.

Output format

Krillscan will store the raw and processed echograms, detection masks and NASC tables in 10-min long snippets, with the filename containing the start date of each snippet:

D20140125-T060417_echogram.nc
D20140125-T060417_mask_dbdiff.h5
D20140125-T060417_mask_swarm.h5
D20140125-T060417_nasctable.csv
D20140125-T060417_nasctable.h5
D20140125-T060417_rawechogram.nc

Here is an example of the contents of the echogram files, they are stored and read using the xarray module:

import xarray as xr
file="D20140125-T055416_rawechogram.nc"
xr_sv = xr.open_dataarray(file)
print(xr_sv)

<xarray.DataArray (frequency: 2, depth: 1000, time: 427)>
[854000 values with dtype=float32]
Coordinates:
  * time       (time) datetime64[ns] 2014-01-25T05:54:16.770000 ... 2014-01-2...
  * depth      (depth) float64 0.0 0.5 1.0 1.5 2.0 ... 498.0 498.5 499.0 499.5
  * frequency  (frequency) float64 3.8e+04 1.2e+05

Here is an example of the contents of the mask files, they are stored and read as pandas DataFrames:

import pandas as pd
file="D20140125-T060417_mask_swarm.h5"
df = pd.read_hdf(file,key='df')
print(df)

2014-01-25 06:04:17.003  False  False  False  False  ...  False  False  False  False
2014-01-25 06:04:18.405  False  False  False  False  ...  False  False  False  False
2014-01-25 06:04:19.807  False  False  False  False  ...  False  False  False  False
2014-01-25 06:04:21.209  False  False  False  False  ...  False  False  False  False
2014-01-25 06:04:22.611  False  False  False  False  ...  False  False  False  False
                       ...    ...    ...    ...  ...    ...    ...    ...    ...
2014-01-25 06:14:09.135  False  False  False  False  ...  False  False  False  False
2014-01-25 06:14:10.537  False  False  False  False  ...  False  False  False  False
2014-01-25 06:14:11.949  False  False  False  False  ...  False  False  False  False
2014-01-25 06:14:13.351  False  False  False  False  ...  False  False  False  False
2014-01-25 06:14:14.753  False  False  False  False  ...  False  False  False  False

[427 rows x 1000 columns]

Here is an example of the contents of the NASC tables, they are stored and read as pandas dataframes:

import pandas as pd
file="D20140125-T055416_nasctable.h5"
df = pd.read_hdf(file,key='df')
print(df.columns)
print(df)

Index(['lat', 'lon', 'distance_m', 'bottomdepth_m', 'nasc_swarm',
       'nasc_dbdiff', 'nasc_manual'],
      dtype='object')
                               lat        lon  ...  nasc_dbdiff  nasc_manual
ping_time                                      ...                          
2014-01-25 05:54:16.770        NaN        NaN  ...    68.604281          0.0
2014-01-25 05:54:18.172        NaN        NaN  ...   188.157065          0.0
2014-01-25 05:54:19.574 -60.010743 -48.319883  ...   223.696955          0.0
2014-01-25 05:54:20.976 -60.010727 -48.320011  ...    89.158265          0.0
2014-01-25 05:54:22.378 -60.010712 -48.320137  ...     0.000000          0.0
                           ...        ...  ...          ...          ...
2014-01-25 06:04:08.591 -60.004068 -48.374217  ...     0.000000          0.0
2014-01-25 06:04:09.993 -60.004052 -48.374346  ...   704.198641          0.0
2014-01-25 06:04:11.395 -60.004037 -48.374469  ...   334.853738          0.0
2014-01-25 06:04:12.797 -60.004021 -48.374597  ...  1719.993121          0.0
2014-01-25 06:04:14.199 -60.004004 -48.374726  ...     0.000000          0.0

[427 rows x 7 columns]

Here is an example for 1-month of echosunder data from a krill fishing vessel, processed entirely automatic using the swarm detection method.

Visual inspection of processed echograms

Krillscan also contains a GUI tool (PyQT5) that lets you inspect the automatically processed echograms and correct the swarm detection masks by adding or removing areas. To start the GUI enter:

ks.ks.inspect()

The GUI looks like this: A menu bar lets you open processed NetCDF files, show the data folder, undo changes and quite the GUI. The Main window shows the echogram for each 10-min long NetCDF and the ping-by-ping NASC (black line) and 1-min NASC average (blue line).

The tool bar below the menu bar has these functions:

• Choose which frequency channel to watch

• Tick box that lets you check whether to just browse that data or save changes. Whenever you close a 10-min snippet by clicking "next" or "previous" section a new detection mask is stored as HDF file with the ending "...mask_manual.h5". And the column "nasc_manual" is added to the nasc table file "...nasctable.h5".

• You can toggle weather to watch the raw or processed echogram

• You can set the lower and upper NASC integration boundaries, and color scheme

• You can go forward or backward in the dataset by pressing next (or press right arrow) or previous (or press left arrow), this opens the next 10-min echogram NetCDF and detection mask

• You can add or remove areas of the detection mask (shown as red overlay) using double clicks to draw a polygon of you choice. Once finished with drawing press enter to apply the change and update the NASC accordingly.

Run krillscan automatic processing from a Docker container