hifis-surveyval 1.2.0rc0

This project is used to develop analysis scripts for the HIFIS Software survey.

HIFIS-Surveyval

This project is used to develop analysis scripts for the HIFIS Software survey.

Table of Content

• Installation
• Getting Started
• Start Analysis from Command-Line-Interface
• Contribute with Own Analysis Scripts
• Resources
• Author Information
• Contributors
• License

Getting Started

The project's documentation contains a section to help you as a user of the analysis scripts to run the analysis scripts or as a developer of the framework to set up the development environment.

Getting Started for Users

Installation

To install the package locally, you can use Pip.

pip install hifis-surveyval

After the installation, you can use the tool from the command line with hifis-surveyval --help.

Getting Started for Developers

Installation

To install the package locally, you can use Poetry.

Using Poetry

If you want to actively contribute changes to the project, you are required to also install the development packages alongside the framework.

git clone https://gitlab.hzdr.de/hifis/overall/surveys/hifis-surveyval.git
cd hifis-surveyval
poetry install

After the installation, you can use the tool from the command line with poetry run hifis-surveyval --help

Poetry installs some packages that are required for performing quality checks. Usually they are also performed via GitLab CI, but can also be executed locally.

It is common practice to run some checks locally before pushing them online. Therefore, execute below commands:

$ # Order your imports
$ isort -rc .
$ make lint

The following documentation references the pip installation. You can use the same commands with a poetry installation, if you prefix your commands with poetry run COMMAND.

Start Analysis from Command-Line-Interface

The survey analysis package is a program to be executed on the Command-Line-Interface (CLI).

Quick Start Example: Run Analysis

Due to sensible defaults of the project's configurations you need to have the analysis scripts, the preprocessing script as well as metadata and data files in certain locations in order to run the survey analysis. This configuration file hifis-surveyval.yml which includes these defaults is created with the command hifis-surveyval init. Please put your analysis scripts into a sub-folder called scripts. The preprocessing script preprocess.py is expected in the root folder of the project. Make sure that the file meta.yml is put into sub-folder metadata. Finally, copy the CSV data file of your survey to a central location like a data sub-folder and tell the program the path to that data file on the command line when running the survey analysis.

Now you can do the following to start the survey analysis from the CLI:

hifis-surveyval analyze data/<data_file_name>.csv

The output is then put into a sub-folder within the folder output which is named after the stamp of the current date-time if not specified differently.

Caution: Depending on the Operating System used an issue with the file encoding might occur. There might be data-CSV-files around which are encoded with UTF-8-BOM which causes errors when read in on Windows OS. In this case you need to change the encoding to UTF-8 before running the survey analysis.

Flags

The program accepts two flags:

1. Help flag
2. Verbosity flag

Help flag

Calling the program with the help-flag is the first thing to do when being encountered with this program. It outputs a so-called Usage-message to the CLI:

$ hifis-surveyval --help

Please issue this command on the CLI and read the detailed Usage-message before continuing with reading the documentation of the Usage-message here.

Verbosity flag

The verbosity-flag can be provided in order to specify the verbosity of the output to the CLI. This flag is called --verbose or -v for short:

hifis-surveyval --verbose <COMMAND>

hifis-surveyval -v <COMMAND>

The verbosity of the output can be increased even more by duplicating the flag --verbose or -v up to two times:

hifis-surveyval --verbose --verbose --verbose <COMMAND>

hifis-surveyval -vvv <COMMAND>

Commands

There are three different commands implemented which come with its own set of flags and parameters:

1. Command version
2. Command init
3. Command analyze

Command version

The version command outputs the version number of this CLI-program like so:

hifis-surveyval version

Command init

Before you start the analysis you may want to change the defaults of the configuration variables. In order to do so, you can create a configuration file that is named hifis-surveyval.yml by issuing the init command:

hifis-surveyval init

This file contains the following information:

ANONYMOUS_QUESTION_ID: _
DATA_ID_SEPARATOR: _
ID_COLUMN_NAME: id
METADATA: metadata/meta.yml
OUTPUT_FOLDER: output
OUTPUT_FORMAT: SCREEN
PREPROCESSING_FILENAME: preprocess.py
SCRIPT_FOLDER: scripts
SCRIPT_NAMES: []

• ANONYMOUS_QUESTION_ID defines a placeholder for Questions IDs. The CSV data might not explicitly mention a full ID of a Question but solely the QuestionCollection ID. In this case the HIFIS Surveyval Framework adds a character to the QuestionCollection ID to mark this situation.
• There are different ways to separate the QuestionCollection ID from the Question ID in the full ID mentioned in the header of the CSV data. DATA_ID_SEPARATOR indicates which character is used to separate these IDs.
• With ID_COLUMN_NAME you may want to specify the name of the id column in the CSV data file.
• Each analysis needs metadata about the questions asked in the survey and answers that participants may give. Setting METADATA specifies the location of the Metadata file which is by default located in a folder called metadata and named meta.yml.
• You may specify the output folder by setting OUTPUT_FOLDER which is named output by default.
• You may prefer a specific output format like PDF, PNG, SVG or SCREEN which you may select via OUTPUT_FORMAT. The default value is SCREEN. Note: Be aware that other output formats like text or markdown files may be created, which depends largely upon the implementation of the analysis scripts.
• You might want to tell the program where to find the preprocessing file preprocess.py that preprocesses and filters your survey data according to specific rules. You can do so by setting PREPROCESSING_FILENAME.
• You may specify the folder which contains the analysis scripts with setting SCRIPT_FOLDER, which is the scripts folder by default.
• With SCRIPT_NAMES you may select a subset of the analysis scripts available as a list that ought to be executed. This list is empty by default, which means, all scripts are executed.

Additional to the configuration file there are two more files created:

1. File preprocess.py is created in the root folder of the project.
2. File example_script.py is created in the scripts folder of the project.

Command analyze

The more interesting command is the analyze command which comes with a data-parameter. The data-parameter can not be omitted and need to be given explicitly in order to be able to start the analysis. This is an example of how to do the analysis:

hifis-surveyval analyze data/<data_file_name>.csv

Contribute with Own Analysis Scripts

Essential Requirements for Developing Own Analysis Scripts

As you might have read in the previous sections the actual analysis scripts reside in a specific folder called scripts. All scripts in that folder will be automatically discovered by the package hifis-surveyval when running the analysis. In order that the program recognizes the scripts in that folder as analysis scripts they need to fulfill the following two criteria:

1. The filename need to end on .py.
2. The file need to contain a function called run without any parameters.

"""
A dummy script for testing the function dispatch

.. currentmodule:: hifis_surveyval.scripts.dummy
.. moduleauthor:: HIFIS Software <[e-mail]>
"""

def run():
    print("Example Script")

If both requirements are satisfied the program will execute the run-functions of the analysis scripts in an arbitrary order.

File-System Structure of the Core Component

$ tree hifis_surveyval/
hifis_surveyval/
├── answer.py
├── cli.py
├── core
│   ├── environment.py
│   ├── __init__.py
│   └── settings.py
├── data.py
├── dispatch.py
├── globals.py
├── __init__.py
├── plot.py
├── question.py
└── util.py

Resources

Below are some handy resource links:

• Project Documentation
• Click is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.
• Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Geog Brandl and licnsed under the BSD license.
• pytest helps you write better programs.
• GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.

Author Information

HIFIS-Surveyval was created by HIFIS Software Services.

Contributors

We would like to thank and give credits to the following contributors of this project:

• Be the first to be named here!

License

Copyright © 2021 HIFIS Software [e-mail]

This work is licensed under the following license(s):

• Everything else is licensed under GPL-3.0-or-later

Please see the individual files for more accurate information.

Hint: We provided the copyright and license information in accordance to the REUSE Specification 3.0.