mano 0.6.0

Python API for the Beiwe Research Platform

Mano

Mano is a Python library and CLI tool that helps you write applications that interact with the Beiwe Research Platform.

Mano is developed and maintained by Onnela Lab, at the Harvard T.H. Chan School of Public Health, and is part of The Beiwe Platform. It is distributed under the open source BSD-3-Clause license.

Current List of Features:

• Programmatically request lists of the studies you are authorized on from your Beiwe server.
• Request additional information: the list of participants in your study, and the device settings currently enabled on your study.
• Download data generated by the participants in your study.

[!Important] Some features of Mano may not be available if the server is running an older version of the Beiwe Backend. Please let your System Administrator know if you run into any issues. The Beiwe Platform provides extensive documentation for sysadmins to handle any issues they may encounter in the upgrade process. Resources can be found on the Beiwe Backend Wiki, and they can follow announcement issues on backend issues.

If you are encountering any other issue Please Report Them Here

Table of contents

1. Requirements and Compatibility
2. Installation
3. Initial Setup
4. API For Keyring Access
5. API For Accessing Study Information
6. API For Downloading Data

Requirements and Compatibility

• Mano is compatible with modern versions of Python [at time of writing this means 3.10+]
• Mano's CI runs on macOS (Unix), Ubuntu (Linux), and Windows.
  • We have not historically had many Windows users, so Please Report Any Issues You Encounter with Windows compatibility.
  • Mano should be fully compatible with the Linux Subsystem for Windows (WSL).

Old SSL Library Compatibility

Beiwe servers require modern, secure connections, so old versions of SSL/TLS libraries provided by your operating system may cause issues. The simplest known solution is to install one of the Miniconda Python distributions, which bundles a more up to date version of OpenSSL.

Installation

The simplest way to install mano is to just use pip

pip install mano

Initial Setup

To interact with Beiwe and download files you will need your Beiwe Platform url, username, password, access key, and secret key in a JSON file. Don't worry, we're going to eventually encrypt this file. (Note: we are phasing out the need for the password field, but Beiwe servers running an older version of the backend software may still require it.)

{
    "beiwe.onnela": {
        "URL": "...",
        "USERNAME": "...",
        "PASSWORD": "...",
        "ACCESS_KEY": "...",
        "SECRET_KEY": "..."
    }
}

[!Tip] You can also use the environment variables BEIWE_URL, BEIWE_USERNAME, BEIWE_PASSWORD, BEIWE_ACCESS_KEY, and BEIWE_SECRET_KEY to store these settings. If you do, load your keyring using mano.keyring(None). You won't be able to use an environment variable for storing study-specific secrets (next).

[!Note] You generate, name and manage your access keys under the Manage Credentials section of your Beiwe Platform website.

If you wish to use mano to encrypt certain downloaded data stream files at rest, you should provide a study-specific passphrase (which you must generate) in an extra SECRETS section.

{
    "beiwe.onnela": {
        "URL": "...",
        "USERNAME": "...",
        "PASSWORD": "...",
        "ACCESS_KEY": "...",
        "SECRET_KEY": "...",
        "SECRETS": {
            "Beiwe Study Omega": "...",
        }
    }
}

You don't want this file sitting around as plain text, so Mano requires you encrypt it. Mano uses the crypt.py utility from the cryptease library which was installed along with the mano package.

$ crypt.py --encrypt ~/.nrg-keyring.json --output-file ~/.nrg-keyring.enc

[!Note] "crypt.py" looks like a file name, but it is an executable command. You may have a common CLI program named simply "crypt" installed on your system (or even autocompleted in your CLI), it is not the same thing and you should not confuse them.

It is up to you to decide where to store the encrypted version of this file, but we strongly recommend deleting the unencrypted version.

API For Keyring Access

Before making any API calls Mano must read in your keyring file. The first parameter is the name of the keyring section as shown above

import mano
Keyring = mano.keyring('beiwe.onnela')

Mano still requires that you provide the decryption key for your keyring file. By default it will prompt you to type it in directly, but there are two mechanisms for providing it programmatically.

• Setting the environment variable NRG_KEYRING_PASS where Mano is running.
• As the second argument to the mano.keyring function in your code.
  • We recommend against placing the decryption key as text in your code, or in any file that gets committed to a source control system like Git. This mechanism is provided so that you can programmatically source it from another location.
  • It's tough to know where to store a credential securely. If you are on your own computer we recommend using the full drive encryption capability of your operating system to secure it.

[!Important] Non-interactive invocations of your code that do not have access to a decryption key will probably cause your code to hang as it waits for user input that cannot happen.

API For Accessing Study Information

With your Keyring loaded you can now access information about your studies, users (a.k.a. participants, subjects), and device settings using simple functions defined within the mano module.

for study in mano.studies(Keyring):
    print(study)

_, study_id = study  # get the last printed study id

for user_id in mano.users(Keyring, study_id):
    print(user_id)

for setting in mano.device_settings(Keyring, study_id):
    print(setting)

API For Downloading Data

With your Keyring loaded, you can download collected data from your Beiwe server and extract it to your filesystem using the mano.sync module. While we're at it, we will turn on more verbose logging so we can see what's happening.

[!Note] The msync.download function returns a Python Standard Library zipfile.ZipFile object from which you extract files.

import logging
from mano import sync as msync

logging.basicConfig(level=logging.INFO)

output_folder = '/tmp/beiwe-data'  # set this to a real folder location

zf = msync.download(Keyring, study_id, user_id, data_streams=['identifiers'])

zf.extractall(output_folder)

[!WARNING] We passed data_streams=['identifiers'] to msync.download. Without that parameter that function will request all data for all data streams, which may amount to many gigabytes of data. Check out the backfill section for more information.

Encrypt Data Files At Rest

You can pass the ZipFile object to msync.save if you wish to encrypt data stream files.

lock_streams = ['gps', 'audio_recordings']

zf = msync.download(Keyring, study_id, user_id)

data_encryption_key = Keyring['SECRETS']['Beiwe Study Omega']  # not the keyring decryption key!

msync.save(
    Keyring,
    zf,
    user_id,
    output_folder,
    lock=lock_streams,
    passphrase=data_encryption_key,
)

Backfill

By default msync.download attempts to download all of the data for the specified user_id, which could end up being prohibitively large. For this reason, the msync.download function exposes parameters for data_streams, time_start, and time_end. By using these parameters you can limit your download operation to those constraints.

data_streams = ['accel', 'ios_log', 'gps']

time_start = '2015-10-01T00:00:00'
time_end = '2015-12-01T00:00:00'

zf = msync.download(
    Keyring,
    study_id,
    user_id,
    data_streams=data_streams,
    time_start=time_start,
    time_end=time_end,
)

zf.extractall(output_folder)

[!Note] The full list of data stream keys is: accelerometer, app_log, audio_recordings, bluetooth, calls, devicemotion, gps, gyro, identifiers, image_survey, ios_log, magnetometer, power_state, proximity, reachability, survey_answers, survey_timings, texts, and wifi.

Eventually you may find yourself day-dreaming about a backfill function that will slide a window from some arbitrary starting point to the present time in order to download all of your data in more digestible chunks. You'll be happy to know that the mano.sync module exposes a function for this.

start_date = '2015-01-01T00:00:00'

msync.backfill(
    Keyring,
    study_id,
    user_id,
    output_folder,
    start_date=start_date,
    lock=lock_streams,
    passphrase=passphrase,
)

[!Note] If you don't pass anything for the lock argument, you will not need passphrase either.