openedx-caliper-tracking 0.14.3

Open edX Caliper Tracking app can be used to transform Open edX tracking events into Caliper standard compliant events.

Open edX Caliper Tracking can be used to transform the edX traditional event logs into Caliper Analytics Specifications provided by IMS Global. Generated logs can be consumed by any analytics application which is compatible with Caliper Standard. All the transformed logs are Caliper Certified.

Installation

To install openedx-caliper-tracking in your Open edX instance, please add the following line to your requirements file. (For most Open edX installations it should be located at edx-platform/requirements/edx/base.txt).

openedx-caliper-tracking==0.14.3

For manual installation:

pip install openedx-caliper-tracking

Configuration

To enable and use openedx-caliper-tracking:

1. Add ENABLE_EVENT_CALIPERIZATION flag under FEATURES in the following files:

• lms.env.json

• cms.env.json

These files should be located at /edx/app/edxapp/ directory, see the example below:

"FEATURES": {
    ...

    "ENABLE_EVENT_CALIPERIZATION": true,

    ...
}

2. Add the following lines of code:

if FEATURES.get('ENABLE_EVENT_CALIPERIZATION'):
    INSTALLED_APPS.insert(
        INSTALLED_APPS.index('eventtracking.django.apps.EventTrackingConfig'),
        'openedx_caliper_tracking'
    )

in the following files:

• lms/envs/production.py (aws.py for hawthorn release)

• cms/envs/production.py (aws.py for hawthorn release)

Note:

Must make sure that after doing all the required changes in env (lms.env.json and cms.env.json) and auth (lms.auth.json and cms.auth.json) files you restart the server to reflect the changes that you have made.

Sending logs to external APIs (Optional)

There are two ways we can send caliper transformed logs to any external API

• Rest API

• Kafka

Using REST API

To do this, we have to add the following configurations

1. Add ENABLE_CALIPER_EVENTS_DELIVERY flag under FEATURES in the following files:

• lms.env.json

• cms.env.json

These files should be located at /edx/app/edxapp/ directory, see the example below:

"FEATURES": {
    ...

    "ENABLE_CALIPER_EVENTS_DELIVERY": true,

    ...
}

2. Add the key CALIPER_DELIVERY_ENDPOINT and its value in the env files (lms.env.json and cms.env.json).

3. Add the key CALIPER_DELIVERY_AUTH_TOKEN and its value in the auth files ( lms.auth.json and cms.auth.json).

4. Add the following lines of code:

if FEATURES.get('ENABLE_CALIPER_EVENTS_DELIVERY'):
    CALIPER_DELIVERY_ENDPOINT = ENV_TOKENS.get('CALIPER_DELIVERY_ENDPOINT')
    CALIPER_DELIVERY_AUTH_TOKEN = AUTH_TOKENS.get('CALIPER_DELIVERY_AUTH_TOKEN')

in the following files:

• lms/envs/production.py (aws.py for hawthorn release)

• cms/envs/production.py (aws.py for hawthorn release)

Using Kafka Broker API

To do this, we have to add the following configurations

1. Add ENABLE_KAFKA_FOR_CALIPER flag under FEATURES in the following files:

• lms.env.json

• cms.env.json

These files should be located at /edx/app/edxapp/ directory, see the example below:

"FEATURES": {
    ...

    "ENABLE_KAFKA_FOR_CALIPER": true,

    ...
}

2. Add the following keys and their values in the lms.env.json and cms.env.json files. Please note that all parameters in the PRODUCER_CONFIG are unique to the broker instances. You can set whatever parameters are required for your instance.

"CALIPER_KAFKA_SETTINGS": {
    "PRODUCER_CONFIG": {
        "bootstrap_servers": [
            "<List of Kafka Brokers URLs>"
        ],
        ...
    },

    "TOPIC_NAME": "<Kafka Topic Name>",

    "ERROR_REPORT_EMAILS": [
        "<Reporting Email Address 1>",
        "<Reporting Email Address 2>",
        ...
    ]
    "MAXIMUM_RETRIES": <An Integer>
},

Keys	Description
MAXIMUM_RETRIES	Number of times the app will try to send the logs to Kafka in case of failure
PRODUCER_CONFIG	Configurations for initializing the Kafka Producer Can further contain: “bootstrap_servers”: List of Kafka Brokers URLs Any other supported paramter in the Kafka-python docs Please note that it’s better to store the sensitive information in the *.auth.json files
TOPIC_NAME	Topic name for the Kafka broker
ERROR_REPORT_EMAILS	Email Addresses to notify when number of failures exceeds the MAXIMUM_RETRIES

3. Add the following keys and their values in the lms.auth.json and cms.auth.json files. Please note that all parameters in the PRODUCER_CONFIG are unique to the broker instances. You can set whatever parameters are required for you.

"CALIPER_KAFKA_AUTH_SETTINGS": {
    "PRODUCER_CONFIG": {
        ...
        "sasl_plain_username": "<Username>",
        "sasl_plain_password": "<Password>",
        "security_protocol": "<Secuirty Protocol>",
        "ssl_cafile": "<Path/to/the/ca/file>",
        ...
    }
}

Keys

Description

PRODUCER_CONFIG

Configurations for initializing the Kafka Producer. Use this confiration to store all sensitive configuration like authentication parameters. For example: Use this to configure paramters like: sasl_plain_username sasl_plain_password security_protocol sasl_mechanism It can further contain: Any other supported paramter in the Kafka-python docs Please note that it’s better to store the insensitive information in the *.env.json files

4. Add the following lines of code:

if FEATURES.get('ENABLE_KAFKA_FOR_CALIPER'):
    CALIPER_KAFKA_SETTINGS = ENV_TOKENS.get('CALIPER_KAFKA_SETTINGS')
    CALIPER_KAFKA_AUTH_SETTINGS = AUTH_TOKENS.get('CALIPER_KAFKA_AUTH_SETTINGS')

in the following files:

• lms/envs/production.py (aws.py for hawthorn release)

• cms/envs/production.py (aws.py for hawthorn release)

Location of Transformed Logs

Note: This doesn’t work locally.

Transformed events are logged using 'logging.handlers.SysLogHandler' with 'facility: local2'.

We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:

1. Create a log file with read/write permissions given to syslog user (e.g: /edx/var/log/caliper-analytics/caliper.log).

cd /edx/var/log
mkdir -p caliper-analytics && cd caliper-analytics
touch caliper.log
chown syslog caliper.log

2. Create a mapping for 'local2' in the configuration files present in /etc/rsyslog.d/ (e.g: in 99-edx.conf).

local2.*                 /edx/var/log/caliper-analytics/caliper.log;tracking

3. Run the following command on server to restart the rsyslog daemon:

sudo service rsyslog restart

Location of Logs Whose Delivery to Kafka is failed

Note:

This doesn’t work locally. Do this only if you are sending logs to external source using Kafka broker API.

Transformed events are logged using 'logging.handlers.SysLogHandler' with 'facility: local3'.

We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:

1. Create a log file with read/write permissions given to syslog user (e.g: /edx/var/log/caliper-analytics/delivery_failure.log).

cd /edx/var/log
mkdir -p caliper-analytics && cd caliper-analytics
touch delivery_failure.log
chown syslog delivery_failure.log

2. Create a mapping for 'local3' in the configuration files present in /etc/rsyslog.d/ (e.g: in 99-edx.conf).

local3.*                 /edx/var/log/caliper-analytics/delivery_failure.log;tracking

3. Run the following command on server to restart the rsyslog daemon:

sudo service rsyslog restart

Running Tests Locally

To run the unit tests of this app locally, follow the following steps:

• Clone the repository

git clone [e-mail]:ucsd-ets/openedx-caliper-tracking.git

• Run the following command in the same directory in which you have cloned the repository

sudo ./openedx-caliper-tracking/openedx_caliper_tracking/tests/local_test_script.sh

License

The code in this repository is licensed under the GPL v3.0 unless otherwise noted. Please see LICENSE for details.

How To Contribute

To contribute, please make a pull request in the repository on Github . If you have any questions or issues, please feel free to open an issue on Github: Open edX Caliper Tracking.

Contributors

• Muhammad Zeeshan

• Tasawer Nawaz

• Aroosha Arif

• Osama Arshad

• Danial Malik

• Hamza Farooq

• Hassan Tariq

• Muhammad Umar Khan

• Tehreem Sadat

• Muhammad Arslan

• Saad Ali

• Husnain Raza Ghaffar