Open edX Caliper Tracking app can be used to transform Open edX tracking events into Caliper standard compliant events.
Project description
Open edX Caliper Tracking can be used to transform the edX traditional event logs into Caliper Analytics Specifications provided by IMS Global. Generated logs can be consumed by any analytics application which is compatible with Caliper Standard. All the transformed logs are Caliper Certified.
Installation
To install openedx-caliper-tracking in your Open edX instance, please add the following line to your requirements file. (For most Open edX installations it should be located at edx-platform/requirements/edx/base.txt).
openedx-caliper-tracking==0.14.1
For manual installation:
pip install openedx-caliper-tracking
Configuration
To enable and use openedx-caliper-tracking:
Add ENABLE_EVENT_CALIPERIZATION flag under FEATURES in the following files:
lms.env.json
cms.env.json
These files should be located at /edx/app/edxapp/ directory, see the example below:
"FEATURES": { ... "ENABLE_EVENT_CALIPERIZATION": true, ... }
2. Add the following lines of code:
if FEATURES.get('ENABLE_EVENT_CALIPERIZATION'): INSTALLED_APPS.insert( INSTALLED_APPS.index('eventtracking.django.apps.EventTrackingConfig'), 'openedx_caliper_tracking' )
in the following files:
lms/envs/production.py (aws.py for hawthorn release)
cms/envs/production.py (aws.py for hawthorn release)
Note:
Must make sure that after doing all the required changes in env (lms.env.json and cms.env.json) and auth (lms.auth.json and cms.auth.json) files you restart the server to reflect the changes that you have made.
Sending logs to external APIs (Optional)
There are two ways we can send caliper transformed logs to any external API
Rest API
Kafka
Using REST API
To do this, we have to add the following configurations
Add ENABLE_CALIPER_EVENTS_DELIVERY flag under FEATURES in the following files:
lms.env.json
cms.env.json
These files should be located at /edx/app/edxapp/ directory, see the example below:
"FEATURES": { ... "ENABLE_CALIPER_EVENTS_DELIVERY": true, ... }
Add the key CALIPER_DELIVERY_ENDPOINT and its value in the env files (lms.env.json and cms.env.json).
Add the key CALIPER_DELIVERY_AUTH_TOKEN and its value in the auth files ( lms.auth.json and cms.auth.json).
Add the following lines of code:
if FEATURES.get('ENABLE_CALIPER_EVENTS_DELIVERY'): CALIPER_DELIVERY_ENDPOINT = ENV_TOKENS.get('CALIPER_DELIVERY_ENDPOINT') CALIPER_DELIVERY_AUTH_TOKEN = AUTH_TOKENS.get('CALIPER_DELIVERY_AUTH_TOKEN')
in the following files:
lms/envs/production.py (aws.py for hawthorn release)
cms/envs/production.py (aws.py for hawthorn release)
Using Kafka Broker API
To do this, we have to add the following configurations
Add ENABLE_KAFKA_FOR_CALIPER flag under FEATURES in the following files:
lms.env.json
cms.env.json
These files should be located at /edx/app/edxapp/ directory, see the example below:
"FEATURES": { ... "ENABLE_KAFKA_FOR_CALIPER": true, ... }
2. Add the following keys and their values in the lms.env.json and cms.env.json files. Please note that all parameters in the PRODUCER_CONFIG are unique to the broker instances. You can set whatever parameters are required for your instance.
"CALIPER_KAFKA_SETTINGS": { "PRODUCER_CONFIG": { "bootstrap_servers": [ "<List of Kafka Brokers URLs>" ], ... }, "TOPIC_NAME": "<Kafka Topic Name>", "ERROR_REPORT_EMAILS": [ "<Reporting Email Address 1>", "<Reporting Email Address 2>", ... ] "MAXIMUM_RETRIES": <An Integer> },
Keys |
Description |
---|---|
MAXIMUM_RETRIES |
Number of times the app will try to send the logs to Kafka in case of failure |
PRODUCER_CONFIG |
Configurations for initializing the Kafka Producer
|
TOPIC_NAME |
Topic name for the Kafka broker |
ERROR_REPORT_EMAILS |
Email Addresses to notify when number of failures exceeds the MAXIMUM_RETRIES |
3. Add the following keys and their values in the lms.auth.json and cms.auth.json files. Please note that all parameters in the PRODUCER_CONFIG are unique to the broker instances. You can set whatever parameters are required for you.
"CALIPER_KAFKA_AUTH_SETTINGS": { "PRODUCER_CONFIG": { ... "sasl_plain_username": "<Username>", "sasl_plain_password": "<Password>", "security_protocol": "<Secuirty Protocol>", "ssl_cafile": "<Path/to/the/ca/file>", ... } }
Keys |
Description |
---|---|
PRODUCER_CONFIG |
Configurations for initializing the Kafka Producer. Use this confiration to store all sensitive configuration like authentication parameters.
|
Add the following lines of code:
if FEATURES.get('ENABLE_KAFKA_FOR_CALIPER'): CALIPER_KAFKA_SETTINGS = ENV_TOKENS.get('CALIPER_KAFKA_SETTINGS') CALIPER_KAFKA_AUTH_SETTINGS = AUTH_TOKENS.get('CALIPER_KAFKA_AUTH_SETTINGS')
in the following files:
lms/envs/production.py (aws.py for hawthorn release)
cms/envs/production.py (aws.py for hawthorn release)
Location of Transformed Logs
Note: This doesn’t work locally.
Transformed events are logged using 'logging.handlers.SysLogHandler' with 'facility: local2'.
We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:
1. Create a log file with read/write permissions given to syslog user (e.g: /edx/var/log/caliper-analytics/caliper.log).
cd /edx/var/log mkdir -p caliper-analytics && cd caliper-analytics touch caliper.log chown syslog caliper.log
Create a mapping for 'local2' in the configuration files present in /etc/rsyslog.d/ (e.g: in 99-edx.conf).
local2.* /edx/var/log/caliper-analytics/caliper.log;tracking
Run the following command on server to restart the rsyslog daemon:
sudo service rsyslog restart
Location of Logs Whose Delivery to Kafka is failed
Note:
This doesn’t work locally. Do this only if you are sending logs to external source using Kafka broker API.
Transformed events are logged using 'logging.handlers.SysLogHandler' with 'facility: local3'.
We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:
1. Create a log file with read/write permissions given to syslog user (e.g: /edx/var/log/caliper-analytics/delivery_failure.log).
cd /edx/var/log mkdir -p caliper-analytics && cd caliper-analytics touch delivery_failure.log chown syslog delivery_failure.log
Create a mapping for 'local3' in the configuration files present in /etc/rsyslog.d/ (e.g: in 99-edx.conf).
local3.* /edx/var/log/caliper-analytics/delivery_failure.log;tracking
Run the following command on server to restart the rsyslog daemon:
sudo service rsyslog restart
Running Tests Locally
To run the unit tests of this app locally, follow the following steps:
Clone the repository
git clone git@github.com:ucsd-ets/openedx-caliper-tracking.git
Run the following command in the same directory in which you have cloned the repository
sudo ./openedx-caliper-tracking/openedx_caliper_tracking/tests/local_test_script.sh
License
The code in this repository is licensed under the GPL v3.0 unless otherwise noted. Please see LICENSE for details.
How To Contribute
To contribute, please make a pull request in the repository on Github . If you have any questions or issues, please feel free to open an issue on Github: Open edX Caliper Tracking.
Contributors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for openedx-caliper-tracking-0.14.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 399fb88721584d6d08ed04b2587639975e2a79b0d410519fe2a36e1b545f59e6 |
|
MD5 | e54a1235a8bd9b1919a3a55c45d4f438 |
|
BLAKE2b-256 | c9963a33d4fba5171d96c9c2d8927d61c9423d351de607d5aa94726dbc8f1d11 |
Hashes for openedx_caliper_tracking-0.14.1-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d09a60a3a82925dcd3937285579a1998e77e59c69a0f584292cea6c80feab38 |
|
MD5 | bfe2882e6bb1688b534c7f3b0cb4e2e9 |
|
BLAKE2b-256 | 2bdd376eee220a56f968406d90786c1fe661bcf400a7ddcc380d830d9dec409e |