platform-plugin-aspects 0.11.4

Aspects plugins for edx-platform

Platform Plugin Aspects

Purpose

This repository holds various Aspects plugins for the Open edX platform.

Sinks

Sinks are components that listen for events and store them in ClickHouse. The events are emitted by the Open edX platform via Open edX events or Django signals.

Available Sinks

• CourseOverviewSink - Listens for the COURSE_PUBLISHED event and stores the course structure data through XBlockSink in ClickHouse.

• ExternalIdSink - Listens for the post_save Django signal on the ExternalId model and stores the external id data in ClickHouse.

• UserProfile - Listens for the post_save Django signal on the UserProfile model and stores the user profile data in ClickHouse.

• UserRetirementSink - Listen for the USER_RETIRE_LMS_MISC Django signal and remove the user PII information from ClickHouse.

• CourseEnrollmentSink - Listen for the ENROLL_STATUS_CHANGE event and stores the course enrollment data.

Commands

In addition to being an event listener, this package provides the following commands:

• dump_data_to_clickhouse - This command allows bulk export of the data from the Sinks. Allows bootstrapping a new data platform or backfilling lost or missing data.

  python manage.py cms dump_data_to_clickhouse

• load_test_tracking_events - This command allows loading test tracking events into ClickHouse. This is useful for testing the ClickHouse connection to measure the performance of the different data pipelines such as Vector, Event Bus (Redis and Kafka), and Celery.

  Do not use this command in production as it will generate a large amount of data and will slow down the system.

  python manage.py cms load_test_tracking_events

• monitor_load_test_tracking - Monitors the load test tracking script and saves output for later analysis.

  python manage.py cms monitor_load_test_tracking

Instructor Dashboard Integration

Dashboards from Aspects are integrated into the Instructor Dashboard via Superset Embedded SDK. See Configuration for more details.

Getting Started with Development

Please see the Open edX documentation for guidance on Python development in this repo.

Deploying

The Platform Plugin Aspects component is a django plugin which doesn’t need independent deployment. Therefore, its setup is reasonably straightforward. First, it needs to be added to your service requirements, and then it will be installed alongside requirements of the service.

Configuration

Instructor Dashboard Configuration

The Instructor Dashboard integration uses the Open edX Filters. To learn more about the filters, see the Open edX Filters documentation. Make sure to configure the superset pipeline into the filter as follows:

OPEN_EDX_FILTERS_CONFIG = {
  "org.openedx.learning.instructor.dashboard.render.started.v1": {
    "fail_silently": False,
    "pipeline": [
      "platform_plugin_superset.extensions.filters.AddSupersetTab",
    ]
  },
}

• SUPERSET_CONFIG - This setting is used to configure the Superset Embedded SDK. The configuration is a dictionary that contains the following keys:

  • internal_service_url - The URL of the Superset instance (useful in development, omit in production).

  • service_url - The URL of the Superset instance.

  • username - The username of the Superset user.

  • password - The password of the Superset user.

• ASPECTS_INSTRUCTOR_DASHBOARDS - This setting is used to configure the dashboards that will be displayed in the Instructor Dashboard. The configuration is a list of dictionaries that contains the following keys:

  • name - The name of the dashboard.

  • slug - The slug of the dashboard.

  • uuid - The UUID of the dashboard.

  • allow_translations - A boolean value that determines if the dashboard is translated in Aspects.

• SUPERSET_EXTRA_FILTERS_FORMAT - This setting is used to configure the extra filters that will be applied to the dashboards. The configuration is a list of strings that can be formatted with the following variables:

  • user - The user object.

  • course - The course object.

• SUPERSET_DASHBOARD_LOCALES - This setting is used to configure the available locales for the dashboards. The configuration is a list of supported locales by Aspects.

Event Sink Configuration

• EVENT_SINK_CLICKHOUSE_BACKEND_CONFIG - This setting is used to configure the ClickHouse connection. The configuration is a dictionary that contains the following keys:

  • url - The host of the ClickHouse instance.

  • database - The database name.

  • username - The username of the ClickHouse user.

  • password - The password of the ClickHouse user.

  • timeout_secs - The timeout in seconds for the ClickHouse connection.

• EVENT_SINK_CLICKHOUSE_PII_MODELS - This setting is used to configure the models that contain PII information. The configuration is a list of strings that contain the table names where the PII information is stored.

• EVENT_SINK_CLICKHOUSE_MODEL_CONFIG - This setting is used to provide compatibility with multiple Open edX models. The configuration is a dictionary that contains the following a key per model that contains a dictionary with the following keys:

  • module - The module path of the model.

  • model - The model class name.

Event Sinks are disabled by default. To enable them, you need to enable the following waffle flag: event_sink_clickhouse.{{model_name}}.enabled where model name is the name of the model that you want to enable. Or, you can enable them via settings by setting EVENT_SINK_CLICKHOUSE_{{model_name}}_ENABLED to True.

Getting Help

Documentation

PLACEHOLDER: Start by going through the documentation. If you need more help see below.

(TODO: Set up documentation)

More Help

If you’re having trouble, we have discussion forums at https://discuss.openedx.org where you can connect with others in the community.

Our real-time conversations are on Slack. You can request a Slack invitation, then join our community Slack workspace.

For anything non-trivial, the best path is to open an issue in this repository with as many details about the issue you are facing as you can provide.

https://github.com/openedx/platform-plugin-aspects/issues

For more information about these options, see the Getting Help page.

License

Please see LICENSE.txt for details.

Contributing

Contributions are very welcome. Please read How To Contribute for details.

This project is currently accepting all types of contributions, bug fixes, security fixes, maintenance work, or new features. However, please make sure to have a discussion about your new feature idea with the maintainers prior to beginning development to maximize the chances of your change being accepted. You can start a conversation by creating a new issue on this repo summarizing your idea.

The Open edX Code of Conduct

All community members are expected to follow the Open edX Code of Conduct.

People

The assigned maintainers for this component and other project details may be found in Backstage. Backstage pulls this data from the catalog-info.yaml file in this repo.

Reporting Security Issues

Please do not report security issues in public. Please email [e-mail].

Change Log

Unreleased

0.11.2 - 2024-10-04

Fixes

• wait for transaction commit before trying to sink models.

0.11.1 - 2024-09-06

Fixes

• delay loading of unfocused dashboards in LMS.

0.11.0 - 2024-09-04

Added

• A sink for the object tags, tags and taxonomy.

0.10.0 - 2024-06-17

Added

• A sink for the course enrollment model.

0.9.7 - 2024-06-14

Fixes

• 0.9.6 introduced a bug in the CourseOverview sink that caused block sink to fail to import to ClickHouse, this fixes it.

0.9.6 - 2024-06-07

Fixes

• The CourseOutline sink was not always ordering blocks correctly, leading to issues with blocks appearing in the wrong sections/subsection.

0.9.5 - 2024-05-24

Fixes

• UserProfile sink now runs after the transaction is committed, preventing UserProfileNotFound errors and creation of rows in ClickHouse that don’t exist in MySQL in the case of a rollback.

0.9.4 - 2024-05-16

Fixes

• Allow to serialize dates as strings in JSON.

0.9.3 - 2024-05-15

Fixes

• Change wording of the “go to Superset” link, and make it optional.

0.9.2 - 2024-05-08

Fixes

• Remove caching of Superset guest token to fix various display errors and token refresh edge cases.

0.9.1 - 2024-05-08

Fixes

• Fix an ImportError in the Aspects Xblock on pre-Quince releases

0.9.0 - 2024-05-06

Added

• Allow embedded dashboard tab names to be localized

0.8.0 - 2024-05-06

Added

• Added tags to xblocks in course dump

0.7.4 - 2024-04-30

Fixed

• Fixed Superset XBlock and default filters.

0.7.3 - 2024-04-30

Fixed

• Fixed UUID generation for localized Superset assets, which caused embedded dashboards to fail to load when localized.

0.7.2 - 2024-04-19

Fixed

• Fixed cms url configuration

0.7.1 - 2024-04-17

Fixed

• Fixed issue with embedded dashboards throwing javascript errors

• Fixed issues with translated embedded dashboards erroring in Superset

0.7.0 - 2024-04-12

Added

• Add endpoint for fetchGuestToken

0.6.0 - 2024-04-08

Added

• Allow to embed translated Superset Dashboards.

0.5.0 - 2024-04-01

Added

• Load testing and test monitoring scripts.

0.4.0 - 2024-03-18

Added

• Embed multiple Superset Dashboards.

0.3.1 - 2024-03-14

Added

• Fixed development server configuration.

0.3.0 – 2024-03-10

Added

• Imported XBlock code from platform-plugin-superset

0.2.0 – 2024-03-05

Added

• Imported code from event-sink-clickhouse.

0.1.0 – 2024-02-29

Added

• First release on PyPI.