Panorama Tutor Plugin

Introduction

Panorama is the analytics solution developed by Aulasneo for Open edX. It is a complete stack that includes data extraction, load, transformation, visualization and analysis. The data extracted is used to build a datalake that can easily combine multiple LMS installations and even other sources of data.

This utility is in charge of connecting to the MySQL and MongoDB tables and extracting the most relevant tables. Then it uploads the data to the datalake and updates all tables and partitions.

Installation

The Tutor 22 / Verawood package is currently a prerelease. The commands below opt into the 22.x prerelease series; use them after the package is published. For an unpublished checkout, install with pip install -e . instead.

1. Install as a Tutor plugin:

   pip install --pre 'tutor-contrib-panorama>=22,<23'
   

2. Enable the plugin:

   tutor plugins enable panorama
   

Panorama Modes

Starting with version 16.3.0, the Tutor Panorama Plugin now offers four modes to use Panorama:

• DEMO: Full access to the standard Panorama service, with anonymized data.
• FREE: Hosted Panorama service for free, with limited functionalities.
• SAAS: Hosted Panorama service provided by Aulasneo with most typical features enabled.
• CUSTOM: Full Panorama flexibility, either in SaaS modality or self-hosted.

Since 16.3.0, the default mode of Panorama is DEMO.

Panorama DEMO Mode

In DEMO mode you can try the functionality of Panorama with anonymized data. You will be able to experiment with the power of Panorama as you would with your own data. What you will see is the actual Panorama SaaS solution from our production servers, showing the dashboards offered out of the box for the SAAS mode.

In DEMO mode, Panorama will not extract any data from your server.

To activate DEMO mode, install the plugin, rebuild the openedx and mfe images, and restart your deployment. No specific configuration is needed.

pip install --pre 'tutor-contrib-panorama>=22,<23'
tutor plugins enable panorama
tutor images build openedx
tutor images build mfe
tutor {local|k8s} restart

Panorama FREE Mode

Panorama FREE mode offers a basic yet powerful set of dashboards that you can use for free. It is part of the Aulasneo SaaS offering. To get your FREE credentials, please register at Panorama and send us an email to info@aulasneo.com.

In FREE mode, only the relational and courseware data is extracted. No logs are processed. Therefore you will not be able to get statistics based on events, such as video views, forum activity, or PDF downloads.

The free mode is part of the SaaS offering. Please be aware that data from your instance will be uploaded to our servers.

To activate FREE mode, install the plugin, rebuild the openedx and mfe images, and restart your deployment. No specific configuration is needed. Contact us at info@aulasneo.com to get the additional settings needed to activate Panorama.

pip install --pre 'tutor-contrib-panorama>=22,<23'
tutor plugins enable panorama
tutor images build openedx
tutor images build mfe
tutor {local|k8s} restart

Panorama SaaS Mode

Panorama SAAS mode offers a full set of dashboards that you can use out of the box. This is a paid service offered by Aulasneo to any Open edX user.

Please be aware that data from your instance will be uploaded to our servers.

To connect to Panorama SaaS, please contact us at info@aulasneo.com to get instructions.

pip install --pre 'tutor-contrib-panorama>=22,<23'
tutor plugins enable panorama
tutor images build openedx
tutor images build mfe
tutor {local|k8s} restart

Panorama Custom Mode

The Panorama CUSTOM mode offers the highest flexibility to use Panorama. To set up custom mode, you will have to deploy your own data infrastructure.

Setting Up the Datalake

The Panorama plugin for Tutor is configured to work with an AWS datalake.

To set up your AWS datalake, you will need to:

• Create or use an IAM user or role with permissions to access the S3 buckets, KMS if encrypted, Glue, and Athena.
• Create one S3 bucket to store the data, one for raw logs (optional), and another as the Athena query results location.
• Use encrypted buckets and strict access policies to prevent unauthorized access.
• Create the Panorama database in Athena with CREATE DATABASE panorama.
• Create the Athena workgroup panorama to keep the queries isolated from other projects.
• Set the query result location to the bucket created for this workgroup.

User Permissions to Work With the AWS Datalake

In order to work with an AWS datalake, you will need to create a user, for example panorama-elt, and assign a policy, for example PanoramaELT, with at least the following permissions.

Replace <panorama_data_bucket>, <panorama_logs_bucket>, <panorama_athena_bucket>, <region>, and <account_id> with proper values.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": [
        "arn:aws:s3:::<panorama_data_bucket>/openedx/*",
        "arn:aws:s3:::<panorama_logs_bucket>/tracking_logs/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::<panorama_data_bucket>/PanoramaConnectionTest"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<panorama_athena_bucket>",
        "arn:aws:s3:::<panorama_athena_bucket>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:BatchCreatePartition",
        "glue:GetDatabase",
        "athena:StartQueryExecution",
        "glue:CreateTable",
        "athena:GetQueryExecution",
        "athena:GetQueryResults",
        "glue:GetDatabases",
        "glue:GetTable",
        "glue:DeleteTable",
        "glue:GetPartitions",
        "glue:UpdateTable"
      ],
      "Resource": [
        "arn:aws:athena:<region>:<account_id>:workgroup/panorama",
        "arn:aws:glue:<region>:<account_id>:database/panorama",
        "arn:aws:glue:<region>:<account_id>:catalog",
        "arn:aws:glue:<region>:<account_id>:table/panorama/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "*"
    }
  ]
}

If you have encrypted S3 buckets with KMS, you may need to add permissions to get the KMS keys.

Additionally, the user must have Lake Formation permissions to access the data locations and query the database and all tables.

Finally, you will have to connect QuickSight to Athena to visualize the data.

Configuration

Tutor 22 / Verawood implementation and validation details are in docs/verawood-validation.md. Panorama remains a standalone MFE; the plugin contributes native frontend-base navigation as well. Before testing new images, set PANORAMA_OPENEDX_BACKEND_VERSION to the PyPI package version (default 22.0.0), and PANORAMA_MFE_REPO / PANORAMA_MFE_VERSION, and PANORAMA_ELT_REPO / PANORAMA_ELT_VERSION to your reviewed source references. The MFE defaults to release/verawood/v20260926; the extractor still defaults to v1.0.1. Confirm the selected backend package and source references contain the changes being validated before building. For standalone MFE builds Tutor also supports a mounted source build context.

PANORAMA_MFE_ENABLED=False disables the MFE, navigation, backend install/settings and migrations without disabling separately configured extraction. Backend init skips DEMO mode and otherwise only applies committed migrations. Backend installation uses the target platform's requirements as constraints and runs pip check. The install step explicitly bind-mounts the constraints from Tutor's edx-platform build stage because the mount used by Tutor's earlier install step does not persist.

Recurring jobs forbid overlap, retain one successful/two failed jobs and default to one retry, a 3300-second active deadline and a 300-second scheduling deadline. Adjust PANORAMA_K8S_JOB_BACKOFF_LIMIT, PANORAMA_K8S_JOB_ACTIVE_DEADLINE_SECONDS and PANORAMA_K8S_CRON_STARTING_DEADLINE_SECONDS for the configured schedule. The one-off initialization Job retains Kubernetes default retries and has no active deadline. Both jobs use PANORAMA_K8S_JOB_MEMORY_REQUEST and PANORAMA_K8S_JOB_MEMORY_LIMIT.

Fluent Bit uses a digest-pinned 3.4.15 image (Fluent Bit 5.0.9). Enable PANORAMA_RUN_K8S_FLUENTBIT in exactly one SAAS/CUSTOM site per cluster and disable it in every other site. That site's DaemonSet reads lms*.log across all namespaces, excluding lms-worker*.log, using the collector site's bucket/credentials. This selects LMS pod log files, including matching sidecars; it does not filter by container name. RBAC names and host state are scoped to the hosting installation, not the source sites. Offsets and S3 buffers persist under /var/lib/panorama-fluentbit/<collector-namespace> on each Kubernetes node; Compose uses Tutor's data/panorama-fluentbit directory. Node/disk loss is not covered by this persistence. On Linux, Docker creates the Compose state directory and the root-running collector writes state as root; cleanup can require sudo. If running the collector as a different user, provision this directory with ownership writable by that container UID before starting Compose. Default S3 spool capacity is 1G (PANORAMA_FLB_STORE_LIMIT), tail memory 32M (PANORAMA_FLB_TAIL_MEMORY_LIMIT) and line limit 1M (PANORAMA_FLB_LINE_MAX_SIZE). Oversized physical lines are skipped with a warning so subsequent records remain readable; spool exhaustion can lose data. Monitor object arrival and disk/backlog growth rather than relying on output-success counters. Kubernetes keys retain /tracking_logs/<source-namespace>/...; Compose keys retain their existing LMS-host paths. PANORAMA_LOGS_UPLOAD_CHUNK_SIZE is retained as a deprecated configuration key but is unused with PutObject. Drain existing buffers before switching collectors and account for the new offset path when checking duplicates at cutover.

When upgrading a Kubernetes collector from the legacy global RBAC names, applying the new manifest does not remove the old objects. Complete this migration once per cluster after the replacement collector is running:

1. Inspect kubectl get clusterrolebinding pod-log-crb -o yaml and kubectl get clusterrole pod-log-reader -o yaml. Confirm these are the legacy Panorama resources and identify every service account relying on them.

2. Verify the new panorama-<collector-namespace>-pod-log-crb binding references panorama-<collector-namespace>-pod-log-reader and the active collector's fluent-bit service account. Confirm the collector can retrieve Kubernetes metadata. Migrate any other consumers before removing their old grants.

3. Remove the confirmed legacy binding, then the unused legacy role:

   kubectl delete clusterrolebinding pod-log-crb --ignore-not-found
   kubectl delete clusterrole pod-log-reader --ignore-not-found
   

Do not delete a role still referenced by another binding. These are cluster-wide resources; disabling a site's collector or applying the new manifest alone does not revoke a leftover legacy binding.

Set the following variables to configure Panorama:

Datalake Directory Structure

For each table, or for each field-based partition in each table when enabled, one CSV file will be generated and uploaded. The file will have the same name as the table, with a .csv extension.

Each CSV file will be uploaded to the following directory structure:

s3://<bucket>/[<base prefix>/]<table name>/[<base partitions>/][field partitions/]<table name>.csv

Where:

• bucket: Bucket name, configured in the panorama_raw_data_bucket setting.
• base prefix: Optional subdirectory to hold tables of the same kind of system, for example openedx. It can receive files from multiple sources, as long as the table names are the same and share a field structure.
• table name: Base location of the datalake table. All text files inside this directory must have exactly the same column structure.
• base partitions: Partitions common to the same installation, in Hive format. These are not based on fields in the data sources, but will appear as fields in the datalake. For multiple Open edX installations, the default is to use lms as field name and the LMS host as the value, for example lms=openedx.example.com.
• field partitions: Optional partitioning for large tables. The field will be removed from the CSV file, but will appear as a partition field in the datalake. In Open edX installations, the default setting is to partition courseware_studentmodule by course_id.

License

This software is licensed under the Apache 2.0 license. Please see LICENSE for more details.

Contributing

Contributions are welcome. Please submit your PR and we will review it. For questions, please send an email to andres@aulasneo.com.