One Time Setup

# Clone the repository
git clone git@github.com:edx/edx-arch-experiments.git
cd edx-arch-experiments

# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.11 edx-arch-experiments

Local testing

Two options are available for testing changes locally:

First, via make test-shell a dockerized development envrionment can be launched to run pytest and do basic migration work.

Second, a local copy of the package can be installed into edx-platform. For example, if using devstack, copy or clone your branch into <devstack-parent>/src/edx-arch-experiments. Then, in an lms or cms shell, run pip install -e /edx/src/edx-arch-experiments. The plug-in configuration will automatically be picked up once installed, and changes will be hot reloaded.

Every time you develop something in this repo

# Activate the virtualenv
workon edx-arch-experiments

# Grab the latest code
git checkout main
git pull

# Install/update the dev requirements
make requirements

# Run the tests and quality checks (to verify the status before you make any changes)
make validate

# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>

# Using your favorite editor, edit the code to make your change.
vim …

# Run your new tests
pytest ./path/to/new/tests

# Run all the tests and quality checks
make validate

# Commit all your changes
git commit …
git push

# Open a PR and ask for review.

Fixed

• Accept plain or retired__user_-prefixed usernames when retiring per-user certificates.

Fixed

• Added trailing slash to retire_certs_s3_for_user/ URL pattern.

Added

• Added RetireCertificatesS3ForUserView (POST /api/certificates/v1/retire_certs_s3_for_user) for per-user certificate retirement as a step in retire_one_learner.py. Returns HTTP 207 on partial failure.

• Added unit tests for edx_arch_experiments.certificates.views.

Added

• Added optional ?batch_size=N query parameter to retire_certs_s3 to limit certificates processed per call.

Added

• Added RetireCertificatesS3View (POST /api/certificates/v1/retire_certs_s3) to handle deletion of retired learners’ PDF certificate files from S3 and mark the corresponding GeneratedCertificate records as deleted. This removes the need for the CheckRetireUsers Jenkins job to hold direct RDS database credentials — authentication is now via OAuth token. Supports ?dry_run=true query parameter. Returns HTTP 207 on partial failure so unprocessed certificates are retried on the next scheduled run.

Changed

• Updated dummy model for testing migrations (0003_rename_dummy_field2_boms227_dummy_field3)

Changed

• Updated dummy model for testing migrations (0002_rename_dummy_field_boms227_dummy_field2)

Added

• Dummy model for testing migrations and rollback.

Removed

• Removed codejail_service plugin app (unused experiment)

• Removed temporary rollout span tag code_owner_plugin used for the code owner monitoring move from edx-django-utils. Now that edx-django-utils monitoring has been disabled, this span tag serves no purpose.

Changed

• Completes code owner monitoring updates, which drops owner theme and finalizes the code owner span tags. See doc and ADR updates for more details.

  • The code_owner_theme_2 tag was dropped altogether.

  • The temporary suffix (_2) was removed from other span tags.

  • The code_owner (formerly code_owner_2) tag no longer includes the theme name.

  • The new name for the django setting is CODE_OWNER_TO_PATH_MAPPINGS (formerly CODE_OWNER_MAPPINGS).

  • The django setting CODE_OWNER_THEMES was dropped.

  • Updates the generate_code_owner_mappings.py script accordingly.

Removed

• Removes CodeOwnerMonitoringMiddleware, in favor of using new signals sent from edx-django-utils’s MonitoringSupportMiddleware.

Added

• Adds search script datadog_search.py, for searching Datadog monitors and dashboards.

Added

• Added Datadog monitoring app which adds code owner monitoring. This is the first step in moving code owner code from edx-django-utils to this plugin.

  • Adds near duplicate of code owner middleware from edx-django-utils.

  • Adds code owner span tags for celery using Datadog span processing of celery.run spans.

  • Uses temporary span tags names using _2, like code_owner_2, for rollout and comparison with the original span tags.

  • Span tag code_owner_2_module includes the task name, where the original code_owner_module does not. In both cases, the code owner is computed the same, because it is based on a prefix match.

Removed

• Deleted Datadog diagnostics plugin app and middleware, which are no longer in use in edxapp.

Added

• Datadog diagnostics middleware can now attempt to close anomalous spans. Can be enabled via Waffle flag datadog.diagnostics.close_anomalous_spans (controlled separately from logging feature).

Changed

• Datadog diagnostics now logs ancestor spans when an anomaly is encountered, up to a limit of 10 (controlled by new Django setting DATADOG_DIAGNOSTICS_LOG_SPAN_DEPTH). Spans are logged in full and on separate lines, so this logging is now much more verbose; consider only enabling this logging for short periods. Log format of first line has also changed slightly.

Added

• Added celery lifecycle logging for Datadog diagnostics, to be enabled using DATADOG_DIAGNOSTICS_CELERY_LOG_SIGNALS.

Fixed

• Fixed loading of DATADOG_DIAGNOSTICS_ENABLE, which was previously not loaded properly and therefore was always True. Also fixed loading of DATADOG_DIAGNOSTICS_MAX_SPANS, which was presumably broken as well.

Removed

• Removed early span-start logging. It never worked properly, possibly because workers are continually being destroyed and created, leading to high log volume.

Changed

• Datadog diagnostics will now log all span-starts for the first minute after server startup

• WARNING: Do not use this version; see 4.2.0 release notes.

Changed

• Dropped support for Python 3.8; only testing with 3.11 and above now.

Added

• DatadogDiagnosticMiddleware can now detect and log anomalous traces, enabled by Waffle flag datadog.diagnostics.detect_anomalous_trace

Added

• New middleware edx_arch_experiments.datadog_diagnostics.middleware.DatadogDiagnosticMiddleware for logging diagnostics on traces in Datadog.

Changed

• Added federated-content-connector to the generate_code_owners script.

Added

• Toggle DATADOG_DIAGNOSTICS_ENABLE for disabling that plugin quickly if needed. (Feature remains enabled by default.)

Fixed

• Limit the number of spans collected via new setting DATADOG_DIAGNOSTICS_MAX_SPANS, defaulting to 100. This may help avoid memory leaks.

• Make accidental class variables into member variables in datadog_diagnostics

Added

• Added datadog_diagnostics plugin app

Changed

• Added translatable-xblocks to the generate_code_owners script.

Added

• Added support for Python 3.12

Changed

• Updated ConfigWatcher to include the IDA’s name in change messages if CONFIG_WATCHER_SERVICE_NAME is set

• Enabled ConfigWatcher as a plugin for CMS

Added

• Add codejail_service app for transition to containerized codejail

Fixed

• ConfigWatcher should now respond to model events properly now that it registers receivers with strong references. (Tested in sandbox.)

Changed

• Add log message for each model the ConfigWatcher is listening to

• Ensure that ConfigWatcher only attaches receivers once

Changed

• Renamed ConfigWatcherApp to ConfigWatcher to be less redundant. This is technically a breaking change but the app was not in use yet.

• Enabled ConfigWatcher as a plugin for LMS

Added

• Add edx_arch_experiments.config_watcher Django app for monitoring Waffle changes

• Add script to get github action errors

• Add script to republish failed events

Fixed

• Removed default_app_config (deprecated in Django 3)

Added

• First release on PyPI.