redash-stmo 2020.5.1

Extensions to Redash by Mozilla

Redash extensions for sql.telemetry.mozilla.org.

Or as it should have been called: St. Moredash ;)

Overview

Inherits Redash’s Docker setup

redash-stmo is using Redash’s own Docker image for development to implement a close development/production parity and extends it in various ways, e.g. an own docker-compose configuration, an own docker-entrypoint script.

Specifically it uses Mozilla’s “rc” tagged version of the Redash Docker image, which includes (at the time of writing this, 2019-06-13) many customizations from Mozilla’s pseudo-temporary and regularly updated Redash fork. The “rc” tagged Docker image is updated every time a “rebase” from upstream Redash happens and is put to testing in the “release” Redash environment on Mozilla’s server.

Please review the Redash Docker installation guidelines before continuing. It’s important to know those basics since many decisions for redash-stmo were derived from it. Thank you.

Is mounted under /extension

The current working directory (the directory with this README.rst) is mounted under the path /extension by docker-compose inside the Docker container.

Runs with Redash in /app

Since it reuses the Redash Docker image, you can find all the Redash setup under the /app directory inside the Docker container.

Uses Redash’s “entrypoints” for discovery

The way Redash finds new extensions is by using the so called “entrypoints” of Python packages, metadata that is specified and distributed in Python packages, that is read out by Redash at runtime to find the filesystem locations for Redash extensions.

That’s true for three kinds of entrypoints:

redash.extensions

Python callables to be used to extend the Redash Flask app, e.g. redash_stmo.data_sources.health:extension.

redash.bundles

Python packages that contain additional front-end files for the webpack build process, e.g. redash_stmo.data_sources.link.

redash.scheduled_jobs

Python callables that return parameters for scheduled RQ jobs, e.g. redash_stmo.data_sources.health:scheduled_job.

Hooks into Webpack

Since Redash extensions like redash-stmo can also provide additional Webpack bundles, the development setup runs Redash’s bundle-extension script periodically to copy the files from redash-stmo to the right place for webpack to pick them up (/app/client/app/extensions).

See the section about the webpack development server below for more information.

Development workflow

We provide some convenience Make tasks to be run from your host machine (not inside the Docker container) to ease this non-trivial application setup:

Build and update the local Docker image

Many of the helpers below will implicitely run docker-compose to start the Redash containers and in effect automatically build the local Docker image as well if it doesn’t exist.

If you’d like to build the local Docker image separately or if you’d like to fetch the latest version of the base Redash Docker image (or its child image of the Mozilla Redash fork), e.g. in the event of a new rebase by Mozilla staff, please run the following:

make build

Behind the scenes

This will run docker-compose build --pull which will pull updates to the Docker images used by the docker-compose setup, including the Redash, Redis and Postgres images.

Create the database

On you command line run this ONCE to create the database for Redash/redash-stmo setup:

make database

This uses Redash’s own ability and redash-stmo is just set up to reuse it.

Behind the scenes

This will run docker-compose to create the server container that is running the Redash Python server and in effect the Redis and Postgres containers, too.

It will then initialize the Postgres tables needed for Redash.

Install npm modules

Then we’ll install the Redash npm modules inside the server container:

make node_modules

Behind the scenes

This will run npm install inside the server in the /app directory, which is the directory with Redash’s code from the Redash Docker base image.

NOTE, the redash-stmo development setup mounts the /app/node_modules directory as a separate Docker volume, that will be maintained by Docker and won’t show up in or transfer to the host machine where Docker is running.

Start the containers

To start the whole set of Docker containers for a working environment (Redash server, RQ workers, Redis, Postgres) all you need to run is this:

make up

Behind the scenes

This is pretty simply running docker-compose up, to launch all containers of the redash-stmo Docker setup.

NOTE: This requires first installing npm modules inside the container above and creating the database as well!

Run webpack devserver

If you’re developing a Redash extension that includes an additional webpack bundle (which will need to be included in Redash’s webpack build process to be shipped in the client application bundle) you’ll want to use the webpack development server.

It automatically compiles the Redash client application bundle on files changes and proxies requests for the Redash server via a proxy running on port 8080 (instead of the usual Redash port of 5000).

After starting the containers using the description in the above step, open a second terminal and additionally run:

make devserver

Behind the scenes

This will run the webpack devserver in another instance of the server container (not the same as when running make up) and runs a script that listens for files changes to .js and .jsx files in the /extension directory.

When changes are detected, it’ll automatically run Redash’s bundle-extensions script that does the heavy lifting of copying the changed extension files into the /app/client/app/extensions directory, which triggers the webpack devserver to recompile the client application bundle.

NOTE: This requires opening the Redash instance via http://localhost:8080/ instead of http://localhost:5000/ to go through the webpack devserver.

Start shell

In case you need to do any debugging or file system checks inside the server container, you can create a bash shell by running:

make bash

Behind the scenes

Any changes you make here outside the /extension directory (which is mounted as a Docker volume with the current working directory on the Docker host machine) and the following directores are not persisted.

List of directories inside the container that are mounted as Docker volumes:

/extension

Maps the current working directory (where this README.rst is located) on the host machine for developing the extension.

/home/redash/.cache

Used by pip and other scripts,

/app/client/dist

Directory to retain webpack build results, so webpack builds don’t take as long on consecutive runs.

/home/redash/.local

Directory for “user-installed” Python packages. If you’d like you can easily install additonal Python packages with the Docker container user Redash using pip install --user <package>. Installed scripts from those packages will be found under /home/redash/.local/bin but are also automatically added to PATH.

/app/node_modules

Directory for npm modules, that are installed when running npm install inside of /app in the container. Retained to make use of native npm caching between consecutive runs.

Run tests

Running the Python based tests requires first creating a separate database (implemented by the test_database Make task) and then running the test runner inside the container. The test database is not the same as the databse in use for regular development (e.g. to not overwrite development data).

Frontend or integration tests are currently not supported.

To run the tests (from the host machine) run:

make test

This will automatically run the test_database Make task before running the tests.

Behind the scenes

When launching the tests runner it’ll the regular server container, but also set the REDASH_DATABASE_URL environment variable to the test database to prevent overwriting any data that you added to the database the regular Redash interface (e.g. data sources, queries etc).

By default it uses pytest to run the Python tests in /extension, with a number of parameters as defined in the pytest.ini.

If you’d like to add additional parameters to pytest simply appened the command line arguments in pytest.ini.

Alternatively, e.g. if you’d like to use pdb to debug a test, do this:

create the test database from the host machine

make test_database

start a Bash shell in the container

make bash

set the REDASH_DATABASE_URL env var in the container

export REDASH_DATABASE_URL="postgresql://postgres@postgres/tests"

change direcotry to extensio code

cd /extension

run the tests with whatever parameter

pytest -vvv --pdb

Issues & questions

See the issue tracker on GitHub to open tickets if you have issues or questions about Redash-STMO.