streaming-jupyter-integrations 0.5.0

JupyterNotebook Flink magics

Streaming Jupyter Integrations

Streaming Jupyter Integrations project includes a set of magics for interactively running Flink SQL jobs in Jupyter Notebooks

In order to actually use these magics, you must install our PIP package along jupyterlab-lsp:

python3 -m pip install jupyterlab-lsp streaming-jupyter-integrations

And then register in Jupyter with a running IPython in the first cell:

%load_ext streaming_jupyter_integrations.magics

Variables

Magics allow for dynamic variable substitution in Flink SQL cells.

my_variable = 1

SELECT * FROM some_table WHERE product_id = {my_variable}

Moreover, you can mark sensitive variables like password so they will be read from user input every time one runs the cell:

CREATE TABLE MyUserTable (
  id BIGINT,
  name STRING,
  age INT,
  status BOOLEAN,
  PRIMARY KEY (id) NOT ENFORCED
) WITH (
   'connector' = 'jdbc',
   'url' = 'jdbc:mysql://localhost:3306/mydatabase',
   'table-name' = 'users',
   'username' = '${my_username}',
   'password' = '${my_password}'
);

Local development

Note: You will need NodeJS to build the extension package.

The jlpm command is JupyterLab's pinned version of yarn that is installed with JupyterLab. You may use yarn or npm in lieu of jlpm below. In order to use jlpm, you have to have jupyterlab installed (e.g., by brew install jupyterlab, if you use Homebrew as your package manager).

# Clone the repo to your local environment
# Change directory to the flink_sql_lsp_extension directory
# Install package in development mode
pip install -e .
# Link your development version of the extension with JupyterLab
jupyter labextension develop . --overwrite
# Rebuild extension Typescript source after making changes
jlpm build

The project uses pre-commit hooks to ensure code quality, mostly by linting. To use it, install pre-commit and then run

pre-commit install --install-hooks

From that moment, it will lint the files you have modified on every commit attempt.