kedro-neptune 0.1.0

Neptune.ai integration with Kedro

Neptune + Kedro Integration

Kedro plugin for experiment tracking and metadata management. It lets you browse, filter and sort runs in a nice UI, visualize node/pipeline metadata, and compare pipelines.

What will you get with this integration?

• browse, filter, and sort your model training runs
• compare nodes and pipelines on metrics, visual node outputs, and more
• display all pipeline metadata including learning curves for metrics, plots, and images, rich media like video and audio or interactive visualizations from Plotly, Altair, or Bokeh
• and do whatever else you would expect from a modern ML metadata store

Kedro pipeline metadata in custom dashboard in the Neptune UI

Note: Kedro-Neptune plugin supports distributed pipeline execution and works in Kedro setups that use orchestrators like Airflow or Kubeflow.

Resources

• Documentation
• Code example on GitHub
• Runs logged in the Neptune app
• How to Compare Kedro pipelines
• How to Compare results between Kedro nodes
• How to Display Kedro node metadata and outputs

Installation

Before you start, make sure that:

• You have Python 3.7+ in your system,
• You are already a registered user so that you can log metadata to your private projects.
• You have your Neptune API token set to the NEPTUNE_API_TOKENenvironment variable.

Install neptune-client, kedro, and kedro-neptune

Depending on your operating system open a terminal or CMD and run this command. All required libraries are available via pip and conda:

pip install neptune-client kedro kedro-neptune

For more, see installing neptune-client.

Quickstart

This quickstart will show you how to:

• Connect Neptune to your Kedro project
• Log pipeline and dataset metadata to Neptune
• Add explicit metadata logging to a node in your pipeline
• Explore logged metadata in the Neptune UI.

Before you start

• Have Kedro installed
• Have neptune-client and kedro-neptune plugin installed

Step 1: Create a Kedro project from "pandas-iris" starter

• Go to your console and create a Kedro starter project "pandas-iris"

kedro new --starter=pandas-iris

• Follow instructions and choose a name for your Kedro project. For example, "Great-Kedro-Project"
• Go to your new Kedro project directory

If everything was set up correctly you should see the following directory structure:

Great-Kedro-Project # Parent directory of the template
├── conf            # Project configuration files
├── data            # Local project data (not committed to version control)
├── docs            # Project documentation
├── logs            # Project output logs (not committed to version control)
├── notebooks       # Project related Jupyter notebooks (can be used for experimental code before moving the code to src)
├── README.md       # Project README
├── setup.cfg       # Configuration options for `pytest` when doing `kedro test` and for the `isort` utility when doing `kedro lint`
├── src             # Project source code
    ├── great_kedro_project   
        ├── pipelines   
            ├── data_science
                ├── nodes.py
                ├── pipelines.py
                └── ...

You will use nodes.py and pipelines.py files in this quickstart.

Step 2: Initialize kedro-neptune plugin

• Go to your Kedro project directory and run

kedro neptune init

The command line will ask for your Neptune API token

• Input your Neptune API token:
  • Press enter if it was set to the NEPTUNE_API_TOKEN environment variable
  • Pass a different environment variable to which you set your Neptune API token. For example MY_SPECIAL_NEPTUNE_TOKEN_VARIABLE
  • Pass your Neptune API token as a string

The command line will ask for your Neptune project name

• Input your Neptune project name:
  • Press enter if it was set to the NEPTUNE_PROJECT environment variable
  • Pass a different environment variable to which you set your Neptune project name. For example MY_SPECIAL_NEPTUNE_PROJECT_VARIABLE
  • Pass your project name as a string in a format WORKSPACE/PROJECT

If everything was set up correctly you should:

• see the message: "kedro-neptune plugin successfully configured"
• see three new files in your kedro project:
  • Credentials file:YOUR_KEDRO_PROJECT/conf/local/credentials_neptune.yml
  • Config file:YOUR_KEDRO_PROJECT/conf/base/neptune.yml
  • Catalog file:YOUR_KEDRO_PROJECT/conf/base/neptune_catalog.yml

You can always go to those files and change the initial configuration.

Step 3: Add Neptune logging to a Kedro node

• Go to a pipeline node src/KEDRO_PROJECT/pipelines/data_science/nodes.py
• Import Neptune client toward the top of the nodes.py

import neptune.new as neptune

• Add neptune_run argument of type neptune.handler.Handler to the report_accuracy function

def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
                    neptune_run: neptune.handler.Handler) -> None:
...

You can treat neptune_run like a normal Neptune Run and log any ML metadata to it.

Important
You have to use a special string "neptune_run" to use the Neptune Run handler in Kedro pipelines.

• Log metrics like accuracy to neptune_run

def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
                    neptune_run: neptune.handler.Handler) -> None:
    target = np.argmax(test_y.to_numpy(), axis=1)
    accuracy = np.sum(predictions == target) / target.shape[0]
    
    neptune_run['nodes/report/accuracy'] = accuracy * 100

You can log metadata from any node to any Neptune namespace you want.

• Log images like a confusion matrix to neptune_run

def report_accuracy(predictions: np.ndarray, test_y: pd.DataFrame, 
                    neptune_run: neptune.handler.Handler) -> None:
    target = np.argmax(test_y.to_numpy(), axis=1)
    accuracy = np.sum(predictions == target) / target.shape[0]
    
    fig, ax = plt.subplots()
    plot_confusion_matrix(target, predictions, ax=ax)
    neptune_run['nodes/report/confusion_matrix'].upload(fig)

Note
You can log metrics, text, images, video, interactive visualizations, and more.
See a full list of What you can log and display in Neptune.

Step 4: Add Neptune Run handler to the Kedro pipeline

• Go to a pipeline definition, src/KEDRO_PROJECT/pipelines/data_science/pipelines.py
• Add neptune_run Run handler as an input to the report node

node(
    report_accuracy,
    ["example_predictions", "example_test_y", "neptune_run"],
    None,
    name="report"),

Step 5: Run Kedro pipeline

Go to your console and execute your Kedro pipeline

kedro run

A link to the Neptune Run associated with the Kedro pipeline execution will be printed to the console.

Step 6: Explore results in the Neptune UI

• Click on the Neptune Run link in your console or use an example link

https://app.neptune.ai/common/kedro-integration/e/KED-632

• Go to the kedro namespace where metadata about Kedro pipelines are logged (see how to change the default logging location)

• See pipeline and node parameters in kedro/catalog/parameters

• See execution parameters in kedro/run_params

• See metadata about the datasets in kedro/catalog/datasets/example_iris_data

• See the metrics (accuracy) you logged explicitly in the kedro/nodes/report/accuracy

• See charts (confusion matrix) you logged explicitly in the kedro/nodes/report/confusion_matrix

Support

If you got stuck or simply want to talk to us, here are your options:

• Check our FAQ page
• You can submit bug reports, feature requests, or contributions directly to the repository.
• Chat! When in the Neptune application click on the blue message icon in the bottom-right corner and send a message. A real person will talk to you ASAP (typically very ASAP),
• You can just shoot us an email at [e-mail]