Jupyter Storage Explorer

Explore your data lake without leaving Jupyter: built for data professionals

StorageExplorer is a file explorer for cloud storage and data lakes, right inside your Jupyter notebook, in the spirit of Azure Storage Explorer.

Browse Amazon S3, Azure Data Lake / Blob Storage, Google Cloud Storage, local disks and more; preview Parquet files, Delta Lake tables, CSV, Excel, JSON and other formats; page through, sort and filter tables; get ready-to-run code that loads the data into your notebook; and upload, download, move and edit files without leaving JupyterLab. See Using the explorer for all you can do with it.

This package also provides the DataExplorer widget, which lets you browse tabular data in your notebook: any dataframe of pandas, polars, Spark, DuckDB and more. Page through, sort and filter it in the widget, then get the result back in code. See Using the data browser for all you can do with it.

Highlights

• StorageExplorer: any fsspec storage, several in one explorer; copy and paste between storages and notebooks.
• DataExplorer: page, sort and filter any dataframe; get the result back in code.
• Previews of Delta Lake, Parquet, CSV, JSON, Excel, text and images, with code to load them.
• Background file operations with progress and Cancel; folder operations only in Unsafe mode.
• Work in JupyterLab: open storage files in JupyterLab's own editors and viewers, and save them back; query tables in tabs of their own.
• Bookmarks, remembered queries, transfer rules, and a JupyterLab launcher item.

Installation

pip install jupyter-storage-explorer

Install the package for your storage too:

Previewing and querying data files needs a data library, e.g. pip install polars deltalake; the preview tells you which.

Quick start

Try it without installing anything: open the demo on Binder, a made-up data lake in a notebook and in the launcher's tab (it may take a minute to start).

From the JupyterLab launcher

jupyter-storage-explorer-launcher adds Storage Explorer to JupyterLab's launcher, opening the explorer in its own tab:

pip install jupyter-storage-explorer-launcher

From Python code

from jupyter_storage_explorer import StorageExplorer

StorageExplorer("s3://my-bucket/data")

Options after the address go to the storage package, e.g. profile="analytics". Double-click a file to preview it; right-click for everything else.

Browse a dataframe and get the rows you filtered back:

import polars as pl
from jupyter_storage_explorer import DataExplorer

explorer = DataExplorer(pl.scan_parquet("s3://my-bucket/lake/sales/*.parquet"))
explorer

explorer.result().collect()  # after filtering and sorting in the widget

The notebooks folder has a demo of each widget on sample data.

Documentation

• StorageExplorer: storages, using the explorer, transfer policy, formats and options.
• DataExplorer: the data browser, result(), on_change(), QuerySettings and options.
• The launcher: configuring its tab with launcher.py files.
• Development: building, releasing, and custom icons.

Requirements

• Python 3.11 or later
• JupyterLab 4 (other Jupyter frontends show the widgets, without drag and drop from JupyterLab's file browser)
• For querying data: polars (tested with 2.0)