Jupyter Storage Explorer

A file explorer for cloud storage and data lakes, right inside your Jupyter notebook, in the spirit of Azure Storage Explorer.

Browse Amazon S3, Azure Data Lake / Blob Storage, Google Cloud Storage, local disks and more; preview Parquet files, Delta Lake tables, CSV, Excel, JSON and other formats; page through, sort and filter tables; get ready-to-run code that loads the data into your notebook; and upload, download, move and edit files without leaving JupyterLab.

Two widgets:

• StorageExplorer: the file explorer, with previews and the data browser.
• DataExplorer: the data browser alone, for one table.

Installation

pip install jupyter-storage-explorer

Install the package for your storage too:

Previewing and querying data files needs a data library, e.g. pip install polars deltalake. If one is missing, the preview tells you what to install.

Quick start

from jupyter_storage_explorer import StorageExplorer

StorageExplorer("s3://my-bucket/data")

The explorer appears as the cell's output. Other examples:

StorageExplorer()  # the current folder
StorageExplorer("abfs://lake@myaccount.dfs.core.windows.net/raw")  # Azure Data Lake
StorageExplorer("s3://my-bucket/data", profile="analytics")  # Amazon S3, with options
StorageExplorer("gs://my-bucket", token="google_default")  # Google Cloud

Options after the address (like profile above) are passed to the storage package, for example to choose credentials.

To open one table directly, see DataExplorer. The notebooks folder has a demo of each widget (storage_explorer.ipynb, data_explorer.ipynb) on sample data made by make_samples.py.

From the JupyterLab launcher

The separate package jupyter-storage-explorer-launcher adds Storage Explorer to JupyterLab's launcher and command palette, opening the explorer in its own tab on the launcher's folder (the one the file browser showed):

pip install jupyter-storage-explorer-launcher

What the tab shows can be configured with a launcher.py file; see the launcher's README.

Several storages in one explorer

Give each storage a name; a drop-down on the left of the address bar switches between them, and each one remembers the folder you were in.

StorageExplorer(
    filesystems={
        "Data lake": "abfs://lake@myaccount.dfs.core.windows.net",
        "Raw files (S3)": "s3://my-bucket/raw",
        "Local": "/home/me/data",
    },
)

Using the explorer

• Browse: click folders in the tree or list; ↑ goes up, ⟳ refreshes. Ctrl+click (⌘+click) selects several items. The list shows 100 items per page (◀ ▶ above it).
• Filter: type in the fields above the tree and the list (and in Select columns of the data browser) to show only names containing the text, or matching a regular expression with .*. Case is ignored; ✕ or Esc clears.
• Preview: double-click a file, or Preview (with nothing selected: the current folder, e.g. a Delta Lake table). Tables open in the data browser.
• Right-click: Preview as, Edit, Python code to load the data, Copy, Paste, Delete, Create a folder, Download to local or to JupyterLab (see Options), Copy URL.
• Drag and drop: drop files from JupyterLab's file browser or your computer to upload them; drag files onto a folder to move them, or onto JupyterLab's file browser to download them (needs the launcher).

Unsafe operations

Folder operations need Unsafe mode, turned on with the slider in the status bar:

Query data

With polars installed (plus deltalake for Delta Lake, fastexcel for Excel), tables open in a data browser that reads 1,000 rows at a time:

• ◀ ▶ to page, click a column name to sort, its funnel to filter (Null / empty, Range, Contains, Regex).
• Select columns to hide columns, drag a header's edge to resize, Reset to clear filters, ⟳ to reload.

Filters, sorting, hidden columns and widths are remembered per table until the kernel restarts, or in a file with query_settings=QuerySettingsStore("query-settings.json") (from jupyter_storage_explorer.query). polars reads the data with the credentials in the environment, not the explorer's options.

Supported formats

The name identifies the format in DataExplorer(..., format=...) and in CSS (see Customizing icons).

Format options

format_options is one dictionary for all formats; each format uses the options it takes. Options are named after the library they are for, and the ones for a single format also after the format:

StorageExplorer(
    "s3://my-bucket/lake",
    format_options={
        "polars_storage_options": {"aws_region": "eu-west-1"},
        "pandas_storage_options": {"anon": False},
        "polars_delta_use_pyarrow": True,
    },
)

Storage options that look like credentials are left out of the code. Without storage options, the code relies on the environment's configuration and credentials.

Working with the explorer from Python

explorer = StorageExplorer("s3://my-bucket/data")
explorer

explorer.folder_url  # the URL of the folder shown, e.g. to pass to polars or pandas
explorer.selected_urls  # the URLs of the selected items

StorageExplorer options

DataExplorer: one table

DataExplorer opens the same data browser for a single table, without the file explorer around it:

from jupyter_storage_explorer import DataExplorer

DataExplorer(
    "s3://my-bucket/lake/sales"
)  # a Delta Lake table, Parquet, CSV, JSON Lines or Excel
DataExplorer("data/orders.tsv", format="csv")  # choose the format by name
DataExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw/orders.parquet", height="600px"
)

It remembers its settings like StorageExplorer's data browser, and shares them with it. If polars (or e.g. deltalake) is missing, it raises an error with the %pip install command.

DataExplorer options

Requirements

• Python 3.11 or later
• JupyterLab 4. The widgets also show in other Jupyter frontends, but dragging files from JupyterLab's file browser works only in JupyterLab.
• For Query data and DataExplorer: a recent polars (tested with polars 2.0).

Development

Building from source, the JupyterLab extension, releasing, and customizing the explorer's icons with CSS: see DEVELOPMENT.md.