Jupyter Storage Explorer

A file explorer for cloud storage and data lakes, right inside your Jupyter notebook, in the spirit of Azure Storage Explorer.

Browse Amazon S3, Azure Data Lake / Blob Storage, Google Cloud Storage, local disks and more; preview Parquet files, Delta Lake tables, CSV, Excel, JSON and other formats; page through, sort and filter tables; get ready-to-run code that loads the data into your notebook; and upload, download, move and edit files without leaving JupyterLab.

Two widgets:

• StorageExplorer: the file explorer, with previews and the data browser.
• DataExplorer: the data browser alone, for one table.

Installation

pip install jupyter-storage-explorer

Install the package for your storage too:

Storage	Package	Address
Amazon S3	s3fs	s3://bucket/path
Azure Data Lake / Blob Storage	adlfs	abfs://container@account.dfs.core.windows.net/path
Google Cloud Storage	gcsfs	gs://bucket/path
Local disk, in-memory	(included)	/path, memory://

Previewing and querying data files needs a data library, e.g. pip install polars deltalake. If one is missing, the preview tells you what to install.

Quick start

from jupyter_storage_explorer import StorageExplorer

StorageExplorer("s3://my-bucket/data")

The explorer appears as the cell's output. Other examples:

StorageExplorer()  # the current folder
StorageExplorer("abfs://lake@myaccount.dfs.core.windows.net/raw")  # Azure Data Lake
StorageExplorer("s3://my-bucket/data", profile="analytics")  # Amazon S3, with options
StorageExplorer("gs://my-bucket", token="google_default")  # Google Cloud

Options after the address (like profile above) are passed to the storage package, for example to choose credentials.

To open one table directly, see DataExplorer. The notebooks folder has a demo of each widget (storage_explorer.ipynb, data_explorer.ipynb) on sample data made by make_samples.py.

From the JupyterLab launcher

The separate package jupyter-storage-explorer-launcher adds Storage Explorer to JupyterLab's launcher and command palette, opening the explorer in its own tab on the launcher's folder (the one the file browser showed):

pip install jupyter-storage-explorer-launcher

What the tab shows can be configured with a launcher.py file; see the launcher's README.

Several storages in one explorer

Give each storage a name; a drop-down on the left of the address bar switches between them, and each one remembers the folder you were in.

StorageExplorer(
    filesystems={
        "Data lake": "abfs://lake@myaccount.dfs.core.windows.net",
        "Raw files (S3)": "s3://my-bucket/raw",
        "Local": "/home/me/data",
    },
)

Using the explorer

• Browse: click folders in the tree, double-click them in the list, or click the address bar; ↑ goes up, ⟳ refreshes. Ctrl+click (⌘+click) selects several items.
• Preview: double-click a file, or use Preview (with nothing selected, it previews the current folder, e.g. a Delta Lake table). Tables open in the data browser instead (see Query data). Esc closes the panel.
• Right-click: Preview as, Open, Edit (text files), Python code to load the data (polars, pandas, PySpark, …), Download, Copy URL.
• Drag and drop: drop files from JupyterLab's file browser or your computer to upload them; drag files onto a folder to move them. Both ask for confirmation.

Query data

With polars installed (plus deltalake for Delta Lake, fastexcel for Excel), tables open in a data browser that reads 1,000 rows at a time:

• ◀ ▶ to page, click a column name to sort, its funnel to filter (Null / empty, Range, Contains, Regex).
• Select columns to hide columns, drag a header's edge to resize, Reset to clear filters, ⟳ to reload.

Filters, sorting, hidden columns and widths are remembered per table until the kernel restarts, or in a file with query_settings=QuerySettingsStore("query-settings.json") (from jupyter_storage_explorer.query). polars reads the data with the credentials in the environment, not the explorer's options.

DataExplorer: one table

DataExplorer opens the same data browser for a single table, without the file explorer around it:

from jupyter_storage_explorer import DataExplorer

DataExplorer(
    "s3://my-bucket/lake/sales"
)  # a Delta Lake table, Parquet, CSV, JSON Lines or Excel
DataExplorer("data/orders.tsv", format="csv")  # choose the format by name
DataExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw/orders.parquet", height="600px"
)

It takes the same storage options, format_options= (see Format options) and query_settings= as StorageExplorer, and shares its remembered settings. If polars (or e.g. deltalake) is missing, it raises an error with the %pip install command.

Supported formats

Format	Name	Preview	Query data	Load with
Delta Lake tables	delta	First rows	✓	polars (with deltalake), PySpark (with delta-spark)
Parquet files, and folders of them (e.g. Spark, Hive output)	parquet	First rows	✓	polars, pandas, PySpark
CSV, TSV	csv	First rows	✓	polars, pandas, PySpark
JSON Lines	json-lines	First records	✓	polars, pandas, PySpark
JSON	json	Collapsible tree		Python, polars, pandas
Excel (.xlsx)	excel	First rows of the first sheet	✓	polars, pandas
Markdown	markdown	Rendered		Python
Text, code, configuration (YAML, SQL, Python, …)	text	With syntax highlighting		Python
Images (PNG, JPEG, GIF, SVG, …)	image	Shown		Pillow

The name identifies the format in DataExplorer(..., format=...) and in CSS (see Customizing icons).

Format options

format_options is one dictionary for all formats; each format uses the options it takes. Options are named after the library they are for, and the ones for a single format also after the format:

StorageExplorer(
    "s3://my-bucket/lake",
    format_options={
        "polars_storage_options": {"aws_region": "eu-west-1"},
        "pandas_storage_options": {"anon": False},
        "polars_delta_use_pyarrow": True,
    },
)

Option	Formats	Meaning
polars_storage_options	delta, parquet, csv, json-lines	polars' storage_options (object_store's names, not fsspec's), in the preview, the data browser and the polars code.
polars_credential_provider	delta, parquet, csv, json-lines	polars' credential_provider, in the preview and the data browser (not in the code).
pandas_storage_options	parquet, csv, json-lines, json, excel	pandas' storage_options (fsspec's), in the pandas code.
fsspec_storage_options	json, excel, markdown, text, image	fsspec.open's keyword arguments, in the code that reads with fsspec.
polars_delta_table_options	delta	scan_delta's delta_table_options.
polars_delta_use_pyarrow	delta	scan_delta's use_pyarrow.
polars_delta_pyarrow_options	delta	scan_delta's pyarrow_options.

Storage options that look like credentials are left out of the code. Without storage options, the code relies on the environment's configuration and credentials.

Working with the explorer from Python

explorer = StorageExplorer("s3://my-bucket/data")
explorer

explorer.folder_url  # the URL of the folder shown, e.g. to pass to polars or pandas
explorer.selected_urls  # the URLs of the selected items

Options

Option	Meaning
height="400px"	Height of the tree and list; the preview panel adds its own space below. "fill": as tall as the container, e.g. the launcher's tab.
max_upload_bytes=10 * 1024 * 1024	Largest file you can upload from your computer (default 10 MB; None for no limit).
filesystems={...}	Several named storages, as shown above.
query_settings=...	Where the data browser keeps its settings; see Query data.
format_options={...}	Options for the formats (see Format options). Also for DataExplorer.

Requirements

• Python 3.11 or later
• JupyterLab 4. The widgets also show in other Jupyter frontends, but dragging files from JupyterLab's file browser works only in JupyterLab.
• For Query data and DataExplorer: a recent polars (tested with polars 2.0).

Development

Building from source, the JupyterLab extension, releasing, and customizing the explorer's icons with CSS: see DEVELOPMENT.md.