jupylite-duckdb 0.0.18a3

Testing

Experimental

This is highly experimental and will change frequently.

There are three ways to demo this:

• JupyerLite: Open a JupyterLite site, and use the examples from =notebooks
• JupyterLite Code Console REPL
• Pyodide Console

Demonstration

Code Console REPL Example

Code Console REPL

jupyterlite_duckdb_wasm

Python wrapper to run DuckDB_WASM within JupyterLite with a Pyodide Kernel

See notebooks for example of running this within jupyterlite

Cell Magic %%dql

Following the example of magic_duckdb, there's an initial proof of concept for a duckdb for JupyterLite. See Magic Example

Pyodide Console

pyodide console

import micropip;
await micropip.install('pandas');
await micropip.install('jupylite-duckdb');
import jupylite_duckdb as jd;
conn = await jd.connect();
r1 = await jd.query("pragma version", conn);
r2 = await jd.query("create or replace table xyz as select * from 'https://raw.githubusercontent.com/Teradata/kylo/master/samples/sample-data/parquet/userdata2.parquet'", conn);
r3 = await jd.query("select gender, count(*) as c from xyz group by gender", conn);
print(r1);
print(r2);
print(r3);

To Do

• Embed POC in a JupyterLite Code Console REPL
• Implement a proof of concept version of dataframe registration
• Reduce startup time, probably a combination of the jupyterlite config (preloading modules) and wasm
• Handling errors: detect and display errors in Jupyter: too much sfuff buried in console, such as CORS errors
• invalidate pip browser cache (as/if needed); annoying for development purposes
• think through async/await/transform_cell approach and whether there's a better solution.

Some development notes

• Zero copy data exchange (js/duckdb arrow -> python/dataframe and python/df -> js/duckdb): Blocked by Pyarrow support
• If you're adding local .py files, use importlib.invalidate_caches(). Even then, it was flaky to import.
• Careful with caching... %pip install will pull from browser cache. I had to clear frequently within dev tools
• To clear local storage, which is annoyingly persistent, https://superuser.com/questions/519628/clear-html5-local-storage-on-a-specific-page
• %autoawait is part of why this works in notebooks, which is enabled by default. The %%dql cell magic patches transform-cell to push an await into the cell transformation.: https://ipython.readthedocs.io/en/stable/interactive/autoawait.html