ElasticKernel

Never lose your Jupyter variables to a kernel restart again.

ElasticKernel is a custom IPython kernel that automatically checkpoints your notebook's execution state and restores it after a restart or crash — no manual pickle.dump required. Pick up exactly where you left off.

🇯🇵 日本語版は README.ja.md を参照してください。

Demo

The same workflow in both clips: define a variable a = 1, then restart the kernel. The difference is what happens next.

If the videos don't play inline, click them to view: ipykernel · ElasticKernel.

Why ElasticKernel?

Every Jupyter user has been there: a long computation finishes, then an accidental kernel restart (or an out-of-memory crash) wipes every variable in your session. The usual workaround is scattering pickle.dump / joblib.dump calls everywhere and remembering to reload them by hand.

ElasticKernel removes that chore entirely:

• 🔄 Automatic state recovery — your variables survive kernel restarts and shutdowns, with zero changes to your code.
• 🧠 Dependency-aware — tracks how cells and variables depend on one another to restore a consistent state.
• ⚡ Cost-optimized checkpoints — for each variable it decides whether to serialize it or recompute it on restore, based on serialization size vs. recomputation cost (a min-cut optimization).
• 🪄 Drop-in — just pick the Python 3 (ElasticKernel) kernel; the rest of your workflow is unchanged.

Installation & Usage

Local

1. Install the package:

   $ pip install elastic-kernel
   

2. Install the kernel:

   $ elastic-kernel install
   Elastic Kernel installed from: /path/to/elastic_kernel
   

3. Verify the kernel is installed:

   $ jupyter kernelspec list
   Available kernels:
     elastic_kernel    /path/to/Jupyter/kernels/elastic_kernel
   

4. Launch JupyterLab:

   $ jupyter lab --ip=0.0.0.0
   

5. Open JupyterLab in your browser.

6. Select the Python 3 (ElasticKernel) kernel.

Docker

1. Pull the image:

   docker pull ghcr.io/mryutaro/elastickernel
   

2. Start a container:

   docker run -p 8888:8888 ghcr.io/mryutaro/elastickernel
   

3. Open JupyterLab in your browser.

4. Select the Python 3 (ElasticKernel) kernel.

Supported versions

ElasticKernel is tested on every push and pull request against a CI matrix of Python versions. The checkpoint save → restore round trip (record_event → checkpoint → load_checkpoint) is verified on each of them.

Python 3.8 reached end-of-life in October 2024 and is no longer tested. The version matrix lives in .github/workflows/test.yml.

Supported libraries

ElasticKernel can restore a checkpointed object two ways, and a cost optimizer (min-cut) picks one per object at checkpoint time — neither is merely a fallback:

• Migrate — serialize the object with dill and reload it.
• Recompute — re-run the cell that produced it.

CI verifies several representative object types per library below — a library exposes many types, so this is a sample rather than a whole-library guarantee — and reports whether each restore path reproduces each one.

Legend:

• ✅ — restored correctly via this path.
• ➖ — not serializable, so the Migrate path does not apply (ElasticKernel uses Recompute instead).
• ❌ — failed: the object could not be restored, or it is a known limitation.

The table is generated by scripts/library_coverage.py and kept in sync by the library-coverage workflow.

How It Works

ElasticKernel extends the IPython kernel to observe each cell execution. As you run cells it builds a dependency graph of variables and the cell executions that produce them. When the kernel shuts down or restarts, it profiles serialization speed, runs a cost optimizer to split variables into a migrate set (serialized to disk) and a recompute set (regenerated by re-running cells), and writes a checkpoint. On the next start it loads the checkpoint, injects the migrated variables back into your namespace, and recomputes the rest.

Documentation

• On-demand checkpoint/restore API: docs/CHECKPOINT_API.md — trigger checkpoint/restore at any time from an external orchestrator (REST API)
• Developer guide: docs/DEVELOPERS.md
• 日本語 README: README.ja.md

Publication

This project was presented in the following paper. If you use ElasticKernel in your research, please cite:

R. Matsumoto, K. Taniguchi, T. Hayami, K. Takahashi, and S. Date. "ElasticHub: A Cost-Efficient JupyterHub Platform via Automated Scaling with Kubernetes on Hybrid Cloud." Proceedings of the 16th International Conference on Cloud Computing and Services Science, pp. 261–268, 2026. DOI: 10.5220/0014840200004039

@inproceedings{matsumoto2026elastichub,
  author    = {Matsumoto, R. and Taniguchi, K. and Hayami, T. and Takahashi, K. and Date, S.},
  title     = {ElasticHub: A Cost-Efficient JupyterHub Platform via Automated Scaling with Kubernetes on Hybrid Cloud},
  booktitle = {Proceedings of the 16th International Conference on Cloud Computing and Services Science},
  year      = {2026},
  pages     = {261--268},
  isbn      = {978-989-758-829-7},
  issn      = {2184-5042},
  doi       = {10.5220/0014840200004039}
}

Acknowledgments

This project includes code from ElasticNotebook, developed at the University of Illinois. ElasticNotebook is licensed under the Apache License 2.0.

Zhaoheng Li, Pranav Gor, Rahul Prabhu, Hui Yu, Yuzhou Mao, Yongjoo Park. "ElasticNotebook: Enabling Live Migration for Computational Notebooks." Proceedings of the VLDB Endowment, Vol. 17, No. 2, pp. 119-133, 2023.

License

Licensed under the Apache License 2.0.