HeaderGen: Automated cell header generator
Project description
HeaderGen
HeaderGen is a tool-based approach to enhance the comprehension and navigation of undocumented Python based Jupyter notebooks by automatically creating a narrative structure in the notebook.
Data scientists build an ML-based solution notebook by first preparing the data, then extracting key features, and then creating and training the model. HeaderGen leverages the implicit narrative structure of an ML notebook to add structural headers as annotations to the notebook.
Features
-
Automated Markdown Header Insertion: Through a taxonomy for machine-learning operations, HeaderGen annotates code cells with relevant markdown headers.
-
Function Call Taxonomy: Methodically classifies function calls based on a machine-learning operations taxonomy.
-
Advanced Call Graph Analysis: Enhances PyCG framework with flow-sensitivity and external library return-type resolution.
-
Precision in External Libraries: capability to accurately resolve function return types from external libraries using typestubs.
-
Syntax Pattern Matching: Employs type data for pattern matching.
Folder Structure
callsites-jupyternb-micro-benchmark
: Micro benchmarkcallsites-jupyternb-real-world-benchmark
: Real-world benchmarkevaluation
: Contains manual header annotation and user study resultsframework_models
: Function calls to ML Taxonomy mappingtypestub-database
: Type-stbs for ML librariesheadergen
: Source code of HeaderGenpycg_extended
: Source code of extended PyCGheadergen-extension
: Jupyter notebook plugin for HGheadergen_output
: Folder where the generated notebooks from the docker container are stored
1. Build container
-
Get source files
git clone --recursive git submodule update --init --recursive git pull --recurse-submodules
-
Linux
docker build -t headergen . docker run -v {$PWD}/headergen_output:/results -it headergen bash
-
Windows
docker build -t headergen . docker run -v "%cd%"/headergen_output:/results -it headergen bash
2. Run HeaderGen benchmarks from inside contatiner
Output generated from the following commands, such as annotated notebooks, reports, callsites, headers, etc, are stored in the local folder headergen_output
after the following commands are done executing.
-
Micro Benchmark (generates a csv file with results)
make microbench
-
Real-world Benchmark (generates annotated notebooks and csv file that reproduce table 2)
make realworldbench
-
Both Benchmarks
make all
-
Clean generated output
make clean
This repo contains code for the paper "Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis" published at the SANER Conference 2023.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for headergen-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06fc012d079e56650fea720e8680fd54eb6a984ce352324ad37003ec8d29a1db |
|
MD5 | c72a7f6e5040e5c000ab0b29739fee96 |
|
BLAKE2b-256 | 74382a6c2520420998b21e1eed14218a85cb6af5fa2c283e34fd135781f1d6bf |