litrepl 3.1.0

LitREPL is a command-line tool and a Vim plugin for code snippet execution.

LitREPL

LitREPL is a command-line text processing tool for code evaluation aimed at enabling literate programming experience in common editors.

Features

• Document formats
  Markdown (Example [MD]) | LaTeX (Examples [TEX][PDF])
• Interpreters
  Python | IPython | GPT4All-cli
• Editor integration
  Vim (plugin included)

Requirements

• POSIX-compatible OS, typically a Linux. The tool relies on POSIX pipes and depends on certain shell commands.
• More or less recent Vim
• Python3 packages: lark-parser, psutil (Required).
• Command line tools: GNU socat (Optional)

Contents

• Installation
• Usage
  • Quick start
    • Python
    • AI
  • Reference
    • Vim and command-line commands
    • Variables and arguments
  • Hints
    • Command line, basic usage
    • Command line, foreground evaluation
    • Vim, adding keybindings
    • Vim, inserting new sections
    • Vim, executing first section after restart
    • Vim, running shell commands
• Development
  • Development shells
  • Other Nix targets
  • Common workflows
• Gallery
• Technical details
• Limitations
• Related projects
• Third-party issues

Installation

This repository includes the Litrepl tool in Python and an interface Vim plugin. The Python part might be installed with pip install . run from the project folder. The Vim part requires hand-copying ./vim/plugin/litrepl.vim to a ~/.vim config folder or using any Vim plugin manager, e.g. Vim-Plug.

The repository also includes a set of Nix expressions that automate installation on Nix-enabled systems.

pip-install and Vim-Plug

Instructions for the Pip and Vim-plug:

1. Install the litrepl Python package with pip:

   $ pip install --user git+https://github.com/grwlf/litrepl.vim
   $ litrepl --version
   

2. Install the Vim plugin by adding the following line between the plug#begin and plug#end lines of your .vimrc file:

   Plug 'https://github.com/grwlf/litrepl.vim' , { 'rtp': 'vim' }
   

   Note: rtp sets the custom vim-plugin source directory of the plugin.

Nix and vim_configurable

Nix/NixOS users might follow the formalized path:

Nix supports configurable Vim expressions. To enable the Litrepl plugin, add the vim-litrepl.vim-litrepl-release to the list of Vim plugins and put this version of vim into your Nix profile. Litrepl and its dependencies will be installed automatically.

{ litrepl }:
...
vim_configurable.customize {
  name = "vim";
  vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; {
    start = [
      ...
      litrepl.vim-litrepl-release
      ...
    ];
  };
}

Note: vim-demo expression from the default.nix provides an example Vim configuration. Use nix build '.#vim-demo' to build it and then ./result/bin/vim-demo to run the editor.

See the Development section for more details.

Usage

Quick start

Python

Create separate python and result sections within your Markdown or LaTeX document. Insert Python code into the 'python' section, then use the :LEval command to execute that section at the cursor location.

Markdown                             LaTeX
--------                             -----

``` python                           \begin{python}
print('Hello Markdown!')             print('Hello LaTeX!')
```                                  \end{python}


``` result                           \begin{result}
Hello Markdown!                      Hello LaTeX!
```                                  \end{result}

Note: LaTeX documents need a preamble introducing python/result tags to the text processor. For details, see:

• Formatting Markdown documents
• Formatting LaTeX documents

Litrepl leaves the Python interpreter running in the background, its input and output pipes are associated with the folder where the document resides. We can use :LStatus in vim or litrepl status command line to know its status:

$ litrepl status
# Format:
# TYP  PID      ECODE  CMD
python 3718077  -      python3 -m IPython --config=/tmp/nix-shell.KcUxp9/litrepl_1000_a2732d/python/litrepl_ipython_config.py --colors=NoColor -i

In order to attach to the running interpreter and examine its state, one might use repl command. Note: Litrepl disables command line propmts so >>> symbols are not going to appear. Here we assign a Python variable and then detach.

$ litrepl repl
Opening the interpreter terminal (NO PROMPT, USE `Ctrl+D` TO DETACH)
W='Hello from repl'
^D

The variable W now resides in the session and could be queried during subsequent evaluations.

AI

Litrepl experimentally supports GPT4All-cli allowing users to query local LLMs. In order to try it, install the interpreter and use ai as the name for code sections. For low-speed models it would be convenient to use :LEvalMon command rather then :LEval for evaluation.

``` ai
/model "~/.local/share/nomic.ai/GPT4All/Meta-Llama-3-8B-Instruct.Q4_0.gguf"
Hi chat! What is your name?
```

``` result
I'm LLaMA, a large language model trained by Meta AI. I'm here to help answer
any questions you might have and provide information on a wide range of topics.
How can I assist you today?
```

Reference

Vim and command-line commands

Vim	Command line	Description
:LStart [T]	litrepl start [T]	Start the interpreter
:LStop [T]	litrepl stop [T]	Stop the interpreter
:LStatus [T]	litrepl status [T] <F	Print the daemon status
:LRestart [T]	litrepl restart [T]	Restart the interpreter
:LEval N	lirtepl eval-sections N <F	Run or update section under the cursor and wait until the completion
:LEvalAbove N	lirtepl eval-sections '0..N' <F	Run sections above and under the cursor and wait until the completion
:LEvalBelow N	lirtepl eval-sections 'N..$' <F	Run sections below and under the cursor and wait until the completion
:LEvalAll	lirtepl eval-sections <F	Evaluate all code sections
:LEvalAsync N	lirtepl --timeout=0.5,0 eval-sections N <F	Run section under the cursor and wait a bit before going asynchronous. Also, update the output from the already running section.
:LInterrupt N	lirtepl interrupt N <F	Send Ctrl+C signal to the interpreter and get a feedback
:LEvalMon	while .. do .. done	Monitor asynchronous code evaluation
N/A	lirtepl eval-code <P	Evaluate the given Python code
:LTerm	lirtepl repl [T]	Open the terminal to the interpreter
:LOpenErr	litrepl ... 2>F	Open the stderr window
:LVersion	litrepl --version	Show version

Where

• T type of the interpreter: python or ai (some commands also accept all)
• F Path to a Markdown or LaTeX file
• P Path to a Python script
• N number of code section to evaluate, starting from 0.
• L:C denotes line:column of the cursor.

Variables and arguments

Vim setting	CLI argument	Description
set filetype	--filetype=D	Input file type: latex|markdown
let g:litrepl_python_interpreter=B	--python-interpreter=B	The Python interpreter to use: python|ipython|auto (the default)
let g:litrepl_ai_interpreter=B	--ai-interpreter=B	The AI interpreter to use: gpt4all-cli|auto (the default)
let g:litrepl_debug=0/1	--debug=0/1	Print debug messages to the stderr
let g:litrepl_timeout=FLOAT	--timeout=FLOAT	Timeout to wait for the new executions, in seconds, defaults to inf

• D type of the document: tex or markdown (the default).
• B interpreter binary to use, defaults to auto which guesses the best one.
• FLOAT should be formatted as 1 or 1.1 or inf. Note: command line argument also accepts a pair of timeouts.

More arguments are available, see help.

Hints

Command line, basic usage

To evaluate code section in a document:

$ cat doc/example.md | litrepl eval-sections >output.md

To evaluate a Python script:

$ cat script.py | litrepl eval-code

Note that both commands above share the same background interpreter session.

Command line, foreground evaluation

For batch processing of documents, it may be necessary to have an on-demand interpreter session available, which would exist solely for the duration of the evaluation process.

$ cat >document.md.in <<EOF
``` python
raise Exception("D'oh!")
```
EOF
$ cat document.md.in | litrepl --foreground --exception-exit=200 eval-sections >document.md
$ echo $?
200

Here, the --foreground argument tells Litrepl to run a new interpreter session and then stop it before exiting, --exception-exit=200 sets the exit code returned in the case of unhandled exceptions.

Vim, adding keybindings

The plugin does not define any keybindings, but users could do it by themselves, for example:

nnoremap <F5> :LEval<CR>
nnoremap <F6> :LEvalAsync<CR>

Vim, inserting new sections

Below we define :C command inserting new sections.

command! -buffer -nargs=0 C normal 0i``` python<CR>```<CR><CR>``` result<CR>```<Esc>4k

Vim, executing first section after restart

We define the :LR command running first section after the restart.

command! -nargs=0 LR LRestart | LEval 0

Vim, running shell commands

Thanks to IPython features, we can use exclamation to run shell commands directly from Python code sections.

``` python
!cowsay "Hello, Litrepl!"
```

``` result
 _________________
< Hello, Litrepl! >
 -----------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
```

Development

This project uses Nix as a primary development framework. flake.nix handles the source-level Nix dependencies while the default.nix defines the common build targets including Pypi and Vim packages, demo Vim configurations, development shells, etc.

Development shells

The default development shell is defined in the ./default.nix as a Nix expression named shell which is the default name for development shells. Running

$ nix develop

will ask Nix to install the development dependencies and open the shell.

Other Nix targets

Another shell which might be useful is shell-screencast. This would build the full set of Litrepl tools and makes sure that the screencasting software is available. To enter it, specify its Nix-flake path as follows:

$ nix develop '.#shell-screencast'

To build individual Nix expressions, run nix build '.#NAME' passing the name of Nix-expression to build. If succeeded, Nix publishes the last build' results under the ./result symlink.

$ nix build '.#vim-demo'
$ ./result/bin/vim-demo  # Run the pre-configured demo instance of Vim

The list of Nix build targets includes:

• litrepl-release - Litrepl script and Python lib
• litrepl-release-pypi - Litrepl script and Python lib
• vim-litrepl-release - Vim with locally built litrepl plugin
• vim-litrepl-release-pypi - Vim with litrepl plugin built from PYPI
• vim-test - A minimalistic Vim with a single litrepl plugin
• vim-demo - Vim configured to use litrepl suitable for recording screencasts
• vim-plug - Vim configured to use litrepl via the Plug manager
• shell-dev - The development shell
• shell-screencast - The shell for recording demonstrations, includes vim-demo.

See Nix flakes manual for other Nix-related details.

Common workflows

The top-level Makefile encodes common development workflows:

[LitREPL-develop] $ make help
LitREPL is a macroprocessing Python library for Litrate programming and code execution
Build targets:
help:       Print help
test:       Run the test script (./sh/test.sh)
wheel:      Build Python wheel (the DEFAULT target)
version:    Print the version
upload:     Upload wheel to Pypi.org (./_token.pypi is required)

Gallery

Basic usage

Using LitREPL in combination with the Vimtex plugin to edit Latex documents on the fly.

Asynchronous code execution

Technical details

The following events should normally happen after users type the :LitEval1 command:

1. On the first run, LitREPL starts the Python interpreter in the background. Its standard input and output are redirected into UNIX pipes in the current directory.
2. LitREPL runs the whole document through the express Markdown/Latex parser determining the start/stop positions of code and result sections. The cursor position is also available and the code from the right code section can reach the interpreter.
3. The process which reads the interpreter's response is forked out of the main LitREPL process. The output goes to the temporary file.
4. If the interpreter reports the completion quickly, the output is pasted to the resulting document immediately. Otherwise, the temporary results are pasted.
5. Re-evaluating sections with temporary results causes LitREPL to update these results.

Limitations

• Formatting: Nested code sections are not supported.
• Formatting: Special symbols in the Python output could invalidate the document.
• Interpreter: Extra newline is required after Python function definitions.
• Interpreter: Stdout and stderr are joined together.
• Interpreter: Evaluation of a code section locks the editor.
• Interpreter: Tweaking os.ps1/os.ps2 prompts of the Python interpreter could break the session.
• Interpreter: No asynchronous code execution.
• Interpreter: Background Python interpreter couldn't be interrupted

Related projects

Edititng:

• https://github.com/lervag/vimtex (LaTeX editing, LaTeX preview)
• https://github.com/shime/vim-livedown (Markdown preview)
• https://github.com/preservim/vim-markdown (Markdown editing)

Code execution:

• Vim-medieval https://github.com/gpanders/vim-medieval
  • Evaluates Markdown code sections
• Pyluatex https://www.ctan.org/pkg/pyluatex
• Magma-nvim https://github.com/dccsillag/magma-nvim
• Codi https://github.com/metakirby5/codi.vim
• Pythontex https://github.com/gpoore/pythontex
  • Evaluates Latex code sections
• Codebraid https://github.com/gpoore/codebraid
• Vim-ipython-cell https://github.com/hanschen/vim-ipython-cell
• Vim-ipython https://github.com/ivanov/vim-ipython
• Jupytext https://github.com/goerz/jupytext.vim
  • Alternative? https://github.com/mwouts/jupytext
• Ipython-vimception https://github.com/ivanov/ipython-vimception

Useful Vim plugins:

• https://github.com/sergei-grechanik/vim-terminal-images (Graphics in vim terminals)

Useful tools:

• https://pandoc.org/

Third-party issues

• Vim-plug https://github.com/junegunn/vim-plug/issues/1010#issuecomment-1221614232
• Pandoc https://github.com/jgm/pandoc/issues/8598
• Jupytext https://github.com/mwouts/jupytext/issues/220#issuecomment-1418209581
• Vim-LSC https://github.com/natebosch/vim-lsc/issues/469