Skip to main content

Jupyter quant research environment.

Project description

Jupyter Quant

A dockerized Jupyter quant research environment.

Highlights

  • It can be used as a docker image or pypi package.
  • Includes tools for quant analysis, statsmodels, pymc, arch, py_vollib, zipline-reloaded, PyPortfolioOpt, etc.
  • The usual suspects are included, numpy, pandas, sci-py, scikit-learn, yellowbricks, shap, optuna.
  • ib_async for Interactive Broker connectivity. Works well with IB Gateway docker image. ib_insync has been invaluable all this time, RIP Ewald.
  • Includes all major Python packages for statistical and time series analysis, see requirements. For an extensive list check list installed packages section.
  • Zipline-reloaded, pyfolio-reloaded and alphalens-reloaded.
  • ib_fundamental for IBKR fundamental data.
  • You can install it as a python package, just pip install -U jupyter-quant
  • Designed for ephemeral containers. Relevant data for your environment will survive your container.
  • Optimized for size, it's a 2GB image vs 4GB for jupyter/scipy-notebook
  • Includes jedi language server, jupyterlab-lsp, black and isort.
  • It does NOT include conda/mamba. All packages are installed with pip under ~/.local/lib/python. Which should be mounted in a dedicated volume to preserve your environment.
  • Includes Cython, Numba, bottleneck and numexpr to speed up things
  • sudo, so you can install new packages if needed.
  • bash and stow, so you can BYODF (bring your dotfiles). Plus common command line utilities like git, less, nano (tiny), jq, ssh, curl, bash completion and others.
  • Support for apt cache. If you have other Linux boxes using it can leverage your package cache.
  • It does not include a built environment. If you need to install a package that does not provide wheels you can build your wheels, as explained in common tasks

Quick Start

To use jupyter-quant as a pypi package see install quant package.

Create a docker-compose.yml file with this content

services:
  jupyter-quant:
    image: gnzsnz/jupyter-quant:${IMAGE_VERSION}
    environment:
      APT_PROXY: ${APT_PROXY:-}
      BYODF: ${BYODF:-}
      SSH_KEYDIR: ${SSH_KEYDIR:-}
      START_SCRIPTS: ${START_SCRIPTS:-}
      TZ: ${QUANT_TZ:-}
    restart: unless-stopped
    ports:
      - ${LISTEN_PORT}:8888
    volumes:
      - quant_conf:/home/gordon/.config
      - quant_data:/home/gordon/.local
      - ${PWD}/Notebooks:/home/gordon/Notebooks

volumes:
  quant_conf:
  quant_data:

You can use .env-dist as your starting point.

cp .env-dist .env
# verify everything looks good
docker compose config
docker compose up

Volumes

The image is designed to work with 3 volumes:

  1. quant_data - volume for ~/.local folder. It contains caches and all Python packages. This enables to install additional packages through pip.
  2. quant_conf - volume for ~/.config, all config goes here. This includes jupyter, ipython, matplotlib, etc
  3. Bind mount (but you could use a named volume) - volume for all notebooks, under ~/Notebooks.

This allows to have ephemeral containers and to keep your notebooks (3), your config (2) and your additional packages (1). Eventually, you would need to update the image, in this case, your notebooks (3) can move without issues, your config (2) should still work but no warranty and your packages in quant_data could still be used but you should refresh it with a new image. Eventually, you would need to refresh (1) and less frequently (2)

Common tasks

Get running server URL

docker exec -it jupyterquant jupyter-server list
Currently running servers:
http://40798f7a604a:8888/?token=
ebf9e870d2aa0ed877590eb83b4d3bbbdfbd55467422a167 :: /home/gordon/Notebooks

or

docker logs -t jupyter-quant 2>&1 | grep '127.0.0.1:8888/lab?token='

You will need to change hostname (40798f7a604a in this case) or 127.0.0.1 by your docker host ip.

Show jupyter config

docker exec -it jupyter-quant jupyter-server --show-config

Set password

docker exec -it jupyter-quant jupyter-server password

Get command line help

docker exec -it jupyter-quant jupyter-server --help
docker exec -it jupyter-quant jupyter-lab --help

List installed packages

docker exec -it jupyter-quant pip list
# outdated packages
docker exec -it jupyter-quant pip list -o

Pass parameters to jupyter-lab

docker run -it --rm gnzsnz/jupyter-quant --core-mode
docker run -it --rm gnzsnz/jupyter-quant --show-config-json

Run a command in the container

docker run -it --rm gnzsnz/jupyter-quant bash

Build wheels outside the container

Build wheels outside the container and import wheels into the container

# make sure python version match .env-dist
docker run -it --rm -v $PWD/wheels:/wheels python:3.11 bash
pip wheel --no-cache-dir --wheel-dir /wheels numpy

This will build wheels for numpy (or any other package that you need) and save the file in $PWD/wheels. Then you can copy the wheels in your notebook mount (3 above) and install it within the container. You can even drag and drop into Jupyter.

Install your dotfiles

git clone your dotfiles to Notebook/etc/dotfiles, set environment variable BYODF=/home/gordon/Notebook/etc/dotfiles in your docker-compose.yml When the container starts up stow will create links like /home/gordon/.bashrc

Install your SSH keys

You need to define environment variable SSH_KEY_DIR which should point to a location with your keys. The suggested place is SSH_KEYDIR=/home/gordon/Notebooks/etc/ssh, make sure the director has the right permissions. Something like chmod 700 Notebooks/etc/ssh should work.

The entrypoint.sh script will create a symbolic link pointing to $SSH_KEYDIR on /home/gordon/.ssh.

Within Jupyter's terminal, you can then:

# start agent
eval $(ssh-agent)
# add keys to agent
ssh-add
# open a tunnel
ssh -fNL 4001:localhost:4001 gordon@bastion-ssh

Run scripts at start up

If you define START_SCRIPTS env variable with a path, all scripts on that directory will be executed at start up. The sample .env-dist file contains a commented line with START_SCRIPTS=/home/gordon/Notebooks/etc/start_scripts as an example and recommended location.

Files should have a .sh suffix and should run under bash. in directory start_scripts you will find example scripts to load ssh keys and install python packages.

Install jupyter-quant package

Jupyter-quant is available as a package in pypi. It's a meta-package that pulls all dependencies in it's highest possible version.

Dependencies:

# ubuntu/debian, see install instructions above for TA-lib
sudo apt-get install libhdf5-dev

# osx
brew install hdf5 ta-lib

Install pypi package.

pip install -U jupyter-quant

Additional options supported are

pip install -U jupyter-quant[bayes] # to install pymc & arviz/graphviz

pip install -U jupyter-quant[sk-util] # to install skfolio & sktime

jupyter-quant it's a meta-package that pins all it's dependencies versions. If you need/want to upgrade a dependency you can uninstall jupyter-quant, although this can break interdependencies. Or install from git, where it's updated regularly.

# git install
pip install -U git+https://github.com/quantbelt/jupyter-quant.git

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jupyter_quant-2409.3.tar.gz (15.6 kB view details)

Uploaded Source

Built Distribution

jupyter_quant-2409.3-py3-none-any.whl (13.5 kB view details)

Uploaded Python 3

File details

Details for the file jupyter_quant-2409.3.tar.gz.

File metadata

  • Download URL: jupyter_quant-2409.3.tar.gz
  • Upload date:
  • Size: 15.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.6

File hashes

Hashes for jupyter_quant-2409.3.tar.gz
Algorithm Hash digest
SHA256 a296cd6931be1e6afe186e6620a12e4f1b34eaed93ab1cb835f61f9416b6ded5
MD5 fe5450a26ee0274476e8033b1dc5a623
BLAKE2b-256 7c1456b27f124e278731c727f5d043464824c2fc61b045f1b24a019c6262211f

See more details on using hashes here.

File details

Details for the file jupyter_quant-2409.3-py3-none-any.whl.

File metadata

File hashes

Hashes for jupyter_quant-2409.3-py3-none-any.whl
Algorithm Hash digest
SHA256 f01ca475a89bdfd38d0fceb6b18e241476313bcd004d37c1539121a6db8c022d
MD5 baef825c080b7bce22d5282f803543ee
BLAKE2b-256 00fe05e25fbf44a8ccab74db96cd14c17de883f7fa108907c7bbdd25e93cd869

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page