mmf-setup 0.4.12

Meta-package to quickly initialize my python environment.

mmf-setup

This meta-project provides an easy way to install all of the python tools I typically use. It also serves as a fairly minimal example of setting up a package that pip can install, and specifying dependencies.

In particular, I structure it for the following use-cases:

1. Rapid installation and configuration of the tools I need. For example, I often use CoCalc. Whenever I create a new project, I need to perform some initialization. With this project, it is simply a matter of using pipx to install this package, and then using some of the tools. Specifically:

   pipx install mmf-setup
   mmf_setup cocalc mercurial hg-evolve hg-git black
   

   This does the following, and then links various configuration files to the home directory:

   pipx install mmf-setup
   pipx inject mmf-setup mercurial hg-evolve hg-git black
   curl -sSL https://install.python-poetry.org | python3 -
   

2. Initial setup of a python distribution on a new computer. This is a little more involved since one needs to first install python (I recommend using Miniconda) and then updating the tools.

3. A place to document various aspects of installing and setting up python and related tools. Some of this is old, but kept here for reference.

4. A generic way of setting sys.path for development work using the following (in order of decreasing precedence) by calling mmf_setup.set_path().

   • An entry in a pyproject.toml file somewhere in a higher-level directory.
   • An entry in a setup.cfg file somewhere in a higher-level directory.
   • The first parent directory with either of these files, a setup.py file, or .hg or .git directories.

Quickstart (TL;DR)

1. To get the notebook initialization features without having to install the package, just copy nbinit.py to your project. Importing this will try to execute import mmf_setup;mmf_setup.nbinit() but failing this, will manually run a similar code.

2. Install this package from the source directory, PyPI, etc. with one of the following:

   • Directly from PyPI

     python3 -m pip install --user mmf-setup[nbextensions]
     

   • From Source

     python3 -m pip install --user
     hg+https://alum.mit.edu/www/mforbes/hg/forbes-group/mmf-setup[nbextensions]
     

   • From Local Source (Run this from the source directory after you unpack it.)

     python3 -m pip install --user .[nbextensions]
     

   Note: these includes the nbextensions extra. You and run without the --user flag if you want to install them system-wide rather than into site.USER_BASE.

3. To get the notebook tools for Jupyter (IPython) notebooks, execute the following as a code cell in your notebook and then trust the notebook with File/Trust Notebook:

   import mmf_setup; mmf_setup.nbinit()
   

   This will set the theme which is implemented in the output cell so it is stored for use online such as when rendered through NBViewer. One can specify different themes. (Presently only theme='default' and theme='mmf' are supported.)

4. Mercurial: If you want to install mercurial with the hg-git and Evolve extensions, then you can do that with the hg extra:

   python3 -m pip install --user .[hg]
   

   This essentially runs pip install mercurial hg-git hg-evolve. You can then enable these in your environment by sourcing the mmf_setup script in one of the following ways:

   eval $"(mmf_setup -v)"     # Enable hg lg but not evolve, etc.
   eval $"(mmf_setup -v -H)"  # Enable evolve, etc.
   mmf_setup -v [-H]          # Shows what will be set set (dry run).
   

   To do this automatically when you login, add this line to your ~/.bashc or ~/.bash_profile scripts.

   Warning: the eval $"(mmf_setup -v -H)" option also includes a mercurial update hook which will add %include ../.hgrc to your projects .hg/hgrc file upon hg > update. This allows you to include project-specific mercurial customizations in your repository, but is a potential security risk. See the discussion below.

   These can also be enabled manually by adding the following to your ~/.hgrc file:

   # ~/.hgrc
   ...
   [extensions]
   evolve =
   topics =
   hggit =
   

Setting sys.path

The preferred way to work with python code is to install it, preferably into a virtual environment or a conda environment. By installing it, you will assure that your program or library is available to notebooks, etc. by being properly inserted into sys.path. When developing, code, one can also do a "editable" installs with python -m install -e \<path\> so that code updates are seen.

When developing code, however, this may not meet all use-cases, so we provide mmf_utils.set_path(cwd='.') which will set mmf_setup.ROOT and insert it at the start of sys.path.

The algorithm for determining ROOT is as follows starting from cwd (which is '.' by default, but can be specified in the call to set_path()). Note: in each case, the specified path must be an existing directory or it will be ignored.

• A value specified as mmf_setup.ROOT will override everything.

• A value in the environmental variable MMF_SETUP_ROOT will override everything else.

• An explicit ROOT entry in the first pyproject.toml found:

  # pyproject.toml
  [tool.mmf_setup]
  ROOT = 'src'
  

• An explict entry in the first setup.cfg:

  # setup.cfg
  [mmf_setup]
  ROOT = src
  

• The first parent directory to contain any of the files setup.py, setup.cfg, or pyproject.toml, or directories .git, or .hg.

If none of these yields a valid existing directory for ROOT, then it will not be set.

Mercurial (hg) Tools

If you source the output of the mmf_setup script with one of the following:

eval $"(mmf_setup -v)"
eval $"(mmf_setup -v -H)"

then your HGRCPATH will be amended to include hgrc.lga or hgrc.full respectively. The first adds a useful hg lga (hg lg for short) command which provides a concise graphical display:

$ hg lg
@  200:d michael (64 minutes ago) 0.4.0[0.4.0] tip
|   ENH,DOC,TST,API: Cleaned up set_path...
o  199:d michael (31 hours ago) 0.4.0[0.4.0]
|   WIP,CHK: Cleaning up set_path.
o  198:d michael (20 hours ago) 0.4.0[0.4.0]
|   DOC: Added developer Notes.md
...

The second version with -H adds some useful extensions: hg-git, Evolve, and enables Topics. The latter are required, for example, to interface with Heptapod.

Finally, the -H option enables an hg update hook, which adds %include ../.hgrc to your project's .hg/hgrc file when you call hg update. This is a potential security risk, because an untrusted repo could include dangerous commands in .hgrc. Thus, we require user intervention before including this:

$ eval $"(mmf_setup -v -H)"
$ hg clone https://alum.mit.edu/www/mforbes/hg/forbes-group/mmf-setup
...
updating to branch default
88 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd mmf-setup
$ hg up 0.4.0
Repository .../mmf-setup contains an .hgrc file.
Including this in .hg/hgrc is a potential security risk.
Only do this if you trust the repository.

Include this .hgrc in .hg/hgrc? [yN]y
Adding '%include ../.hgrc' to '.hg/hgrc'
...
$ tail .hg/hgrc
...
%include ../.hgrc

CoCalc

We provide some tools for working on CoCalc. To get started, simply do to the following once you have enabled network access on your project by applying a license:

pipx install mmf-setup
mmf_setup cocalc

To see exactly what this will do, run:

mmf_setup cocalc -v

Once this is done, you should ensure that, whenever you use a terminal on CoCalc, you appropriately set the following variables:

• LC_HG_USERNAME: Used by mercurial.
• LC_GIT_USEREMAIL, LC_GIT_USERNAME: Used by git.

These will identify you as you when you commit your work. (Needed, because all users share the same project. See cocalc#370 for details.)

I recommend the following:

1. Forward these variables in your SSH config file (~/.ssh/config):

   # ~/.ssh/config
   Host cc-project1
     User ff1cb986f...
   
   Host cc*
     HostName ssh.cocalc.com
     ForwardAgent yes
     SetEnv LC_HG_USERNAME=Your Full Name <[e-mail]>
     SetEnv LC_GIT_USERNAME=Your Full Name
     SetEnv LC_GIT_USEREMAIL=[e-mail]
     SetEnv LC_EDITOR=vi
   

   The appropriate value for User can be found in the project Settings on CoCalc.

   Optionally, use SendEnv ... and set these on your personal computer:

   # ~/.bashrc or similar
   LC_HG_USERNAME=Your Full Name <[e-mail]>
   LC_GIT_USEREMAIL=[e-mail]
   LC_GIT_USERNAME=Your Full Name
   

2. If you want to run Git or Mercurial from the CoCalc web interface, then create a named terminal -- i.e. Michael.term -- and then set these variables in the terminal startup script. (See https://doc.cocalc.com/terminal.html#startup-files for details.)

Notes

See Notes.md for developer notes. Other notes about python, IPython, etc. are stored in the docs folder.