Skip to main content

colab miscellaneous utils

Project description

colab miscellaneous utils Codacy BadgeCode style: blackLicense: MITPyPI version

Miscellaneous utils mainly intended for use in colab

Installation

pip install clmutils  # clm: colab-misc
# pip install clmutils -U to update

Or clone the repo https://github.com/ffreemt/colab-misc-utils.git in Colab/jupyter:

!git clone https://github.com/ffreemt/colab-misc-utils.git
%cd colab-misc-utils
!pip install -r requirements.txt

Usage: with a setting file dotenv/.env

  1. setup_git

    from clmutils import setup_git, Settings
    config = Settings()
    setup_git(
      user_name=config.user_name,
      user_email=config.user_email,
      priv_key=config.gh_key
    )
    

    Do the usual git pull, amend codes and git push stuff.

    For more on Connecting to GitHub with SSH, refer to https://docs.github.com/cn/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh

  2. setup_ssh_tunnel

    • Run these lines in Colab
    from clmutils import setup_ssh_tunnel, Settings
    config = Settings()
    setup_ssh_tunnel(
      remote_host=config.remote_host,
      remote_user=config.remote_user,
      priv_key=config.cl_key,
      remote_pubkey=config.remote_pubkey
    )
    
    • Append the value of cl_key_pub in dotenv/.env (below) in remote computer's ~/.ssh/authorized_keys.

    • In the remote computer:

    ssh colab
    

clmutils.Setting will look for dotenv or .env in /content/drive/MyDrive if google drive is momunted; otherwise it looks for .env in the current dir and parents dir.

dotenv/.env basically contains the necessary information to setup git for github or ssh tunnel:

  • For git:
    • gh_key: private key
    • user_name/user_email;
  • For reverse ssh tunnel:
    • cl_key/cl_key_pub: private key/public key of Colab
    • remote_pubkey: public_key of the remote computer
    • remote_user: login name

dotenv/.env is read by clmutils.Settin as follows

from clmutils import Settings
config = Settings()
# config = Setting(_env_file=file_path)

config will then have attributes: config.gh_key, user_name, user_email, cl_key, cl_key_pub, remote_pubkey, remote_user. If a particular attribute is not assigned a value in dotenv/.env, it defaults to empty string ("").

dotenv or .env has the following info and format (lines starting with a # are comments):

# for git
user_email = "...@......"
user_name = "......"
gh_key = "-----BEGIN EC PRIVATE KEY-----
...............................................................9
...............................................................J
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
"

# for ssh tunnel
remote_host = "168.138.222.163"
remote_user = "ubuntu"

# colab's private key, to be put in ~/.ssh/id_ed25519
# or as specified in ~/.ssh/config
cl_key = "-----BEGIN OPENSSH PRIVATE KEY-----
.....................................................................W
QyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQAAAJDlRQSF5UUE
hQAAAAtzc2gtZWQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQ
.....................................................................f
DQRG5jrz/VCKKt2uu2eBAAAACWZvci1jb2xhYgECAwQ=
-----END OPENSSH PRIVATE KEY-----
"

# to be put in remote computer's ~/.ssh/autorized_keys
cl_key_pub = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIKVka3/2JsB32jG5Zx+5KOfDQRG5jrz/VCKKt2uu2eB colab-key"

# to be put in colab's ~/.ssh/authorized_keys
remote_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE60sowZ4M0MA5nTGIH1RN54zERTuWSddFKqyeWZzaKv for-colab"

Usage: withtout dotenv/.env

Set up github with ssh using clmutils.setup_git

For manually setting up github with ssh, refer to https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh

Assume you configure git as follows:

git config --global user.email your-email-address
git config --global user.name your-github-username

With clmutils, you'd do:

from clmutils import setup_git`

user_email = "your-email-address"
user_name = "your-github-username"
gh_key = \
"""
-----BEGIN EC PRIVATE KEY-----
MH.............................................................9
AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
""".strip() + "\n"

setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)

You then upload the public key for gh_key to https://github.com/settings/keys.

Refer to Step 2 https://support.cloudways.com/using-git-command-line-ssh/ for how to generate a private/public key pair. You can also use clmutils.gen_keypair to do that in Python.

Alternatively, set up github with ssh in 4 steps

  1. Write a private key to ~/.ssh/gh-key
from clmutils import create_file
gh_key = \
"""
-----BEGIN EC PRIVATE KEY-----
MH.............................................................9
AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
""".strip() + "\n"
# Do not remove .strip() + "\n"
# the private key is very picky about format

create_file(gh_key, dest="~/.ssh/gh-key")
  1. Set up github.com config for git push
from clmutils import append_content
config_github_entry = \
"""
Host github.com
   HostName github.com
   User git
   IdentityFile ~/.ssh/gh-key
"""
append_content(config_github_entry, dest="~/.ssh/config")
  1. Verify that everything is OK, from a cell
!ssh -o StrictHostKeyChecking=no -T git@github.com

If you see something similar to

Hi your-name! You've successfully authenticated, but GitHub does not provide shell access.

you are good to go.

  1. git config --global You can now set up git config global from a cell, e.g.
!git config --global user.email your-email-address
!git config --global user.name your-github-username
# !ssh-keyscan github.com >> ~/.ssh/known_hosts

You are ready to clone your own repo, run your app and generate new data, update the repo and push to github.

Utils planned

  • :white_check_mark: setup_git sets up git for github.com

  • :white_check_mark: reverse_ssh_tunnel sets up a reverse ssh tunnel to a remote host with via autossh

  • Auxiliary utils

    • :white_check_mark: create_file creates a file with given mode, e.g. for .ssh/id_rsa or IdentityFile in .ssh/config

    • :white_check_mark: apppend_content appends some content to a file, e.g., for appended a public key to .ssh/authorized_keys

    • :white_check_mark: chmod600 chmod of a file

    • :white_check_mark: gen_keypair generates private/public key pair.

    • :white_check_mark: run_cmd wraps subprocess.check_output

    • :white_check_mark: run_cmd1 wraps subprocess.Popen

    • :white_check_mark: check_running shows running processes with names containings a certain patter; based on psutil.Proceess()'s cmdlineandstatus`

Demo: notebooks in Colab

git push from Colab: one line (setup_git())

!pip install clmutils
from clmutils import setup_git
gh_key = """..."""
user_name = "..."
user_email = "..."
setup_git(setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)

Open In Colab

git push from Colab in several steps

Open In Colab (in Chinese, shouldn't be too difficult to follow without knowing any Chinese though, just click through :smiley:)

Reverse ssh tunnel for ssh to Colab VM

Open In Colab in English (I may provide a Chinese version later)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

clmutils-0.1.5.tar.gz (10.9 kB view hashes)

Uploaded Source

Built Distribution

clmutils-0.1.5-py3-none-any.whl (13.4 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page