colab miscellaneous utils 



Miscellaneous utils mainly intended for use in colab
Installation
pip install clmutils # clm: colab-misc
# pip install clmutils -U to update
Or clone the repo https://github.com/ffreemt/colab-misc-utils.git in Colab/jupyter:
!git clone https://github.com/ffreemt/colab-misc-utils.git
%cd colab-misc-utils
!pip install -r requirements.txt
Usage: with a setting file dotenv/.env
-
setup_gitfrom clmutils import setup_git, Settings config = Settings() setup_git( user_name=config.user_name, user_email=config.user_email, priv_key=config.gh_key )
Do the usual
git pull, amend codes andgit pushstuff.For more on
Connecting to GitHub with SSH, refer to https://docs.github.com/cn/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh -
setup_ssh_tunnel- Run these lines in Colab
from clmutils import setup_ssh_tunnel, Settings config = Settings() setup_ssh_tunnel( remote_host=config.remote_host, remote_user=config.remote_user, priv_key=config.cl_key, remote_pubkey=config.remote_pubkey )
-
Append the value of
cl_key_pubindotenv/.env(below) in remote computer's~/.ssh/authorized_keys. -
In the remote computer:
ssh colab
clmutils.Setting will look for dotenv or .env in /content/drive/MyDrive if google drive is momunted; otherwise it looks for .env in the current dir and parents dir.
dotenv/.env basically contains the necessary information to setup git for github or ssh tunnel:
- For git:
- gh_key: private key
- user_name/user_email;
- For reverse ssh tunnel:
- cl_key/cl_key_pub: private key/public key of Colab
- remote_pubkey: public_key of the remote computer
- remote_user: login name
dotenv/.env is read by clmutils.Settin as follows
from clmutils import Settings
config = Settings()
# config = Setting(_env_file=file_path)
config will then have attributes: config.gh_key, user_name, user_email, cl_key, cl_key_pub, remote_pubkey, remote_user.
If a particular attribute is not assigned a value in dotenv/.env, it defaults to empty string ("").
dotenv or .env has the following info and format (lines starting with a # are comments):
# for git
user_email = "...@......"
user_name = "......"
gh_key = "-----BEGIN EC PRIVATE KEY-----
...............................................................9
...............................................................J
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
"
# for ssh tunnel
remote_host = "168.138.222.163"
remote_user = "ubuntu"
# colab's private key, to be put in ~/.ssh/id_ed25519
# or as specified in ~/.ssh/config
cl_key = "-----BEGIN OPENSSH PRIVATE KEY-----
.....................................................................W
QyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQAAAJDlRQSF5UUE
hQAAAAtzc2gtZWQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQ
.....................................................................f
DQRG5jrz/VCKKt2uu2eBAAAACWZvci1jb2xhYgECAwQ=
-----END OPENSSH PRIVATE KEY-----
"
# to be put in remote computer's ~/.ssh/autorized_keys
cl_key_pub = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIKVka3/2JsB32jG5Zx+5KOfDQRG5jrz/VCKKt2uu2eB colab-key"
# to be put in colab's ~/.ssh/authorized_keys
remote_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE60sowZ4M0MA5nTGIH1RN54zERTuWSddFKqyeWZzaKv for-colab"
Usage: withtout dotenv/.env
Set up github with ssh using clmutils.setup_git
For manually setting up github with ssh, refer to https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh
Assume you configure git as follows:
git config --global user.email your-email-address
git config --global user.name your-github-username
With clmutils, you'd do:
from clmutils import setup_git`
user_email = "your-email-address"
user_name = "your-github-username"
gh_key = \
"""
-----BEGIN EC PRIVATE KEY-----
MH.............................................................9
AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
""".strip() + "\n"
setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)
You then upload the public key for gh_key to https://github.com/settings/keys.
Refer to Step 2 https://support.cloudways.com/using-git-command-line-ssh/ for how to generate a private/public key pair. You can also use clmutils.gen_keypair to do that in Python.
Alternatively, set up github with ssh in 4 steps
- Write a private key to
~/.ssh/gh-key
from clmutils import create_file
gh_key = \
"""
-----BEGIN EC PRIVATE KEY-----
MH.............................................................9
AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
""".strip() + "\n"
# Do not remove .strip() + "\n"
# the private key is very picky about format
create_file(gh_key, dest="~/.ssh/gh-key")
- Set up
github.comconfig forgit push
from clmutils import append_content
config_github_entry = \
"""
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/gh-key
"""
append_content(config_github_entry, dest="~/.ssh/config")
- Verify that everything is OK, from a cell
!ssh -o StrictHostKeyChecking=no -T git@github.com
If you see something similar to
Hi your-name! You've successfully authenticated, but GitHub does not provide shell access.
you are good to go.
git config --globalYou can now set upgit config globalfrom a cell, e.g.
!git config --global user.email your-email-address
!git config --global user.name your-github-username
# !ssh-keyscan github.com >> ~/.ssh/known_hosts
You are ready to clone your own repo, run your app and generate new data, update the repo and push to github.
Utils planned
-
✅
setup_gitsets upgitforgithub.com -
✅
reverse_ssh_tunnelsets up a reverse ssh tunnel to a remote host with via autossh -
Auxiliary utils
-
✅
create_filecreates a file with given mode, e.g. for.ssh/id_rsaorIdentityFilein.ssh/config -
✅
apppend_contentappends some content to a file, e.g., for appended a public key to.ssh/authorized_keys -
✅
chmod600chmodof a file -
✅
gen_keypairgenerates private/public key pair. -
✅
run_cmdwrapssubprocess.check_output -
✅
run_cmd1wrapssubprocess.Popen -
✅
check_runningshows running processes with names containings a certain patter; based onpsutil.Proceess()'scmdlineandstatus`
-
Demo: notebooks in Colab
git push from Colab: one line (setup_git())
!pip install clmutils
from clmutils import setup_git
gh_key = """..."""
user_name = "..."
user_email = "..."
setup_git(setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)
git push from Colab in several steps
(in Chinese, shouldn't be too difficult to follow without knowing any Chinese though, just click through 😃)
Reverse ssh tunnel for ssh to Colab VM
Metadata
Release files for clmutils 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| clmutils-0.1.5.tar.gz | 10.9 kB | Without provenance | Details for clmutils-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| clmutils-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for clmutils-0.1.5-py3-none-any.whl |
Total release size: 24.3 kB
Release files / clmutils-0.1.5.tar.gz
| Download URL | clmutils-0.1.5.tar.gz |
|---|---|
| Size | 10.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
47233a552bcca142497ca0114a1e2c58bf4b7f6593c2bb40dee41116a8d53da4
|
|
BLAKE2b-256 checksum How to use checksums |
b32b0a4e12920e7fd1d2ea3e3fb752008f52d945cb5f7f8dce3bdbf0feda19b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.5 CPython/3.7.2 Windows/10
|
Release files / clmutils-0.1.5-py3-none-any.whl
| Download URL | clmutils-0.1.5-py3-none-any.whl |
|---|---|
| Size | 13.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
00dfa1e1590db1ba1b994b79f76682300152c6ffae907e311d6ef11af4ab1284
|
|
BLAKE2b-256 checksum How to use checksums |
ee097547a118ad0e43a82f5de6c29fe7598f07e056fc2c6a5bbbe100c933f1eb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.0.5 CPython/3.7.2 Windows/10
|
Security for clmutils 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.