colab miscellaneous utils
Project description
colab miscellaneous utils
Miscellaneous utils mainly intended for use in colab
Installation
pip install clmutils # clm: colab-misc
# pip install clmutils -U to update
Or clone the repo https://github.com/ffreemt/colab-misc-utils.git
in Colab/jupyter:
!git clone https://github.com/ffreemt/colab-misc-utils.git
%cd colab-misc-utils
!pip install -r requirements.txt
Usage: with a setting file dotenv/.env
-
setup_git
from clmutils import setup_git, Settings config = Settings() setup_git( user_name=config.user_name, user_email=config.user_email, priv_key=config.gh_key )
Do the usual
git pull
, amend codes andgit push
stuff.For more on
Connecting to GitHub with SSH
, refer to https://docs.github.com/cn/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh -
setup_ssh_tunnel
- Run these lines in Colab
from clmutils import setup_ssh_tunnel, Settings config = Settings() setup_ssh_tunnel( remote_host=config.remote_host, remote_user=config.remote_user, priv_key=config.cl_key, remote_pubkey=config.remote_pubkey )
-
Append the value of
cl_key_pub
indotenv/.env
(below) in remote computer's~/.ssh/authorized_keys
. -
In the remote computer:
ssh colab
clmutils.Setting
will look for dotenv
or .env
in /content/drive/MyDrive
if google drive is momunted; otherwise it looks for .env
in the current dir and parents dir.
dotenv/.env
basically contains the necessary information to setup git for github or ssh tunnel:
- For git:
- gh_key: private key
- user_name/user_email;
- For reverse ssh tunnel:
- cl_key/cl_key_pub: private key/public key of Colab
- remote_pubkey: public_key of the remote computer
- remote_user: login name
dotenv/.env
is read by clmutils.Settin
as follows
from clmutils import Settings
config = Settings()
# config = Setting(_env_file=file_path)
config
will then have attributes: config.gh_key, user_name, user_email, cl_key, cl_key_pub, remote_pubkey, remote_user
.
If a particular attribute is not assigned a value in dotenv/.env
, it defaults to empty string (""
).
dotenv
or .env
has the following info and format (lines starting with a #
are comments):
# for git
user_email = "...@......"
user_name = "......"
gh_key = "-----BEGIN EC PRIVATE KEY-----
...............................................................9
...............................................................J
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
"
# for ssh tunnel
remote_host = "168.138.222.163"
remote_user = "ubuntu"
# colab's private key, to be put in ~/.ssh/id_ed25519
# or as specified in ~/.ssh/config
cl_key = "-----BEGIN OPENSSH PRIVATE KEY-----
.....................................................................W
QyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQAAAJDlRQSF5UUE
hQAAAAtzc2gtZWQyNTUxOQAAACCClZGt/9ibAd9oxuWcfuSjnw0ERuY68/1QiirdrrtngQ
.....................................................................f
DQRG5jrz/VCKKt2uu2eBAAAACWZvci1jb2xhYgECAwQ=
-----END OPENSSH PRIVATE KEY-----
"
# to be put in remote computer's ~/.ssh/autorized_keys
cl_key_pub = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIKVka3/2JsB32jG5Zx+5KOfDQRG5jrz/VCKKt2uu2eB colab-key"
# to be put in colab's ~/.ssh/authorized_keys
remote_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE60sowZ4M0MA5nTGIH1RN54zERTuWSddFKqyeWZzaKv for-colab"
Usage: withtout dotenv/.env
Set up github
with ssh using clmutils.setup_git
For manually setting up github with ssh, refer to https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh
Assume you configure git as follows:
git config --global user.email your-email-address
git config --global user.name your-github-username
With clmutils
, you'd do:
from clmutils import setup_git`
user_email = "your-email-address"
user_name = "your-github-username"
gh_key = \
"""
-----BEGIN EC PRIVATE KEY-----
MH.............................................................9
AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
""".strip() + "\n"
setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)
You then upload the public key
for gh_key
to https://github.com/settings/keys.
Refer to Step 2 https://support.cloudways.com/using-git-command-line-ssh/ for how to generate a private/public key pair. You can also use clmutils.gen_keypair to do that in Python.
Alternatively, set up github
with ssh in 4 steps
- Write a private key to
~/.ssh/gh-key
from clmutils import create_file
gh_key = \
"""
-----BEGIN EC PRIVATE KEY-----
MH.............................................................9
AwEHoUQDQgAEoLlGQRzIVHYw3gvC/QFw3Ru45zGawaBaCq6jTqdyH2Kp8zIB3TdJ
K9ztlJBRRAOHh5sPhQ4QpdZH1v1rWeDWIQ==
-----END EC PRIVATE KEY-----
""".strip() + "\n"
# Do not remove .strip() + "\n"
# the private key is very picky about format
create_file(gh_key, dest="~/.ssh/gh-key")
- Set up
github.com
config forgit push
from clmutils import append_content
config_github_entry = \
"""
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/gh-key
"""
append_content(config_github_entry, dest="~/.ssh/config")
- Verify that everything is OK, from a cell
!ssh -o StrictHostKeyChecking=no -T git@github.com
If you see something similar to
Hi your-name! You've successfully authenticated, but GitHub does not provide shell access.
you are good to go.
git config --global
You can now set upgit config global
from a cell, e.g.
!git config --global user.email your-email-address
!git config --global user.name your-github-username
# !ssh-keyscan github.com >> ~/.ssh/known_hosts
You are ready to clone your own repo, run your app and generate new data, update the repo and push to github
.
Utils planned
-
:white_check_mark:
setup_git
sets upgit
forgithub.com
-
:white_check_mark:
reverse_ssh_tunnel
sets up a reverse ssh tunnel to a remote host with via autossh -
Auxiliary utils
-
:white_check_mark:
create_file
creates a file with given mode, e.g. for.ssh/id_rsa
orIdentityFile
in.ssh/config
-
:white_check_mark:
apppend_content
appends some content to a file, e.g., for appended a public key to.ssh/authorized_keys
-
:white_check_mark:
chmod600
chmod
of a file -
:white_check_mark:
gen_keypair
generates private/public key pair. -
:white_check_mark:
run_cmd
wrapssubprocess.check_output
-
:white_check_mark:
run_cmd1
wrapssubprocess.Popen
-
:white_check_mark:
check_running
shows running processes with names containings a certain patter; based onpsutil.Proceess()'s
cmdlineand
status`
-
Demo: notebooks in Colab
git push from Colab: one line (setup_git()
)
!pip install clmutils
from clmutils import setup_git
gh_key = """..."""
user_name = "..."
user_email = "..."
setup_git(setup_git(user_email=user_email, user_name=user_name, priv_key=gh_key)
git push
from Colab in several steps
(in Chinese, shouldn't be too difficult to follow without knowing any Chinese though, just click through :smiley:)
Reverse ssh tunnel for ssh to Colab VM
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file clmutils-0.1.5.tar.gz
.
File metadata
- Download URL: clmutils-0.1.5.tar.gz
- Upload date:
- Size: 10.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.0.5 CPython/3.7.2 Windows/10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47233a552bcca142497ca0114a1e2c58bf4b7f6593c2bb40dee41116a8d53da4 |
|
MD5 | 0b67ab737331240e65a36acadca0c310 |
|
BLAKE2b-256 | b32b0a4e12920e7fd1d2ea3e3fb752008f52d945cb5f7f8dce3bdbf0feda19b8 |
File details
Details for the file clmutils-0.1.5-py3-none-any.whl
.
File metadata
- Download URL: clmutils-0.1.5-py3-none-any.whl
- Upload date:
- Size: 13.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.0.5 CPython/3.7.2 Windows/10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00dfa1e1590db1ba1b994b79f76682300152c6ffae907e311d6ef11af4ab1284 |
|
MD5 | d4694e5b3cb2ad05be8bf27c89d554e7 |
|
BLAKE2b-256 | ee097547a118ad0e43a82f5de6c29fe7598f07e056fc2c6a5bbbe100c933f1eb |