# sqre-git-snapshot
LSST DM SQuaRE github snapshot management tools
## Components
### github-snapshot
Makes mirror clones of all public repositories in the specified
organizations.
### snapshot-purger
Removes old snapshots. Retention criteria are:
1. Everything is retained for a week.
2. Saturday backups are retained for at least a month.
3. First-of-the-month backups are retained for at least 3 months.
4. First-of-the-quarter backups are never automatically deleted.
## Installation
### On an AWS host
* In the AWS console:
* Create a new `t2.micro` (or whatever, but micro's plenty big) in
`us-west-2` using `ami-d2c924b2`.
* It needs IAM role `github-snapshot-s3-access`.
* Give it 30GB of SSD as root (you only really need enough for one
repository at a time, so more-free-space-than-the-biggest-repository
is good enough, but you're not charged extra (I think) for 30GB or
less).
* SSH is the only port it needs open.
* Launch it.
* Associate an EIP with it.
* (optional) Add a DNS record for that EIP
* Once it comes up, log in as `centos`, and then:
`sudo -i`
`hostnamectl set-hostname ghsnap.lsst.codes # Or whatever you called it`
`yum update -y`
`yum install -y epel-release && yum repolist`
`yum install -y git python-pip python-virtualenvwrapper jq`
`yum install -y emacs-nox # If you're not a barbarian`
`curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh -o install-git-lfs-repo.sh`
Examine `install-git-lfs-repo.sh` until you're sure it's not nefarious.
`bash ./install-git-lfs-repo.sh`
`yum -y install git-lfs-1.5.2-1.el7.x86_64`
Reboot.
* Once it is up again, log in as `centos`, and then:
`mkdir Venvs gh-snap git`
`cd git`
`git clone https://github.com/lsst-sqre/shelltools.git`
`cd ../gh-snap`
`ln -s ../git/shelltools/lsst-shellfuncs.bash`
`cd`
```bash
cat << 'EOF' >> .bashrc
if [ -f /usr/bin/virtualenvwrapper.sh ] && [ -z "${VIRTUAL_ENV}" ]; then
WORKON_HOME=${HOME}/Venvs
export WORKON_HOME
mkdir -p ${WORKON_HOME}
source /usr/bin/virtualenvwrapper.sh
fi
if [ -f "${HOME}/gh-snap/lsst-shellfuncs.bash" ]; then
source "${HOME}/gh-snap/lsst-shellfuncs.bash"
fi
EOF
```
* Log out and back in (as `centos` still). Then:
`mkvirtualenv github-snapshot`
`pip install sqre-github-snapshot`
`cd gh-snap`
```bash
cat << 'EOF' > run_as_cronjob
#!/bin/bash
action=$1
case $action in
"purge"|"snap")
;;
*)
echo 1>&2 "Action must be 'purge' or 'snap'"
exit 1
esac
declare -F | grep -q '^declare -f workon$'
rc=$?
if [ ${rc} -ne 0 ]; then
. ${HOME}/.bashrc
else
declare -F | grep -q '^declare -f deactivate$'
rc=$?
if [ ${rc} -ne 0 ]; then
. ${HOME}/.bashrc
fi
fi
if [ -n "${VIRTUAL_ENV}" ]; then
vname=$(basename "${VIRTUAL_ENV}")
fi
if [ "${vname}" != "github-snapshot" ]; then
workon github-snapshot
fi
check_github_lfs
set_aws_variables
if [ "${action}" == "purge" ]; then
snapshot-purger
else
github-snapshot
fi
EOF
```
* Install cron job:
`chmod 0755 run_as_cronjob`
set `$EDITOR` if you don't like `vi`
`crontab -e`
Add the following:
```
# Take backup snapshots every night at 12:23 AM
# Purge old backups every night at 4:46 AM
23 0 * * * /home/centos/gh-snap/run_as_cronjob snap
46 4 * * * /home/centos/gh-snap/run_as_cronjob purge
```
### Docker
Build a container from the included `Dockerfile`.
You may want to specify some or all of the following environment
variables with -e when you do a `docker run`:
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`,
`AWS_DEFAULT_REGION`, `GITHUB_ORGS`, `GITHUB_TOKEN`, `S3_BACKUP_BUCKET`
The default command is to take a snapshot; use `snapshot-purger` as an
appended command on the docker run command line to run a purge instead.
You can also just pull the image with `docker pull
lsstsqre/github-snapshot`.
Metadata
Release files for sqre-github-snapshot 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sqre-github-snapshot-0.2.1.tar.gz | 9.2 kB | Without provenance | Details for sqre-github-snapshot-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sqre_github_snapshot-0.2.1-py2-none-any.whl | Python 2 | none | any | Details for sqre_github_snapshot-0.2.1-py2-none-any.whl |
Total release size: 21.4 kB
Release files / sqre-github-snapshot-0.2.1.tar.gz
| Download URL | sqre-github-snapshot-0.2.1.tar.gz |
|---|---|
| Size | 9.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f8f73b60054405f7cc206d9facf3f911233159e54c5eb008d22a4a97609fe8d1
|
|
BLAKE2b-256 checksum How to use checksums |
508130b98d8cf5201b2fbdb2a2391863ee6e9196e74984ee54c986e320d47640
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / sqre_github_snapshot-0.2.1-py2-none-any.whl
| Download URL | sqre_github_snapshot-0.2.1-py2-none-any.whl |
|---|---|
| Size | 12.2 kB |
| Tags | Python 2 |
|
SHA-256 checksum How to use checksums |
798a53ea803dc3cc0f28f0df6ff70b0f0ffe10198013e31ce3cfe0ce4d6135d3
|
|
BLAKE2b-256 checksum How to use checksums |
6594d00f7557e6949e15acbb8154f469af5257beec4eb61a5025961ad5be59f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for sqre-github-snapshot 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.