sqre-github-snapshot 0.2.1

LSST Data Management SQuaRE Github Organization Snapshotter

# 

# sqre-git-snapshot

LSST DM SQuaRE github snapshot management tools

## Components

### github-snapshot

Makes mirror clones of all public repositories in the specified
organizations.

### snapshot-purger

Removes old snapshots. Retention criteria are:

1. Everything is retained for a week.
2. Saturday backups are retained for at least a month.
3. First-of-the-month backups are retained for at least 3 months.
4. First-of-the-quarter backups are never automatically deleted.

## Installation

### On an AWS host

* In the AWS console:
* Create a new `t2.micro` (or whatever, but micro's plenty big) in
`us-west-2` using `ami-d2c924b2`.
* It needs IAM role `github-snapshot-s3-access`.
* Give it 30GB of SSD as root (you only really need enough for one
repository at a time, so more-free-space-than-the-biggest-repository
is good enough, but you're not charged extra (I think) for 30GB or
less).
* SSH is the only port it needs open.
* Launch it.
* Associate an EIP with it.
* (optional) Add a DNS record for that EIP
* Once it comes up, log in as `centos`, and then:
`sudo -i`
`hostnamectl set-hostname ghsnap.lsst.codes # Or whatever you called it`
`yum update -y`
`yum install -y epel-release && yum repolist`
`yum install -y git python-pip python-virtualenvwrapper jq`
`yum install -y emacs-nox # If you're not a barbarian`
`curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh -o install-git-lfs-repo.sh`
Examine `install-git-lfs-repo.sh` until you're sure it's not nefarious.
`bash ./install-git-lfs-repo.sh`
`yum -y install git-lfs-1.5.2-1.el7.x86_64`
Reboot.
* Once it is up again, log in as `centos`, and then:
`mkdir Venvs gh-snap git`
`cd git`
`git clone https://github.com/lsst-sqre/shelltools.git`
`cd ../gh-snap`
`ln -s ../git/shelltools/lsst-shellfuncs.bash`
`cd`
```bash
cat << 'EOF' >> .bashrc
if [ -f /usr/bin/virtualenvwrapper.sh ] && [ -z "${VIRTUAL_ENV}" ]; then
WORKON_HOME=${HOME}/Venvs
export WORKON_HOME
mkdir -p ${WORKON_HOME}
source /usr/bin/virtualenvwrapper.sh
fi

if [ -f "${HOME}/gh-snap/lsst-shellfuncs.bash" ]; then
source "${HOME}/gh-snap/lsst-shellfuncs.bash"
fi
EOF
```
* Log out and back in (as `centos` still). Then:
`mkvirtualenv github-snapshot`
`pip install sqre-github-snapshot`
`cd gh-snap`
```bash
cat << 'EOF' > run_as_cronjob
#!/bin/bash
action=$1
case $action in
"purge"|"snap")
;;
*)
echo 1>&2 "Action must be 'purge' or 'snap'"
exit 1
esac

declare -F | grep -q '^declare -f workon$'
rc=$?
if [ ${rc} -ne 0 ]; then
. ${HOME}/.bashrc
else
declare -F | grep -q '^declare -f deactivate$'
rc=$?
if [ ${rc} -ne 0 ]; then
. ${HOME}/.bashrc
fi
fi

if [ -n "${VIRTUAL_ENV}" ]; then
vname=$(basename "${VIRTUAL_ENV}")
fi
if [ "${vname}" != "github-snapshot" ]; then
workon github-snapshot
fi

check_github_lfs
set_aws_variables

if [ "${action}" == "purge" ]; then
snapshot-purger
else
github-snapshot
fi
EOF
```
* Install cron job:
`chmod 0755 run_as_cronjob`
set `$EDITOR` if you don't like `vi`
`crontab -e`
Add the following:
```
# Take backup snapshots every night at 12:23 AM
# Purge old backups every night at 4:46 AM

23 0 * * * /home/centos/gh-snap/run_as_cronjob snap
46 4 * * * /home/centos/gh-snap/run_as_cronjob purge
```

### Docker

Build a container from the included `Dockerfile`.

You may want to specify some or all of the following environment
variables with -e when you do a `docker run`:

`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`,
`AWS_DEFAULT_REGION`, `GITHUB_ORGS`, `GITHUB_TOKEN`, `S3_BACKUP_BUCKET`

The default command is to take a snapshot; use `snapshot-purger` as an
appended command on the docker run command line to run a purge instead.

You can also just pull the image with `docker pull
lsstsqre/github-snapshot`.