luh3417 0.1.6

A WordPress backup/restore/workflow tool

LUH3417

This is a tool to help you implement a WordPress development workflow. It has 3 main features:

• Snapshot — Take snapshots of a running WordPress instance
• Restore — Restore those snapshots in-place or to a different location
• Transfer — Transfer one instance over another using automated backup, validation and configuration rules

Everything can happen seamlessly in local or through SSH, allowing you to work easily on remote servers from your local machine and to transfer instances from one server to another.

Thanks to this, putting your code to production is as simple as:

python -m luh3417.transfer -g my_project.py local production

While the snapshot and restore operations can be used individually, it is not recommended to use them as the main tools. Indeed, restore can easily override an instance without any previous backup. For this reason, it is better to use transfer whenever possible. It will ensure your safety within the workflow that you defined.

Installation

pip install luh3417

Usage

LUH3417 is made to use with Python's -m option. This way, if you want to invoke the snapshot feature, the base command will be python -m luh3417.snapshot.

If you prefer, there is also equivalent commands installed in the bin directory, namely luh3417_snapshot, luh3417_restore and luh3417_transfer.

All the locations can be in two formats:

• SSH — user@server:/location/on/server
• Local — /location/on/current/machine

This allows you to transfer data between remote servers and local machine quite seamlessly.

NOTE — You need to use an SSH agent in order for all the features to work. No password prompt will show up. Usually it's as simple as to type ssh-add in your terminal once during your session.

snapshot

Creates a snapshot of a running WordPress instance. A snapshot is an archive containing:

• All PHP/theme/media/etc files
• A DB dump
• Meta information about how the snapshot was taken

Usage syntax:

python -m luh3417.snapshot [-h] [-n SNAPSHOT_BASE_NAME] [-t FILE_NAME_TEMPLATE] source backup_dir

Example:

python -m luh3417.snapshot [e-mail]:/var/www/html [e-mail]:/var/backups/wp

Additional options:

• -n/--snapshot-base-name — Base name for your snapshot file. See the --file-name-template option to see how this name is used. The default name is the database's name.
• -t/--file-name-template — This template will be used to generate the snapshot file name. By default it is {base}_{time}.tar.gz but you can put whatever you want. {base} and {time} will be replaced respectively by the base name (see --snapshot-base-name) and the ISO 8601 UTC date. Independently of the name, the file will be placed in the backup_dir.

restore

Restores a snapshot either in-place to its original location using the embedded meta-data or to another location using a patch on the meta-data.

In addition to just restoring the files and database, the patch can trigger changes in wp-settings.php, replace values in the database and much more.

restore will essentially override an instance with the content of a backup, so make sure to use it wisely in order not to loose data. Also, see transfer.

Usage:

python -m luh3417.restore [-p PATCH] [-a ALLOW_IN_PLACE] snapshot

Options:

• -p/--patch — Location to the patch file (see below)
• -a/--allow-in-place — Allows restoring the backup onto its original location. This flag is required because otherwise it would be way too easy to override

Restore in-place

If you want to restore a backup to its original location, you just need to know the file's location and pass the -a flag.

python -m luh3417.restore -a [e-mail]:/path/to/snapshot.tar.gz

NOTE — If the snapshot was made locally, it will always be restored locally because there is no way for LUH3417 to know the originating server so it assumes that the snapshot file was not transferred to another machine.

Restore to another location

In order to restore to another location, you need to use a patch file

python -m luh3417.restore -p patch.json [e-mail]:/path/to/snapshot.tar.gz

Here is an example of patch file:

{
    "args": {
        "source": "[e-mail]:/var/www/html"
    },
    "owner": "www-data:"
}

See below for detailed documentation of patch content

args.source

Set this one to define where to restore the archive.

{
    "args": {
        "source": "[e-mail]:/var/www/html"
    }
}

wp_config

Database configuration from the WordPress

{
    "wp_config": {
        "db_host": "localhost",
        "db_name": "xxx",
        "db_user": "xxx",
        "db_password": "xxx"
    }
}

NOTE — You need to make sure you match those values in php_define unless you're using transfer which sets them automatically

owner

This changes the owner of the files to another one. This only works if:

• When restoring locally, you run as root
• When restoring remotely, you login in as root

{
    "owner": "www-data:"
}

git

Replaces some directories with a Git repository at a given version

{
    "git": [
        {
            "location": "wp-content/themes/jupiter-child",
            "repo": "[e-mail]:your_company/jupiter_child.git",
            "version": "master"
        }
    ]
}

NOTE — .git directories are excluded from snapshots, so unless you specify this option there will be no git-enabled directories in the restored files. On the other hand, git repositories will be created at specified version, so it might not make sense to specify this option when restoring a backup in-place.

setup_queries

A list of SQL queries to be run after the DB was restored

{
    "setup_queries": [
        "delete from wp_options where option_name = 'gtm4wp-options';"
    ]
}

php_define

Values to be changed or added in wp-config.php. Any JSON-serializable value can be used.

{
    "php_define": {
        "WP_CACHE": false,
        "WP_SENTRY_ENV": "new-env"
    }
}

replace_in_dump

A list of strings with their replacement to be changed in the dump before restoring it. This is mainly used to change the domain name of the instance. As WordPress serializes its settings, a simple replace is not possible. This will use a holistic heuristic which will try to keep PHP-serialized values correct even if quoted in a MySQL string.

NOTE — PHP-serialized values are prefixed by their length, this is why a simple replace cannot be effective: if the length changes then the whole value gets corrupted.

{
    "replace_in_dump": [
        {
            "search": "https://old-domain.com",
            "replace": "https://new-domain.com"
        }
    ]
}

mysql_root

In order to create the database and set the user password, the script needs a root access to MySQL. Today, the only supported method is socket, because it is password-less. However it only works when the server is local and properly configured (it's the default behavior in Debian-based distros).

{
    "mysql_root": {
        "method": "socket",
        "options": {
            "sudo_user": "root",
            "mysql_user": "root"
        }
    }
}

About the options:

• sudo_user — don't set it if you don't need to sudo to use the socket, set it to root or whichever user is right otherwise.
• mysql_user — name of the MySQL user to use

outer_files

Creates files on the server's file system. If the file name is relative then the file is created relatively to the WordPress's root, otherwise it is created at the specified absolute location.

{
    "outer_files": [
        {
            "name": "robots.txt",
            "content": "User-agent: *\nDisallow: /\n"
        },
        {
            "name": "/etc/apache2/sites-available/my-host.conf",
            "content": "<VirtualHost> ..."
        }
    ]
}

NOTE — There is not (yet) any form of privilege escalation to create those files, so the local/remote user must have the rights to create those files.

post_install

Those are shell scripts which run on the host server after the install is complete. Typically, you can enable your virtual host and reload Apache.

{
    "post_install": [
        "a2ensite my-website.com",
        "systemctl reload apache2"
    ]
}

dns

You might want to use your DNS provider's API in order to configure the domain that is going to target your website. LUH3417 integrates with libcloud in order to provide an abstraction over the most popular cloud providers.

Here is an example entry:

{
    "dns": {
        "providers": [
            {
                "domain": "my-corp.net",
                "provider": "digitalocean",
                "credentials": {
                    "key": "xxxxxx",
                }
            }
        ],
        "entries": [
            {
                "type": "alias",
                "params": {
                    "domain": "my-wp.my-corp.net",
                    "target": "load-balancer.my-corp.net"
                }
            },
            {
                "type": "ips",
                "params": {
                    "domain": "dns.my-corp.net",
                    "ips": [
                        "2606:4700:4700::1111",
                        "2606:4700:4700::1001",
                        "1.1.1.1",
                        "1.0.0.1"
                    ]
                }
            }
        ]
    }
}

Let's break this down

providers

That's a list of the providers, associated to a domain name. The different keys are used like this:

• domain — root domain name managed by this provider
• provider — domain name provider (you can get the list here, use the lower-case string value)
• credentials — kwargs to be passed to the constructor of the provider

entries

Entries are either a single CNAME either a set of A/AAAA records for a same domain name. LUH3417 will make sure that all records for this (sub-)domain match your specification and will delete other records for that sub-domain.

Suppose the following situation:

• foo.my.org resolves to A 1.2.3.4
• But you want it to be a CNAME of bar.my.org
• The A 1.2.3.4 entry will be deleted and a CNAME bar.my.org will be created

Now, let's dig into the options

"type" = "alias"

That's when you want to create a CNAME.

{
    "type": "alias",
    "params": {
        "domain": "my-wp.my-corp.net",
        "target": "load-balancer.my-corp.net"
    }
}

The two params are:

• domain — target (sub-)domain
• target — target of the CNAME (aka the value of the record)

"type" = "ips"

This will set your (sub-)domain to point on a set if IP addresses, preferably v6 but legacy systems like v4 are still supported.

{
    "type": "ips",
    "params": {
        "domain": "dns.my-corp.net",
        "ips": [
            "2606:4700:4700::1111",
            "2606:4700:4700::1001",
            "1.1.1.1",
            "1.0.0.1"
        ]
    }
}

• domain — is the target (sub-)domain
• ips — is a list of IP address that will be set to AAAA and A records

transfer

The main goal of this package is to allow the setup of a custom workflow that allows easy copy of WordPress instances from an environment from the other.

The basic idea is the following:

• You can specify an origin and target environment names
• There is a settings generator Python file which will generate all the settings and patches appropriate for this transfer.

It's your responsibility to write an settings generator, however there is an a documented example attached in this repository.

Usage:

python -m luh3417.transfer [-h] -g SETTINGS_GENERATOR origin target

Example:

python -m luh3417.transfer -g example/generator.py develop local

To see the content of the generator file, please refer to the example/generator.py file and especially the allow_transfer() method's documentation which will explain the spirit of the file.

replace

Seeks and replaces serialized values. Values could be in quoted MySQL literals or serialized PHP values, they will be replaced and the containing string will be re-serialized.

That's a complicated problem due to PHP serialized format which prefixes the string's length to the string. If you want to replace a value you need to replace its length.

By example:

echo serialize("foo");
// s:3:"foo";

echo serialize("long foo");
// s:8:"long foo";
//   ^-- see here how the prefix changed from 3 to 8

This command will do the replacing in a file and output to another file. Be careful, it will not warn you of overwrites.

Usage:

python -m luh3417.replace [-h] -i INPUT -o OUTPUT [-b BEFORE [BEFORE ...]] [-a AFTER [AFTER ...]] [-c CHARSET]

Example:

python -m luh3417.replace \
    -i dump.sql \
    -o dump_replaced.sql \
    -b old_domain.com \
    -a new_domain.com

FAQ

Why the name LUH3417?

It's a character from THX1138. The author is not particularly fan of this movie, however it expresses quite well the feeling of working with WordPress and especially setting up a professional workflow.

Why using Python to code it?

It felt to the author that this language was more appropriate for this task than PHP.

Do I need to write Python to use the transfer feature?

Yes, fortunately it's pretty easy. The author started with Dive Into Python.

Why can't the transfer feature have a configuration file instead?

A configuration file would mean imposing the skeleton of the author's workflow onto all users. If such a workflow is suitable for your needs, example code and tutorial are provided so just have to adapt the code for yourself.

License

This project is distributed under the terms of the WTFPL. It comes void of warranties and if you break things it's on you.