django-vitess 0.0.2

Custom database driver for Vitess.io

django-vitess

Django-vitess repository is a direct copy from the Django support files of Vitess.io. This repo exists as a packaging source for the pypi package. The people of Vitess promised a long time ago to extract this driver as a separate project and publish it on pypi, but I decided to do it myself as I needed this package.

If someone from Vitess.io wants to publish this officially, please let me know and I will release the project from pypi

Usage

The following guide is written by Alkin Tezuysal and published at Vitess.io blog. I modified some paths and package names, otherwise it is repost here verbatim.

Django is a popular framework for Python application developers. It includes packages which make tasks like authorization and content administration easier. Django supports a number of databases including MySQL which makes it possible to run a Django application over Vitess without having to change the application code. Let’s take a look at how to combine the strengths of these two open source frameworks.

We built this example using Vitess operator. You can see the details of the implementation in the blog post Vitess Operator for Kubernetes.

Prerequisites

• Python environment locally (3.X)
• Kubernetes access (minikube, GKE)
• Support for Django ORM via Vitess

For this example, we’re using GKE with an existing Kubernetes cluster. You can also do this via minikube locally.

The Django example we are using is the weather app . We first launch the vitess operator using the provided configuration .

The following section includes these steps:

• Creating a Vitess Operator pod
• Building Vitess Cluster Components (1x primary tablet, 1x replica tablet, 3x etcd pods, 1x vtgate, 1x vtctld, 1x vitessbackup)
• Create the ‘weatherapp’ database schema and users.

$ kubectl apply -f operator.yaml
customresourcedefinition.apiextensions.k8s.io/etcdlockservers.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackups.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessbackupstorages.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitesscells.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessclusters.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitesskeyspaces.planetscale.com created
customresourcedefinition.apiextensions.k8s.io/vitessshards.planetscale.com created
serviceaccount/vitess-operator created
role.rbac.authorization.k8s.io/vitess-operator created
rolebinding.rbac.authorization.k8s.io/vitess-operator created
priorityclass.scheduling.k8s.io/vitess created
priorityclass.scheduling.k8s.io/vitess-operator-control-plane created
deployment.apps/vitess-operator created

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
vitess-operator-7f9c9d58f6-q5zlf   1/1     Running   0          20s

Initialize this cluster with a sample database called ‘weatherapp’ and user/password to access it will be embedded in the configuration file . We are basically creating a database which is analogous to a keyspace in Vitess.

$ kubectl apply -f 101_initial_cluster.yaml.django
$ kubectl get pods
NAME                                                 READY   STATUS      RESTARTS   AGE
example-90089e05-vitessbackupstorage-subcontroller   1/1     Running     0          94s
example-etcd-faf13de3-1                              1/1     Running     0          94s
example-etcd-faf13de3-2                              1/1     Running     0          94s
example-etcd-faf13de3-3                              1/1     Running     0          94s
example-vttablet-zone1-1542279354-edf1c7bf           2/3     Running     1          94s
example-vttablet-zone1-3763665199-476cbd65           2/3     Running     2          94s
example-weatherapp-x-x-vtbackup-init-75efaeeb        0/1     Completed   0          74s
example-zone1-vtctld-1d4dcad0-67bfd56b8b-4dr9s       1/1     Running     2          94s
example-zone1-vtgate-bc6cde92-59b88bc8d8-6wz86       1/1     Running     2          94s
vitess-operator-7f9c9d58f6-q5zlf                     1/1     Running     0          4m30s

As you can see this brings up a fully functional managed Vitess cluster with an unsharded keyspace consisting of one “Primary (Master)” and one “Replica”.

Step 1 - Set portforwards:

$ cat pf.sh ; ./pf.sh &
#!/bin/sh

kubectl port-forward --address localhost "$(kubectl get service --selector="planetscale.com/component=vtctld" -o name | head -n1)" 15000 15999 &
process_id1=$!
kubectl port-forward --address localhost "$(kubectl get service --selector="planetscale.com/component=vtgate,!planetscale.com/cell" -o name | head -n1)" 15306:3306 &
process_id2=$!
sleep 2
echo "You may point your browser to http://localhost:15000, use the following aliases as shortcuts:"
echo 'alias vtctlclient="vtctlclient -server=localhost:15999 -logtostderr"'
echo 'alias mysql="mysql -h 127.0.0.1 -P 15306 -u user"'
echo "Hit Ctrl-C to stop the port forwards"
wait $process_id1
wait $process_id2

Check Tablets:

$ vtctlclient ListAllTablets
Handling connection for 15999
zone1-1542279354 weatherapp - replica 10.100.1.75:15000 10.100.1.75:3306 [] <null>
zone1-3763665199 weatherapp - master 10.100.3.57:15000 10.100.3.57:3306 [] 2020-10-16T09:06:59Z

Step 2 - Verify database:

$ alias mysql="mysql -h 127.0.0.1 -P 15306 -u djangouser -p"
$ mysql
mysql: [Warning] Using a password on the command line interface can be insecure.
Handling connection for 15306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.9-Vitess MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+------------+
| Databases  |
+------------+
| weatherapp |
+------------+
1 row in set (0.16 sec)

Step 3 - Setup application environment

Now that we have set up the Vitess cluster with a MySQL backend, we can proceed to building the Django application. We will build a Django project using the django-admin command.

$ mkdir my_weather_app
$ cd my_weather_app
$ python3 -m venv env
$ . env/bin/activate
(env) askdba:my_weather_app askdba$
$ pip install django django-vitess
$ django-admin startproject weatherapp
$ cd weatherapp/
$ ls -la
total 8
drwxr-xr-x  4 askdba  staff  128 Oct 16 12:19 .
drwxr-xr-x  4 askdba  staff  128 Oct 16 12:18 ..
-rwxr-xr-x  1 askdba  staff  666 Oct 16 12:18 manage.py
drwxr-xr-x  7 askdba  staff  224 Oct 16 12:18 weatherapp

Edit configuration file and update following section.

$ vi weatherapp/settings.py
import os
# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'vitess',
        'OPTIONS': {
            'read_default_file': '/usr/local/mysql/my.cnf',
        },
    }
}

STATIC_ROOT = os.path.join(BASE_DIR, 'static')

...
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ['127.0.0.1']

# Application definition
...

The contents of mysql configuration:

$ vi /usr/local/mysql/my.cnf [link to sample my.cnf]
[client]
database = weatherapp
user = djangouser
password = ********
port = 15306
host = 127.0.0.1
default-character-set = utf8mb4

Step 4 - Install MySQL Client Connector

$ pip install mysqlclient
Collecting mysqlclient
  Using cached mysqlclient-2.0.1.tar.gz (87 kB)
Using legacy 'setup.py install' for mysqlclient, since package 'wheel' is not installed.
Installing collected packages: mysqlclient
    Running setup.py install for mysqlclient ... done
Successfully installed mysqlclient-2.0.1

Step 5 - Build Django Framework over Vitess cluster

At this stage, we’re ready to run the migrations to create initial Django metadata.

$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK

Step 6 - Create an admin user

Create an administrative user to access the Django Admin interface.

$ python manage.py createsuperuser
Username (leave blank to use 'askdba'): askdba
Email address: [e-mail]
Password:
Password (again):
The password is too similar to the email address.
This password is too short. It must contain at least 8 characters.
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.
(env) askdba:weatherapp askdba$

Step 7 - Start Django daemon.

$ python manage.py runserver 127.0.0.1:8000
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
October 16, 2020 - 09:37:02
Django version 3.1.2, using settings 'weatherapp.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Step 8 - Go to Django Admin page

Point your browser to http://127.0.0.1:8000/admin

You may continue to build the application from this point by following this example .

Conclusion

Vitess is a very powerful sharding framework that comes with a built-in control plane that allows backend developers to adapt their applications easily. Combining powerful application frameworks such as Django allows developers to create scalable applications out of the box with the power of open source tooling.