Simple process supervision agnostic utility for graceful reloading of services
Simple utility for graceful reloading of services.
General usage bases on my other project ianitor. hupwatch is a simple shell command that wraps process and can be simply used in your existing process/service supervision tool like supervisord, circus, runit etc.
Reloading of web application code gracefully is a today’s must. Every DevOps should tell you that already. Many of Python web servers allow such scenario but handle this differently. Usually you end up with only few reasonable options:
Graceful reloading implementations provided by gunicorn and uWSGI are both neat but fail in reality when you try to automate things or use it with any of the popular tools for process/service supervision:
You may of course work around these issues by providing some custom integration for whatever process supervision tool you are using or do some crazy next/current instance switching and mantain doubled configuration of services only for this single purpose. This will make your solution either non-portable to other tools or make the whole solution harder to automate in a reliable way.
hupwatch provides a single solution that can be easily integrated with virtually any supervision tool and allows to reload whole web servers with only single command that is available on any POSIX system. It is kill:
# on terminal or inside supervisord config: $ hupwatch -- gunicorn myapp:application --bind unix:/tmp/myapp.sock => HUP watch [INFO ]: Starting HUP watch (92808) => HUP watch [INFO ]: Child process 92809 started => HUP watch [INFO ]: Pausing for signal [2016-01-27 17:24:13 +0100]  [INFO] Starting gunicorn 19.4.5 [2016-01-27 17:24:13 +0100]  [INFO] Listening at: unix:/tmp/myapp.sock (92809) [2016-01-27 17:24:13 +0100]  [INFO] Using worker: sync [2016-01-27 17:24:13 +0100]  [INFO] Booting worker with pid: 92812 # issued on any other terminal: kill -HUP <hupwatch_pid> # continued result in the hupwatch stdout: [...] => HUP watch [DEBUG ]: HUP: >>> => HUP watch [DEBUG ]: HUP: Waiting for process (92955) to warm up => HUP watch [DEBUG ]: HUP: Sending SIGTERM to old process (92809) => HUP watch [DEBUG ]: HUP: Waiting for process (92809) to quit... [2016-01-27 17:24:46 +0100]  [INFO] Handling signal: term [2016-01-27 17:24:46 +0100]  [INFO] Starting gunicorn 19.4.5 [2016-01-27 17:24:46 +0100]  [INFO] Listening at: unix:/tmp/myapp.sock (92955) [2016-01-27 17:24:46 +0100]  [INFO] Using worker: sync [2016-01-27 17:24:46 +0100]  [INFO] Booting worker with pid: 92964 [2016-01-27 17:24:58 +0100]  [INFO] Worker exiting (pid: 92812) [2016-01-27 17:24:58 +0100]  [INFO] Shutting down: Master => HUP watch [DEBUG ]: CHLD: >>> => HUP watch [INFO ]: CHLD: Child process quit => HUP watch [DEBUG ]: CHLD: <<< => HUP watch [INFO ]: HUP: Old process quit with code: 0 => HUP watch [DEBUG ]: HUP: <<< => HUP watch [INFO ]: Pausing for signal
hupwatch will start anything that you have provided after the -- characters as its own subprocess (using subprocess.Popen()) and listens for incoming Unix signals. Whenever it gets a HUP signal it starts a new process with the same arguments and sends TERM signal to the process that was started previously so it can shutdown gracefuly.
This make the whole realoading process very easy to automate. No need to execute multiple commands and mantain any state between them. Simply HUP’n’go! This is a good news for fabric enthusiasts - don’t need to worry about lost shh connection during whole reload procedure because it takes only one step (at least if you use symlinks). There is nothing to interrupt!
Rolling back the update is also painless: simply change project symlink and issue another HUP signal to the same hupwatch pid. Auto rollback should be also easy to implement and we are open to any contributions!
If you paid attention then you should already notice that this requires only two things to make it working as a solution for graceful reload:
See the usage with hupwatch --help for more information on possible configuration options:
There is also some details important detail of handling failures and what to do when hupwatch receives other signals (e.g. KILL, TERM, INT). By default it assumes that you want to have have your process working no matter what happens with the parent (hupwatch). So in case of failure it leaves it as it is - spawned process will become a child of init. If you that this happened you can clean up the mess manually without interrupting the process of serving web requests. This behaviour can be changed with --kill-at-exit flag.
This is more a proof of concept than a battle-tested tool. Anyway, there are only few lines of code that actually do any work. Most of the code in this package is extensive logging and parsing of arguments. This state of this package will eventually change in a near future, because it solves a real problem that we have in my organization. So give it a try at least in your staging/testing environment.
Contributions are really welcome!
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|File Name & Checksum SHA256 Checksum Help||Version||File Type||Upload Date|
|hupwatch-0.0.3-py2-none-any.whl (13.2 kB) Copy SHA256 Checksum SHA256||2.7||Wheel||Jan 27, 2016|
|hupwatch-0.0.3.tar.gz (9.8 kB) Copy SHA256 Checksum SHA256||–||Source||Jan 27, 2016|