Backup/Restore RabbitMQ queues
Project description
This is a Python library and utilities for saving and loading RabbitMQ messages to and from disk.
Why?
A program cannot publish a message to RabbitMQ, but does not want to lose the message. Using this library, the program can save the message to a file. Later, after fixing the problem, a person can use a utility in this package to publish the message.
The consumer has a bug and messages are piling up in a queue. You have fixed the bug and your tests seem to indicate the consumer will now work, but you want some insurance. Using a program in this package, you can back up the queue to a file. If the program processes the messages incorrectly, you can use use a program in this package to restore the queue from the file.
You want to edit messages in a queue. You can dump the queue to a file, use a text editor or a one-off program to change the file, and then write the messages back into the queue.
Why not?
This library is brand new and has not been battle tested. Proceed with caution.
Backup format
Queues are backed up and restored from text files containing json. Each line in the file is a JSON dictionary with this structure:
{ "body": "This is a test", "delivery_info": { "delivery_tag": 1 }, "properties": { "app_id": null, "cluster_id": null, "content_encoding": null, "content_type": "text/plain", "correlation_id": null, "delivery_mode": 2, "expiration": null, "headers": { "client_id": null, "host_ip": "10.0.0.15", "host_name": "treebeard", "library_id": "olio_msg (Python) 1.9.0", "metadata_version": "1.0.0", "program_name": "olio_msg_send_test_messages", "version": "1.0.0" }, "message_id": "1df3eb23ffeb476b8355d87b475eb627", "priority": null, "reply_to": null, "timestamp": 1434644774, "type": "test", "user_id": null } }
The line was shown pretty-printed, but it’s actually just one line:
{"body": "This is a test", "delivery_info": {...}, "properties: {...}}
Using the command-line utilities
To backup a queue
rabbit_droppings --host localhost --queue jobs \ --file /path/to/save/file --dump
To backup and purge a queue
rabbit_droppings --host localhost --queue jobs \ --file /path/to/save/file --purge
To restore a queue
rabbit_droppings --host localhost --queue jobs \ --file /path/to/save/file --restore
Saving a message with the pika library
If you have a program using the pika library to publish messages to a RabbitMQ server, here’s how to save messages that could not be published:
First, import the library:
import rabbit_droppings
If you have a program that needs to save messages that cannot be published using the pika library, it should create a rabbit_droppings.Writer (probably in the constructor):
file = open('/path/to/my/file', 'a') self._rd_writer = rabbit_droppings.Writer(file)
When you publish a message and a pika exception occurs, save the message:
body = "Message body" properties = pika.spec.BasicProperties( content_type='text/plain' ) try: channel.basic_publish(exchange='', routing_key='some_queue_name', properties=properties, body=body) except (pika.exceptions.AMQPError, pika.exceptions.ChannelError) as e: pika_message = rabbit_droppings.PikaMessage(body, properties=properties) self._rd_writer.write(pika_message)
Versioning
This library practices Semantic Versioning.
This library is currently in alpha; it’s versions look like “0.1.0”, “0.2.0”, etc. There are no guarantees with alpha versions: Any version bump could be any combination of bug fix, backwards compatible API change, or breaking API change.
When the library becomes stable, its version number will be bumped to “1.0.0”. Semantic versioning makes these promises for stable versions:
A patch-level version bump (e.g. “1.0.0” to “1.0.1”) does not change the public API.
A minor-level version bump (e.g. “1.0.0” to “1.1.0”) changes the public API in a backward-compatible manner.
A major-level version bump (e.g. “1.0.0” to “2.0.0”) changes the public API in some way that is not backward compatible.
Python version
Known to work with Python versions:
2.6.9
2.7.9
Development
Running the tests requires a RabbitMQ server installed locally. The tests are known to pass with these RabbitMQ versions:
3.4.1
To run the tests:
./setup.py test
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.