Mostly stateless microservice framework for generating on-the-fly thumbs and previews of a wide variety of file types.
Omni-Converter - omnic
Mostly stateless microservice for generating on-the-fly thumbs and previews of a wide variety of file types. Fully extendable to create any arbitrary conversion pipelines.
Omni Converter (which can be shortened to OmniC or Omnic) is free software, licensed under the GPL 3.0.
WIP WARNING: Omnic is still ‘unreleased software’, a work in progress. The API is subject to rapid change. I intend to release the first stable version before the end of this year (2017).
Installing directly on your host system
NOTE: Only Python 3.5+ is supported.
The first step is simply installing the Python package:
pip3 install omnic
You likely will want to install a few more python dependencies
pip3 install sanic pillow jinja2 requests
Depending on your needs, you will probably want to install a variety of other system-level package conversion programs. This might include:
- unoconv - for converting many document formats - Debian package: sudo apt-get install unoconv
- imagemagick - for raster image conversion and manipulation - Debian package: sudo apt-get install imagemagick
- inkscape - for vector conversion and manipulation - Debian package: sudo apt-get install inkscape
Additionally, converters specific to certain domains might be useful:
- meshlab - for converting between 3D model types - Debian package: sudo apt-get install meshlab
- jsc3d - for rendering 3D models in software - node package: npm install -g jsc3d
- obabel - for converting between chemical molecule filetypes - Debian package: sudo apt-get install obabel
TODO: Using Docker
The ideal situation would be to build a (bulky) docker image to support ALL built-in converters, a ready-to-use kitchen-sink of file conversion. This has not yet been done.
There are 3 principle ways to use Omnic
1. Commandline conversion and thumbnailing system
While OmniConverter was written for the web, the server components are optional. Thus it doubles as a handy “swiss-army knife” of file conversion and thumbnail generation.
- Example: Create thumb of an .doc file with: omnic convert input.doc thumb.jpg:200x200
2. Ready-to-use conversion and thumbnailing web-server
The most common usage of Omnic is as a microservice, supplying on-the-fly file format converter and preview or thumb generator,
- Running a server is as simple as omnic runserver
- Override values in the omnic/default_settings.py by specifying an OMNIC_SETTINGS to point to an import path to your custom settings module.
3. General purpose media conversion web-framework
Omnic is written in a very modular format, with a structure inspired partially by Django. This allows you to tailor-make your own converters, using it as a library, without forking. You can easily swap out any part, also.
- Create a new Python project as you normally would, including omnic in your requirements.txt.
- As with the above, override settings with a custom settings.py
- Write your own converters and include them in settings.py. No full documentation or scaffolding is available for this yet: Take a look at the omnic.builtin for examples on writing your own converters or services.
Launching the admin interface
Omnic comes bundled with a read-only admin interface. It’s main purpose is a sort of configuration sanity check, and queue monitoring, but it also serves as a great demo. Once installed, get the omnic server running:
Now point your browser at http://localhost:8080/admin/ for the admin interface.
From here you can paste in an URL to a resource, that Omnic will attempt to display as a thumbnail. In this example an OBJ file (3D model format) of a trumpet was pasted in, and a 200x200 thumbnail was generated:
Admin interface screenshot
To the right of the thumbnail it has an HTML snippet (the source-code of the thumbnail to the left), and a button that will take you to the conversion graph for that type:
Admin graph screenshot
Setting up a dev environment
- Install Python 3, including pip and venv:
- On Debian-based distros:
- sudo apt-get install python3 python3-env python3-pip
- On macOS, use something like brew
- On Debian-based distros:
- Create a virtualenv. For example:
- mkdir -p ~/.venvs/
- python3 -m venv ~/.venvs/omnic
- Activate virtualenv:
- source ~/.venvs/omnic/bin/activate
- You will need to do this any time you want to work
- Install dependencies:
- pip install -r requirements.txt
- Run test suite, should have 140+ tests pass:
- Start the server:
- ./bin/omnic runserver
If you want to test it without the admin interface, take a look at the following URLs.
To test it, try visiting something like:
The first time you visit it it will just be a single placeholder pixel. Subsequent times it should be 200x200 thumbnail
You might also be able to run this, if you have unoconv and ImageMagick (providing the convert command) installed:
This will convert the .doc into a PDF, then into a JPG thumbnail
The built-in converters interface with a variety of system binaries in order to provide rendering and conversion of many document, image, mesh. Adding new converters and rasterizers is simple, with relatively minimal code!
Omnic is not yet production ready, although you are welcome to try – I look forward to the pull requests!
The intended use is running as a microservice as part of a larger server infrastructure. This is to supplement or fully replace traditional work-queue based systems, such as using Celery. In a reasonable server topology, many omnic servers would sit behind a sticky load balancer (such as nginx), configured to “stick” based on the url GET component. In such a arrangement each omnic server would not need to be aware of its neighbors. The load balancer and/or an upstream proxy should also be configured to cache aggressively, to avoid Python serving static files (same philosophy to the whitenoise package).
The rationale for using omnic over a work-queue system:
- It is stateless with the exception of (disk-based) caching, and, technically, (in-memory) queueing although both are non-critical, as either getting cleared results in only a slower service, not a non-functioning service.
- The load-balancer topology proposed above would eliminate the need of servers to be away of siblings. This results in a much easier to understand topology, and a very light-weight dev environment
- Processing, network, and disk space are coupled, which would make it very cheap to run on AWS or DO (I intend to make the $5 nodes sufficient).
- Used Nekroze’ cookiecutter to start this package: https://github.com/Nekroze/cookiecutter-pypackage
TODO: Add full docs at http://omnic.rtfd.org.”“”