Mostly stateless microservice framework for generating on-the-fly thumbs and previews of a wide variety of file types.
Project description
Omni-Converter - omnic
Mostly stateless microservice for generating on-the-fly thumbs and previews of a wide variety of file types. Fully extendable to create any arbitrary conversion pipelines.
Omni Converter (which can be shortened to OmniC or Omnic) is free software, licensed under the GPL 3.0.
WIP WARNING: Omnic is still ‘unreleased software’, a work in progress. The API is subject to rapid change. I intend to release the first stable version before the end of this year (2017).
Installation
Installing directly on your host system
NOTE: Only Python 3.5+ is supported.
The first step is simply installing the Python package:
pip3 install omnic
You likely will want to install a few more python dependencies
pip3 install sanic pillow jinja2 requests
Depending on your needs, you will probably want to install a variety of other system-level package conversion programs. This might include:
unoconv - for converting many document formats - Debian package: sudo apt-get install unoconv
imagemagick - for raster image conversion and manipulation - Debian package: sudo apt-get install imagemagick
inkscape - for vector conversion and manipulation - Debian package: sudo apt-get install inkscape
Additionally, converters specific to certain domains might be useful:
meshlab - for converting between 3D model types - Debian package: sudo apt-get install meshlab
jsc3d - for rendering 3D models in software - node package: npm install -g jsc3d
obabel - for converting between chemical molecule filetypes - Debian package: sudo apt-get install obabel
TODO: Using Docker
The ideal situation would be to build a (bulky) docker image to support ALL built-in converters, a ready-to-use kitchen-sink of file conversion. This has not yet been done.
Usage
There are 3 principle ways to use Omnic
1. Commandline conversion and thumbnailing system
While OmniConverter was written for the web, the server components are optional. Thus it doubles as a handy “swiss-army knife” of file conversion and thumbnail generation.
Example: Create thumb of an .doc file with: omnic convert input.doc thumb.jpg:200x200
2. Ready-to-use conversion and thumbnailing web-server
The most common usage of Omnic is as a microservice, supplying on-the-fly file format converter and preview or thumb generator,
Running a server is as simple as omnic runserver
Override values in the omnic/default_settings.py by specifying an OMNIC_SETTINGS to point to an import path to your custom settings module.
3. General purpose media conversion web-framework
Omnic is written in a very modular format, with a structure inspired partially by Django. This allows you to tailor-make your own converters, using it as a library, without forking. You can easily swap out any part, also.
Create a new Python project as you normally would, including omnic in your requirements.txt.
As with the above, override settings with a custom settings.py
Write your own converters and include them in settings.py. No full documentation or scaffolding is available for this yet: Take a look at the omnic.builtin for examples on writing your own converters or services.
Launching the admin interface
Omnic comes bundled with a read-only admin interface. It’s main purpose is a sort of configuration sanity check, and queue monitoring, but it also serves as a great demo. Once installed, get the omnic server running:
omnic runserver
Now point your browser at http://localhost:8080/admin/ for the admin interface.
From here you can paste in an URL to a resource, that Omnic will attempt to display as a thumbnail. In this example an OBJ file (3D model format) of a trumpet was pasted in, and a 200x200 thumbnail was generated:
To the right of the thumbnail it has an HTML snippet (the source-code of the thumbnail to the left), and a button that will take you to the conversion graph for that type:
Contributing
Setting up a dev environment
Install Python 3, including pip and venv:
On Debian-based distros:
sudo apt-get install python3 python3-env python3-pip
On macOS, use something like brew
Create a virtualenv. For example:
mkdir -p ~/.venvs/
python3 -m venv ~/.venvs/omnic
Activate virtualenv:
source ~/.venvs/omnic/bin/activate
You will need to do this any time you want to work
Install dependencies:
pip install -r requirements.txt
Run test suite, should have 140+ tests pass:
py.test
Start the server:
./bin/omnic runserver
Misc
Test routes
If you want to test it without the admin interface, take a look at the following URLs.
To test it, try visiting something like:
The first time you visit it it will just be a single placeholder pixel. Subsequent times it should be 200x200 thumbnail
You might also be able to run this, if you have unoconv and ImageMagick (providing the convert command) installed:
This will convert the .doc into a PDF, then into a JPG thumbnail
If you have jsc3d installed (a Node JavaScript based 3D model renderer), then the following should render a delightful trumpet:
http://localhost:8080/media/thumb.jpg:200x200/?url=people.sc.fsu.edu/~jburkardt/data/obj/trumpet.obj
Molecular visualization:
The built-in converters interface with a variety of system binaries in order to provide rendering and conversion of many document, image, mesh. Adding new converters and rasterizers is simple, with relatively minimal code!
Production
Omnic is not yet production ready, although you are welcome to try – I look forward to the pull requests!
The intended use is running as a microservice as part of a larger server infrastructure. This is to supplement or fully replace traditional work-queue based systems, such as using Celery. In a reasonable server topology, many omnic servers would sit behind a sticky load balancer (such as nginx), configured to “stick” based on the url GET component. In such a arrangement each omnic server would not need to be aware of its neighbors. The load balancer and/or an upstream proxy should also be configured to cache aggressively, to avoid Python serving static files (same philosophy to the whitenoise package).
The rationale for using omnic over a work-queue system:
It is stateless with the exception of (disk-based) caching, and, technically, (in-memory) queueing although both are non-critical, as either getting cleared results in only a slower service, not a non-functioning service.
The load-balancer topology proposed above would eliminate the need of servers to be away of siblings. This results in a much easier to understand topology, and a very light-weight dev environment
Processing, network, and disk space are coupled, which would make it very cheap to run on AWS or DO (I intend to make the $5 nodes sufficient).
Misc
Used Nekroze’ cookiecutter to start this package: https://github.com/Nekroze/cookiecutter-pypackage
Documentation
TODO: Add full docs at http://omnic.rtfd.org.”””
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for omnic-0.1.4-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0477b30fc0ac522387bdc078f9c20b1f51546ef270f871191a4326bf3fcbb913 |
|
MD5 | 60c22e8f868f552b65592f1cb24d5bb3 |
|
BLAKE2b-256 | 04d35a3c9dc814eb10aa73919f27d86ef3d8cf19314d4a9a4c98347386f89330 |