Skip to main content

Mostly stateless microservice framework for generating on-the-fly thumbs and previews of a wide variety of file types.

Project description

Omni-Converter - omnic

Travis CI

Travis CI

Mostly stateless microservice for generating on-the-fly thumbs and previews of a wide variety of file types. Fully extendable to create any arbitrary conversion pipelines.

Omni Converter (which can be shortened to OmniC or Omnic) is free software, licensed under the GPL 3.0.

WIP WARNING: Omnic is still ‘unreleased software’, a work in progress. The API is subject to rapid change. I intend to release the first stable version before the end of this year (2017).

Installation

Installing directly on your host system

NOTE: Only Python 3.5+ is supported.

The first step is simply installing the Python package:

pip3 install omnic

You likely will want to install a few more python dependencies

pip3 install sanic pillow jinja2 requests

Depending on your needs, you will probably want to install a variety of other system-level package conversion programs. This might include:

  • unoconv - for converting many document formats - Debian package: sudo apt-get install unoconv

  • imagemagick - for raster image conversion and manipulation - Debian package: sudo apt-get install imagemagick

  • inkscape - for vector conversion and manipulation - Debian package: sudo apt-get install inkscape

Additionally, converters specific to certain domains might be useful:

  • meshlab - for converting between 3D model types - Debian package: sudo apt-get install meshlab

  • jsc3d - for rendering 3D models in software - node package: npm install -g jsc3d

  • obabel - for converting between chemical molecule filetypes - Debian package: sudo apt-get install obabel

TODO: Using Docker

The ideal situation would be to build a (bulky) docker image to support ALL built-in converters, a ready-to-use kitchen-sink of file conversion. This has not yet been done.

Usage

There are 3 principle ways to use Omnic

1. Commandline conversion and thumbnailing system

While OmniConverter was written for the web, the server components are optional. Thus it doubles as a handy “swiss-army knife” of file conversion and thumbnail generation.

  • Example: Create thumb of an .doc file with: omnic convert input.doc thumb.jpg:200x200

2. Ready-to-use conversion and thumbnailing web-server

The most common usage of Omnic is as a microservice, supplying on-the-fly file format converter and preview or thumb generator,

  • Running a server is as simple as omnic runserver

  • Override values in the omnic/default_settings.py by specifying an OMNIC_SETTINGS to point to an import path to your custom settings module.

3. General purpose media conversion web-framework

Omnic is written in a very modular format, with a structure inspired partially by Django. This allows you to tailor-make your own converters, using it as a library, without forking. You can easily swap out any part, also.

  1. Create a new Python project as you normally would, including omnic in your requirements.txt.

  2. As with the above, override settings with a custom settings.py

  3. Write your own converters and include them in settings.py. No full documentation or scaffolding is available for this yet: Take a look at the omnic.builtin for examples on writing your own converters or services.

Launching the admin interface

Omnic comes bundled with a read-only admin interface. It’s main purpose is a sort of configuration sanity check, and queue monitoring, but it also serves as a great demo. Once installed, get the omnic server running:

omnic runserver

Now point your browser at http://localhost:8080/admin/ for the admin interface.

From here you can paste in an URL to a resource, that Omnic will attempt to display as a thumbnail. In this example an OBJ file (3D model format) of a trumpet was pasted in, and a 200x200 thumbnail was generated:

Admin interface screenshot

Admin interface screenshot

To the right of the thumbnail it has an HTML snippet (the source-code of the thumbnail to the left), and a button that will take you to the conversion graph for that type:

Admin graph screenshot

Admin graph screenshot

Contributing

Setting up a dev environment

  1. Install Python 3, including pip and venv:

    • On Debian-based distros:

      • sudo apt-get install python3 python3-env python3-pip

    • On macOS, use something like brew

  2. Create a virtualenv. For example:

    • mkdir -p ~/.venvs/

    • python3 -m venv ~/.venvs/omnic

  3. Activate virtualenv:

    • source ~/.venvs/omnic/bin/activate

    • You will need to do this any time you want to work

  4. Install dependencies:

    • pip install -r requirements.txt

  5. Run test suite, should have 140+ tests pass:

    • py.test

  6. Start the server:

    • ./bin/omnic runserver

Misc

Test routes

If you want to test it without the admin interface, take a look at the following URLs.

To test it, try visiting something like:

The first time you visit it it will just be a single placeholder pixel. Subsequent times it should be 200x200 thumbnail

You might also be able to run this, if you have unoconv and ImageMagick (providing the convert command) installed:

This will convert the .doc into a PDF, then into a JPG thumbnail

If you have jsc3d installed (a Node JavaScript based 3D model renderer), then the following should render a delightful trumpet:

Molecular visualization:

The built-in converters interface with a variety of system binaries in order to provide rendering and conversion of many document, image, mesh. Adding new converters and rasterizers is simple, with relatively minimal code!

Production

Omnic is not yet production ready, although you are welcome to try – I look forward to the pull requests!

The intended use is running as a microservice as part of a larger server infrastructure. This is to supplement or fully replace traditional work-queue based systems, such as using Celery. In a reasonable server topology, many omnic servers would sit behind a sticky load balancer (such as nginx), configured to “stick” based on the url GET component. In such a arrangement each omnic server would not need to be aware of its neighbors. The load balancer and/or an upstream proxy should also be configured to cache aggressively, to avoid Python serving static files (same philosophy to the whitenoise package).

The rationale for using omnic over a work-queue system:

  1. It is stateless with the exception of (disk-based) caching, and, technically, (in-memory) queueing although both are non-critical, as either getting cleared results in only a slower service, not a non-functioning service.

  2. The load-balancer topology proposed above would eliminate the need of servers to be away of siblings. This results in a much easier to understand topology, and a very light-weight dev environment

  3. Processing, network, and disk space are coupled, which would make it very cheap to run on AWS or DO (I intend to make the $5 nodes sufficient).

Misc

Documentation

TODO: Add full docs at http://omnic.rtfd.org.”””

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

omnic-0.1.2.tar.gz (33.8 kB view hashes)

Uploaded Source

Built Distribution

omnic-0.1.2-py2.py3-none-any.whl (10.0 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page