Small web app for printing Colony-based documents.

This project includes two main components:

• The Web App end-point that provides XML to Binie conversion colony_print.controllers
• The structure conversion infra-structure (Visitors, AST, etc.) colony_print.printing

Features

• Cloud printing, with minimal configuration
• Multiple engine support (npcolony, gravo, text)
• XMPL to Binie conversion
• XMPL printing, with the documents converted into Binie by the nodes
• PDF generation with custom fonts and images
• Fonts sent with the print jobs, installed on demand by the nodes
• Node capabilities, advertised by the nodes and verified by the server
• GDI printing (Windows) via Colony NPAPI (npcolony)
• CUPS printing (Linux) via Colony NPAPI (npcolony)
• Windows installer for the nodes, which update themselves from PyPI (see Windows Node)
• Restart and update of the nodes from the admin UI and from the API (see Node Control)

Binie Specification

For a detailed understanding of the Binie file format used in this project, refer to the Binie File Format Specification. This document outlines the structure and organization of the Binie file format, which is essential for developing compatible applications and tools.

XMPL Specification

The XML Markup Language for Printing (XMPL) is integral to our document processing pipeline. For an in-depth understanding of the XMPL structure and its seamless convertibility to Binie, see the XMPL File Format Specification.

Node Capabilities

The features supported by each node (e.g. xmpl or dynamic-fonts) are advertised by the node as capabilities, and the requests that need a capability the node doesn't advertise are refused. The complete list of capabilities is described in Node Capabilities.

Installation

Pre-requisites

apt-get install gcc python-dev
pip install --upgrade appier netius pillow reportlab

Run Server

pip install colony_print
python -m colony_print.main

Run Node

pip install colony_print
BASE_URL=$BASE_URL \
SECRET_KEY=$SECRET_KEY \
NODE_ID=$NODE_ID \
NODE_NAME=$NODE_NAME \
NODE_LOCATION=$NODE_LOCATION \
python -m colony_print.node

The node may also be run by its boot, which loads the configuration of the node from a config.env file (the values of the environment take precedence), updates colony-print and npcolony from PyPI (see Self-Update) and then runs the node, as the service of the Windows nodes does. It's the way to run the nodes that are updated from the admin (see Node Control), unless the boot is told to skip the update (--no-update):

pip install colony_print
python -m colony_print.boot --config config.env

The fonts installed on demand by the node (see Print Fonts) are kept in the FONTS_PATH directory (defaults to ~/.colony_print/fonts, and to the fonts directory of the data directory on the Windows nodes, see Windows Node), with each font file limited to FONT_MAX_SIZE bytes (defaults to 16 MB).

Fonts

To be able to use new fonts (other than the ones provided by the system), one must install them into the /usr/share/fonts/truetype directory so they are exposed and ready to be used by the PDF generation infra-structure. For example, Calibri is one type of font that should be exported to a UNIX machine as many colony-generated documents use it.

The /usr/share/fonts/truetype install path is shared by the PDF generation engine. Linux (CUPS) nodes need the same fonts to lay out the Binie documents they print. They look for the font files by name in the same paths and, as Windows does, fall back to the closest installed font found through fontconfig (fc-match), so installing Calibri (or the metric compatible Carlito font) keeps the layout identical to the Windows one. The same applies to the barcode fonts of the documents (e.g. the 2 of 5 font of the Omni product labels, looked up as 2 of 5.ttf), whose barcodes are otherwise printed as the letters they are encoded with. Binie and XMPL documents may also send their fonts with the print job (see Print Fonts), so that nodes with the dynamic-fonts capability install them on demand, on Windows and Linux alike, without any manual install. The gravo engine receives its fonts on a per print job basis through the extra_fonts field of the gravo print payload (see Gravo Print Payload) and stages them on a per job temporary directory, so the two flows are independent and operators should not confuse them.

Engines

There are currently three engines available for printing in Colony Print:

• npcolony - The Colony NPAPI engine, which is used for GDI printing on Windows and CUPS printing on Linux.
• gravo - Which allows engraving of text and signatures using Gravo Pilot. Accepts an extra_fonts mapping in the print payload to ship .f3s font payloads to the engraving software on a per print job basis (see Gravo Print Payload).
• text - A simple virtual printer text engine that prints text to a simple plain text file and returns the file.

Print Request

Every engine is reached through the same print endpoint and request envelope. A job is submitted to /nodes/<id>/print (or /nodes/<id>/printers/<printer>/print to target a specific printer) with the following fields:

* Exactly one of data or data_b64 must be provided.

The options map is filtered to the following keys:

The save_output and email_* options only take effect on nodes running in email mode (NODE_MODE=email).

The requests that use a feature that requires a capability the node doesn't advertise (e.g. the xmpl format) fail with 409, except the fonts of the print jobs, that are skipped for the nodes that don't support them (see Print Fonts), and the depth of the gravo print jobs, that is skipped by the nodes without the gravo-depth capability (see Gravo Print Payload), see Node Capabilities.

npcolony Print Payload

The npcolony engine is the default and prints through Colony NPAPI using GDI on Windows and CUPS on Linux. Its payload is the binary print document carried in data_b64, typically a Binie document produced by the XMPL to Binie conversion, dispatched directly to the target printer. There are no JSON fields: the printing behaviour is tuned through the options and the optional format field described in Print Request.

On Windows the Binie document is drawn directly through GDI. Linux (CUPS) nodes only print PDF documents, so they convert Binie jobs (with the binie format, or without a format when the payload is a valid Binie document) into a PDF laid out with the same rules as GDI: the paper size of the document when it defines one and the printer accepts it as a custom paper size (as the Windows driver of the printer does), and the printer's default paper size otherwise (e.g. a label printed on an A4 printer comes out at its real size in the top left corner of the page), with the content laid out from the top left corner of the printable area of the page and printed without scaling. The media option doesn't apply to them, as their pages are always laid out for that paper size. PDF documents and any other data are sent to CUPS untouched.

XMPL documents (with the xmpl format) are converted into Binie documents by the nodes with the xmpl capability, and then printed as any other Binie document. The document is verified (converted) by the server when the job is submitted, so that an invalid document fails with 400, and the fonts declared by its font elements (see XMPL Specification) are installed together with the ones of the fonts field. Only inline images (source) are accepted, the documents with image paths (path) are refused by the server and by the nodes, as the paths would be read from the file system of the node.

Linux (CUPS) Printing

The printer of the job (or NODE_PRINTER when the job has none) selects the CUPS queue, and default, the default value of NODE_PRINTER, selects the default queue (or the only queue, when none is the default). Jobs for a queue that does not exist, or that CUPS refuses, fail with an error instead of being reported as printed.

Each queue should use a driver for its printer and a default paper size that matches the loaded paper, as that size is used for the Binie documents that do not define one, or whose size the printer does not accept as a custom paper size (e.g. lpadmin -p receipt -o PageSize=RP80x297):

• Receipt printers - the vendor CUPS driver (e.g. the Epson TM series driver), with its paper reduction options enabled to avoid feeding blank paper at the end of the receipt.
• Label printers - the label drivers shipped with CUPS (Zebra, Dymo) or the vendor ones, with the default size set to the loaded label.
• Office printers - driverless (IPP Everywhere) queues.

The custom paper sizes a printer accepts, and their margins, are the ones of the PPD of its queue, as reported by npcolony. With npcolony versions that don't report them, a Binie document only uses its own size when it matches the default paper size of the queue.

In email mode the PDF document is written to the output file (print to file) instead of being printed, as it happens with the PDF printer on Windows.

Print Fonts

The fonts of a Binie or XMPL document may be sent in the fonts field of the print request, so that the nodes with the dynamic-fonts capability install them on demand before printing the document. The fonts (and the XMPL documents) are only accepted for the jobs of the npcolony type, as the other engines don't use them. Each entry of the fonts array accepts the following fields:

* Exactly one of data_b64 or url (with an optional md5 that is verified against the font file), or md5 alone (a reference to a font installed on the node).

• Only TrueType fonts are supported (single fonts with TrueType outlines), OpenType fonts with PostScript (CFF) outlines and font collections are refused.
• The nodes keep the installed fonts in a cache that never expires, by the MD5 of their files, and download the font of a URL only once (URLs are considered immutable, so a changed font must be published at a new URL).
• The installed fonts are available to every later job of the node, as the fonts installed in the system are, and they are used before the system ones. When several files of the same family and style are installed, the most recently installed (or referenced) one is used.
• A font that can't be installed (download failure, MD5 not installed or not matching, not TrueType, license that doesn't allow embedding the font, name or style not matching the font file, larger than FONT_MAX_SIZE) fails the job with an error that names it on the nodes with the dynamic-fonts capability, the fonts are never silently replaced by other fonts on those nodes.
• The fonts may always be sent, whatever the node: the nodes that don't support them (without the dynamic-fonts capability, e.g. older nodes) print the documents with their own fonts, the fonts of the job being marked as skipped (fonts_skipped in the information of the job). The server doesn't send the fonts to the nodes it knows don't support them (removing the font elements of their XMPL documents too), sending them to the unknown nodes (e.g. not registered since the server restarted), and the nodes skip the fonts they don't support, reporting it in the result of the job, except the 0.23.0 nodes, that fail the job. The fonts endpoints and the jobs of the fonts type still require the capability.
• On Linux (CUPS) the fonts are embedded in the PDF document, on Windows they're loaded in the system for the node process only, so no administration rights are required.
• On Windows the font files are parsed by the font engine of the system (in kernel mode before Windows 10, eg: Windows XP), so fonts should only be sent by trusted clients, as the admin token they require already implies.

The fonts installed on a node are listed by GET /nodes/<id>/fonts, and may be installed before any job (so that later jobs reference them by md5 alone or not at all) by POST /nodes/<id>/fonts, with the same fonts array (data_b64 or url entries), which queues a job of the fonts type whose result lists the installed fonts. The same job may also be sent to the print endpoints, with the fonts type and a {"fonts": [...]} JSON payload with the same entries.

The server side conversion of XMPL documents into PDF (/documents.pdf) also uses the fonts declared by the documents, installed in the font cache of the server (in FONTS_PATH, defaulting to the fonts directory of DATA_PATH), which requires the admin token.

Gravo Print Payload

The gravo engine accepts a JSON payload submitted as base64 to the print endpoint. The accepted fields are documented below:

Text Print Payload

The text engine is a virtual printer that does not talk to any physical device. Its payload is the plain text content carried in data_b64; the engine writes it to a document.txt file and returns that file (base64 encoded) in the job result. It has no JSON fields and ignores the printer, format and option values, which makes it convenient for testing and for capturing print output without hardware.

Node Control

The nodes are restarted and updated from the admin UI (the node view) and from the API, without a session on their machines. The actions reach the node as jobs without data, through the queue of jobs it already polls, so they're listed with the other jobs and may be cancelled while queued:

• Each endpoint requires the admin token and the capability of the node with the same name (see Node Capabilities), failing with 409 when the node doesn't advertise it, as the older nodes and the ones that never registered.
• A node with a restart or update job queued or in flight doesn't get another one, the request returns that job. An update requested while a restart is pending is then not run, as told by the type of the returned job, and must be requested again once the node is back.
• The node only restarts once the remaining jobs it has received are printed and their results posted, so no print is interrupted. The jobs queued meanwhile stay in the server until the node is back.
• A restart or update job is finished by the server when the node registers itself again with another start time (the one of its new process), with the version and the libraries of the node before and after in its result. Until then the job stays in printing, so a node that didn't come back is visible. An update job finishes as an error when the packages could not be updated, with the node running the installed ones.
• The commands carry no packages, versions or package index: what gets installed is the one of the configuration of the node (NODE_VERSION, NODE_NPCOLONY_VERSION and NODE_INDEX_URL, see Self-Update), so the server only triggers what the node would do by itself.
• These jobs are left out of the print statistics of the node and can't be duplicated.

What a node supports depends on how it's run, as a running node can't update itself (the update is a restart whose boot updates the packages):

On Windows, a node only restarts by default when it's run by WinSW or by the service of the Windows XP installer of this version, any other one needs NODE_RESTART set, as explained below.

The remote control of a node is configured in its configuration (config.env or the environment):

• exit - the process ends with an error exit code (75) and its service starts it again. It's the default for the nodes run by WinSW, whose recovery actions restart the service after 10, 30 and then 60 seconds (for the restarts within one hour), Windows logging it as a failure of the service, and for the Windows XP nodes of the installer of this version, that NSSM starts again after 10 seconds.
• exec - the node runs its own command line again, with the environment it was started with, so that a changed config.env applies. It's the default for every other node, except on Windows. On Linux the process is replaced and keeps its PID, so it works for the nodes started by hand and under any supervisor (e.g. systemd or a container). On Windows a process can't be replaced, so a new one is started and the current one exits, which runs the node twice when something else also starts it again (e.g. a script loop or another service wrapper, the case NODE_RESTART=exit is for). For that reason a Windows node that is not run by one of those services has no default: it's only restarted from the admin with NODE_RESTART set (to exec for a node started by hand).

The values set from the admin are kept in a state.env file next to config.env, written by the node and applied by the boot over the configuration, as the node never writes config.env (which holds the secret key): the auto-update (NODE_UPDATE, taking precedence over the one of the configuration) and the update forced for the next start (NODE_UPDATE_ONCE, removed by the boot once read). The node reports its start time (start_time) and the outcome of the update run by its boot (update, with auto, status, time and error) in its information.

Any admin token is able to restart and update the nodes, as it's able to print on them.

Windows Node

Windows 10 and 11 (64 bit) nodes are installed with a setup.exe installer. It installs everything the node needs: an embedded Python, colony-print, Colony NPAPI (npcolony) for the native (GDI) printing, and their dependencies. The node runs as the colony-print-node Windows service, which starts with the machine and updates itself from PyPI every time it starts. The older versions of Windows (from Windows XP on) and the 32 bit ones have their own installer (see Windows XP).

Building the Installer

.\windows\build.ps1 -Python C:\Python314\python.exe

The build requires a 64 bit Python (the embedded Python is the same version), whose version must have the digest of its embedded distribution pinned in build.ps1 (currently 3.14.8, the digest of another version is published by python.org), and Inno Setup 6, as npcolony and the other dependencies are installed from their PyPI wheels (nothing is compiled). It creates the installer (dist\colony-print-node-setup-<version>.exe), which bundles the packages, so that it installs the node without internet access. The Windows Workflow builds it on every push (the colony-print-node-windows artifact) and smoke tests the installer on a Windows runner. When a release is published, it also attaches the installer to the release, for which the tag of the release must match the version in setup.py (e.g. 0.21.0). A failed attach can be retried by running the workflow manually with the tag of the release.

Installing

The installer asks for the server URL and the secret key, which it verifies against the server. It then asks for the mode (normal or email), the node name, location and printer, and, in email mode, the email receivers and the Mailme key. The node ID is derived from the name and kept on later installs. It may also run silently (e.g. for mass deployment), with the configuration given as parameters:

colony-print-node-setup-0.20.0.exe /VERYSILENT /URL=https://print.example.com/ /KEY=$SECRET_KEY /NAME="Shop 1" /LOCATION=Porto /PRINTER="EPSON TM-T20II Receipt"

The parameters that aren't given keep the values of the existing configuration, so running a new installer over a node only replaces its files. The installer exits with code 10 when the service could not be installed or started.

The service runs under the system account, so it only sees the printers installed for all users and has no default printer. The installer suggests the default printer of the user running it, and the printer should be set, otherwise the jobs that don't select one fail. In email mode the printer must be a PDF printer (e.g. Microsoft Print to PDF), as the jobs are printed to PDF files.

The service also prints with the settings of the printer that apply to all users (Printer properties, Advanced, Printing Defaults), and not with the Printing Preferences of a user, as Windows keeps them per user. A node that used to run under a user account (e.g. from a script) may print with other settings once it's installed as a service (e.g. the label size, the orientation or the offsets of a label printer, moving the content of the labels), in which case the values of the Printing Preferences of that user should be copied to the Printing Defaults of the printer.

The node is installed in C:\Program Files\Colony Print Node (other directories are refused, as the service runs its files as the system account). Its configuration (config.env), logs and fonts installed on demand (fonts) are in C:\ProgramData\Colony Print Node, which only the system account and the administrators can access, as it holds the secret key. A data directory (or configuration) owned by, or accessible to, any other user (e.g. created by a user before the install) is never used, the installer removes it and creates the data directory already restricted. The installer verifies the owner and the access with PowerShell when the service isn't installed, and stops (removing nothing) when it can't verify them. Changes to config.env apply on the next start of the service (Restart-Service colony-print-node). Uninstalling keeps the configuration and the logs.

Self-Update

Every time the service starts, the node updates colony-print and npcolony to their newest versions in PyPI (pip install --upgrade), only installing their wheels (nothing is compiled in the node) and skipping the versions that don't support its Python (Requires-Python). Their dependencies are only updated when required. pip ignores its configuration files (e.g. C:\ProgramData\pip\pip.ini, which any user may create), but may be configured with PIP_* values in config.env (e.g. PIP_PROXY). A failed install (e.g. without internet access) only logs a warning and never prevents the node from running, as it keeps the installed packages, and the service runs the boot script from a copy outside of the packages (C:\Program Files\Colony Print Node\boot.py), so a broken or interrupted update never prevents it from starting. Once installed, the packages are verified by importing them (colony_print.node and npcolony) in another Python, as pip installs the packages whose native modules the system doesn't support (e.g. Windows XP). A failed verification is reported as a failed update (with its error, naming the modules that are not loaded), but the new packages are kept (nothing is rolled back) and may prevent the node from running, in which case the node is recovered by pinning the previous versions in its config.env (NODE_VERSION and NODE_NPCOLONY_VERSION).

The update is configured in config.env:

The versions pin a node (or roll it back), as the node installs the newest version they allow, including an older one.

The update may also be requested from the admin, together with the restart of the node and its auto-update (see Node Control). The auto-update set from the admin is kept in the state.env file (next to config.env) and takes precedence over NODE_UPDATE, and a requested update runs even with the auto-update disabled. The nodes installed by an older installer (0.23.0 or older) keep their boot script, which the self-update doesn't replace, so they're not updated from the admin until the installer is run again. The WinSW ones are still restarted from the admin, and their restart updates them when the auto-update is enabled, as their boot runs it at every start. The Windows XP ones of the 0.23.0 installer are not restarted either (see Windows XP).

Windows XP

Windows XP (SP3) nodes are installed with a second installer (colony-print-node-setup-xp-<version>.exe), which runs on every version of Windows from Windows XP SP3 on (32 and 64 bit). It's used, configured and updated as the other one (same parameters, service, config.env and self-update), with the following differences:

• It installs a 32 bit Python 2.7.18 (the last one that runs on Windows XP), together with the Visual C++ 2008 runtime that it requires (Microsoft's redistributable), and requires npcolony 1.7.0 or newer (the first one with wheels for it). The node only updates itself while colony-print, npcolony and their dependencies keep publishing wheels for Python 2.7.
• The service is run by NSSM instead of WinSW, which requires a .NET Framework that Windows XP lacks. The log files are rotated when they reach 10 MB, but the rotated ones are never removed.
• NSSM is not detected by the node, so the installer tells the node about it (NODE_SERVICE=nssm in the environment of the service), for the node to restart by exiting when restarted from the admin, as it does under WinSW, unless NODE_RESTART says otherwise in config.env (see Node Control). The nodes installed by the 0.23.0 installer are not restarted from the admin (they don't advertise it) until they're installed again or have NODE_RESTART=exit in their config.env.
• The node is installed in the 32 bit program files (C:\Program Files (x86)\Colony Print Node on a 64 bit Windows) and, on Windows XP, its configuration and logs are in C:\Documents and Settings\All Users\Application Data\Colony Print Node.
• The access to the configuration is restricted as in the other installer, except on the disks without file security (e.g. FAT32), where it's not possible: the installer warns (and asks to continue) that any user of the machine is able to read the secret key and to change the files of the node, which run as the system account. The access is not verified by the installer, as Windows XP has no PowerShell. So the configuration kept by an uninstall is not used by the next install, which must be given the configuration again, only an install over an installed node keeps its configuration.
• Python 2.7 always searches for its modules in the application paths of the registry (the subkeys of Software\Python\PythonCore\2.7\PythonPath, e.g. added by pywin32) before its own library, which can't be disabled. The installer warns (and asks to continue) when they exist, as any user able to write in those directories is able to run code as the system account.
• The installer doesn't verify the server URL and the secret key on the versions of Windows older than 8.1 when the server uses HTTPS, as they don't enable its secure protocols (TLS 1.2) by default, which the node itself supports.
• Windows XP has no PDF printer, so one must be installed to use the email mode.

Both installers use the same service, so only one of the nodes is installed in a machine. Installing one of them over the node of the other one (e.g. after upgrading the machine to Windows 10) replaces it, uninstalling the other node and keeping its configuration.

.\windows\build.ps1 -XP -Python C:\Python27\python.exe

The build requires a 32 bit Python 2.7.18 (with pip, setuptools and wheel) and Inno Setup 5.6.1, the last one that supports Windows XP. The NSSM service wrapper (the 32 bit build of its 2.24 version, which is public domain) is kept in the repository (windows/nssm.exe, with its digest pinned in build.ps1), as its site is unavailable too often for the build to download it. The Windows Workflow also builds this installer on every push (the colony-print-node-windows-xp artifact), smoke tests it on a Windows runner (not on Windows XP itself) and attaches it to the release. It also installs each of the installers over the node of the other one, to verify that the node is replaced.

Admin UI

A React-based admin interface is available under frontends/admin/ for monitoring nodes, jobs and printers.

cd frontends/admin
npm install
npm run build

The built assets are output to src/colony_print/static/admin-ui/ and served at /admin-ui when the server is running.

Its unit tests (under frontends/admin/react/test/) run with Vitest, with their coverage measured by its --coverage option. They require Node.js 22.22 (or 24.15) and newer, as Vitest and jsdom do, and run on every push in the Admin Workflow.

cd frontends/admin
npm test
npm test -- --coverage

Logo

The logo is a C cut in three slices, its middle slice shifted to the right as a line being printed, followed by the "colony print" logotype, cut in the same slices. Its files are in res/, and switch from ink to white on dark backgrounds by themselves:

The colors are ink #17181c (white on dark backgrounds) and mint #0fa47a, used only for the middle slice. Below 48px of height the logotype is drawn solid, without its cuts, which blur at that size, as the side bar of the admin UI does.

Development

To run a localhost development server, use the following commands:

PORT=8686 \
PYTHONPATH=$BASE_PATH/colony_print/src python \
$BASE_PATH/colony_print/src/colony_print/main.py

End to End Tests

The end to end tests (src/colony_print/test/printing/e2e.py) print the XMPL documents of src/colony_print/test/printing/resources with every renderer and compare their layout: the conversion of XMPL into PDF, the conversion into Binie rendered as PDF (as the Linux nodes do) and the same Binie document printed through the GDI renderer of npcolony (as the Windows nodes do), under Wine. The outputs are rasterized and the texts of each of their lines must have the same positions, extents and ink (telling the bold texts from the regular ones), within the small differences of the metrics of GDI, and the texts that go beyond the printable area must be the same in both renderers of Binie.

They require Wine with a Windows Python that has npcolony (E2E_PYTHON, the path of its python.exe, run by the wine of the PATH or by the one of E2E_WINE), a printer that uses the receipt.ppd description of the resources (E2E_PRINTER, a 72 mm printable area in a sheet wide enough for the texts beyond it to be visible), Ghostscript and Poppler, and they only run when E2E is set, keeping the outputs in E2E_OUTPUT when defined:

E2E=1 E2E_PYTHON=$HOME/wine-python/python.exe E2E_PRINTER=colony-print-e2e \
E2E_OUTPUT=e2e-output pytest src/colony_print/test/printing/e2e.py

The E2E Workflow runs them on every push, with both the stable and the development versions of Wine (from the WineHQ packages), installing the embedded Windows Python (with npcolony from PyPI) and the printer, and keeps the outputs (PDF documents, rasterized pages, Binie documents, and the PostScript of the printer driver and the log of Wine of the GDI prints), including the ones produced before a print that fails, in the colony-print-e2e-stable and colony-print-e2e-devel artifacts.

License

Colony Print Infra-structure is currently licensed under the Apache License, Version 2.0.

Build Automation