thumbor-piliptc-engine 7.4.7.4

Pil imaging engine for Thumbor with IPTC data passthrough

thumbor-piliptc-engine

thumbor-piliptc-engine is a patched version from the legacy Pil imaging engine for thumbor.

Warning : Since Thumbor release 7.5.0 , "piliptc" feature is integrated to official thumbor : this project will no more be maintained.

IPTC tags :

JPEG IPTC (International Press Telecommunications Council) tags are a set of metadata that can be embedded into JPEG image files to provide information about the image content, including ownership and copyright information. These tags can be used by photographers, artists, and publishers to identify their work and protect their intellectual property rights. By including IPTC tags in their JPEG images, creators can ensure that their ownership and copyright information is attached to their work and remains with it as it is shared and distributed across the internet. This can be particularly important for photographers and other creators who rely on their work to generate income, as it can help deter unauthorized use and ensure that they are properly credited for their work. In this way, JPEG IPTC tags can play an essential role in protecting the intellectual property of creators and maintaining the integrity of their work.

In Europe, image copyright is protected by various laws and regulations, such as the Berne Convention for the Protection of Literary and Artistic Works and the Directive on the harmonization of certain aspects of copyright and related rights in the information society. These laws provide creators with exclusive rights over their works, including photographs and images, and require that any use of these works by others be authorized or licensed by the creator.

The use of JPEG IPTC tags can be particularly helpful in enforcing these copyright laws. In fact, some countries in Europe, such as France and Germany, have enacted specific laws that require the use of IPTC tags on certain types of images, such as those used in advertising and editorial content. These laws require that the copyright owner's name and contact information be included in the IPTC tags, making it easier for potential infringers to identify and contact the owner for permission to use the image.

Thumbor :

this project is related with this issue : Preserve IPTC metadata #1301

kkopachev This is not possible now with default Pillow engine, as Pillow itself does not have a way to save IPTC/XMP data.

scorphus You can try and use thumbor-wand-engine, which is a new engine built on top of ImageMagick with support to IPTC/XMP data.

So we must choose between legacy engine 'thumbor.engines.pil' (Pillow) or 'thumbor_wand_engine' (imagemagick) if you want to preserve original image IPTC tags.

I tried the thumbor_wand_engine : it works fine ! it's maybe slower than "pil" but "thumbor" generated image contains iptc tags.

We need this feature for european/french laws compliance" about copyright.

I don't want to speak here about avantage/disavantage from each engine : speed, memory consumption, image rendering/quality ...

I wanted to stay on "pil" engine and provide IPTC preservation : so i start this small project.

In a near futur, i will try to "merge request" this feature on thumbor official project but it can be rejected because i add a new package dependance with iptcinfo3.

Releases:

v0.1.0 : pre-release : just a proof of concept : working version but not ready for production

v0.2.0 : remove iptcinfo3 package requirement : write my own iptc raw copy class to optimize usage within thumbor

v1.0.0 : add JpegIPTC package

v7.4.7 : new version modele

Try it !

https://hub.docker.com/repository/docker/gdegoulet/thumbor_piliptc_engine/general

docker run --rm -it gdegoulet/thumbor_piliptc_engine:latest      pip list | egrep -i "(thumbor|iptc|jpeg|pillow)"
JpegIPTC               1.1
libthumbor             2.0.2
Pillow                 9.4.0
thumbor                7.4.7
thumbor-piliptc-engine 7.4.7.3
thumbor-plugins        0.2.4
thumbor-plugins-gifv   0.1.2
thumbor-wand-engine    0.1.1

docker run --rm -it -p8902:8000 \
  -e LOG_LEVEL=DEBUG \
  -e ENGINE=thumbor_piliptc_engine \
  -e PRESERVE_EXIF_INFO=True \
  -e FILE_LOADER_ROOT_PATH=/data/thumbor/tmp \
  -e FILE_STORAGE_ROOT_PATH=/data/thumbor/storage \
  -e RESULT_STORAGE_FILE_STORAGE_ROOT_PATH=/data/thumbor/result_storage \
  -e RESULT_STORAGE_STORES_UNSAFE=True \
  docker.io/gdegoulet/thumbor_piliptc_engine

wget -O test.jpg "http://localhost:8902/unsafe/x200/filters:quality(70)/i.f1g.fr/media/cms/1936x527_cropupscale/2023/03/11/1a1f02bd710b9768995d58a5e2cdbeb8e89be7fa7215476a8ea55e8c4951ceac.jpg"

iptc test.jpg | head
test.jpg:
 Tag      Name                 Type      Size  Value
 -------- -------------------- --------- ----  -----
 1:000    Model Version        Short        2  2
 1:020    File Format          Short        2  1
 1:022    File Version         Short        2  2
 1:030    Service Identifier   String       9  AFP-PHOTO
 1:040    Envelope Number      NumString    8  12345678
 1:060    Envelope Priority    NumString    1  5
 1:070    Date Sent            Date         8  20230311

Installation

You can install the package from this repository with pip:

$ pip install git+https://github.com/gdegoulet/thumbor-piliptc-engine@v7.4.7.3

or

pip install thumbor-piliptc-engine==7.4.7.*

Requirements

• Python 3.7 or higher
• Thumbor (same version than thumbor-piliptc-engine )
• git (for now, you can only install from github repository )
• JpegIPTC (configured as dependance)

root@44171bd2df65:/# pip install --no-cache-dir thumbor-piliptc-engine==7.4.7.*

Collecting thumbor-piliptc-engine==7.4.7.*
  Downloading thumbor_piliptc_engine-7.4.7.3-py3-none-any.whl (14 kB)
Requirement already satisfied: thumbor==7.4.7 in /app/lib/python3.11/site-packages (from thumbor-piliptc-engine==7.4.7.*) (7.4.7)
Requirement already satisfied: pillow>=9.0 in /app/lib/python3.11/site-packages (from thumbor-piliptc-engine==7.4.7.*) (9.4.0)
Collecting JpegIPTC>=1.4
  Downloading JpegIPTC-1.4-py3-none-any.whl (9.7 kB)
Requirement already satisfied: colorama==0.*,>=0.4.3 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (0.4.6)
Requirement already satisfied: derpconf==0.*,>=0.8.3 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (0.8.3)
Requirement already satisfied: libthumbor==2.*,>=2.0.2 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (2.0.2)
Requirement already satisfied: piexif==1.*,>=1.1.3 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (1.1.3)
Requirement already satisfied: pytz>=2019.3.0 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (2022.7.1)
Requirement already satisfied: statsd==3.*,>=3.3.0 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (3.3.0)
Requirement already satisfied: tornado==6.*,>=6.0.3 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (6.2)
Requirement already satisfied: thumbor-plugins-gifv==0.*,>=0.1.2 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (0.1.2)
Requirement already satisfied: webcolors==1.*,>=1.10.0 in /app/lib/python3.11/site-packages (from thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (1.11.1)
Requirement already satisfied: six in /app/lib/python3.11/site-packages (from derpconf==0.*,>=0.8.3->thumbor==7.4.7->thumbor-piliptc-engine==7.4.7.*) (1.16.0)
Installing collected packages: JpegIPTC, thumbor-piliptc-engine
Successfully installed JpegIPTC-1.4 thumbor-piliptc-engine-7.4.7.3

Usage

To use this engine with thumbor, define thumbor_piliptc_engine as the imaging engine in thumbor.conf:

ENGINE = "thumbor_piliptc_engine"

Example

wget https://i.f1g.fr/media/cms/509x286_crop/2022/11/21/76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg


iptc 76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg
76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg:
 Tag      Name                 Type      Size  Value
 -------- -------------------- --------- ----  -----
 1:000    Model Version        Short        2  2
 1:020    File Format          Short        2  1
 1:022    File Version         Short        2  2
 1:030    Service Identifier   String       9  AFP-PHOTO
 1:040    Envelope Number      NumString    8  12345678
 1:060    Envelope Priority    NumString    1  5
 1:070    Date Sent            Date         8  20221120
 1:080    Time Sent            Time        11  210118+0000
 1:090    Coded Character Set  Binary       3  1b 2d 41
 1:100    Unique Name of Objec String      11  AFP_32PC4R2
 2:000    Record Version       Short        2  2
 2:005    Object Name          String      27  UKRAINE-RUSSIA-WAR-CONFLICT
 2:010    Urgency              NumString    1  5
 2:012:00 Subject Reference    String      45  IPTC:16009000:unrest, conflicts and  war:war:
 2:012:01 Subject Reference    String      41  IPTC:16000000:unrest, conflicts and  war:
 2:015    Category             String       3  WAR
 2:020    Supplemental Categor String       3  war
 2:025:00 Keywords             String       3  war
 2:025:01 Keywords             String      10  Horizontal
 2:055    Date Created         Date         8  20221120
 2:060    Time Created         Time        11  152935+0300
 2:062    Digital Creation Dat Date         8  20221120
 2:063    Digital Creation Tim Time        11  152935+0300
 2:065    Originating Program  String      22  g2mapping/libg2 3.9.39
 2:070    Program Version      String       6  3.9.15
 2:080    By-line              String      12  BULENT KILIC
 2:085    By-line Title        String       3  STF
 2:090    City                 String      12  Chornobaivka
 2:100    Country Code         String       3  UKR
 2:101    Country Name         String       7  Ukraine
 2:110    Credit               String       3  AFP
 2:115    Source               String       3  AFP
 2:116    Copyright Notice     String      16  AFP or licensors
 2:120    Caption/Abstract     String     242  A Ukrainian soldier walks in front of a destroyed building of the International Airport of Kherson in the village of Chornobaivka, outskirts of Kherson, on November 20, 2022, amid the Russian invasion of Ukraine. (Photo by BULENT KILIC / AFP)
 2:135    Language Identifier  String       2  EN



wget -O test.jpg "http://localhost:8901/unsafe/filters:quality(60)/i.f1g.fr/media/cms/509x286_crop/2022/11/21/76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg"
--2023-03-09 19:05:53--  http://localhost:8901/unsafe/filters:quality(60)/i.f1g.fr/media/cms/509x286_crop/2022/11/21/76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8901... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31970 (31K) [image/jpeg]
Saving to: 'test.jpg'

test.jpg                                                  100%[===================================================================================================================================>]  31,22K  --.-KB/s    in 0s

2023-03-09 19:05:53 (348 MB/s) - 'test.jpg' saved [31970/31970]

iptc test.jpg
test.jpg:
 Tag      Name                 Type      Size  Value
 -------- -------------------- --------- ----  -----
 1:000    Model Version        Short        2  2
 1:020    File Format          Short        2  1
 1:022    File Version         Short        2  2
 1:030    Service Identifier   String       9  AFP-PHOTO
 1:040    Envelope Number      NumString    8  12345678
 1:060    Envelope Priority    NumString    1  5
 1:070    Date Sent            Date         8  20221120
 1:080    Time Sent            Time        11  210118+0000
 1:090    Coded Character Set  Binary       3  1b 2d 41
 1:100    Unique Name of Objec String      11  AFP_32PC4R2
 2:000    Record Version       Short        2  2
 2:005    Object Name          String      27  UKRAINE-RUSSIA-WAR-CONFLICT
 2:010    Urgency              NumString    1  5
 2:012:00 Subject Reference    String      45  IPTC:16009000:unrest, conflicts and  war:war:
 2:012:01 Subject Reference    String      41  IPTC:16000000:unrest, conflicts and  war:
 2:015    Category             String       3  WAR
 2:020    Supplemental Categor String       3  war
 2:025:00 Keywords             String       3  war
 2:025:01 Keywords             String      10  Horizontal
 2:055    Date Created         Date         8  20221120
 2:060    Time Created         Time        11  152935+0300
 2:062    Digital Creation Dat Date         8  20221120
 2:063    Digital Creation Tim Time        11  152935+0300
 2:065    Originating Program  String      22  g2mapping/libg2 3.9.39
 2:070    Program Version      String       6  3.9.15
 2:080    By-line              String      12  BULENT KILIC
 2:085    By-line Title        String       3  STF
 2:090    City                 String      12  Chornobaivka
 2:100    Country Code         String       3  UKR
 2:101    Country Name         String       7  Ukraine
 2:110    Credit               String       3  AFP
 2:115    Source               String       3  AFP
 2:116    Copyright Notice     String      16  AFP or licensors
 2:120    Caption/Abstract     String     242  A Ukrainian soldier walks in front of a destroyed building of the International Airport of Kherson in the village of Chornobaivka, outskirts of Kherson, on November 20, 2022, amid the Russian invasion of Ukraine. (Photo by BULENT KILIC / AFP)
 2:135    Language Identifier  String       2  EN

Logs

root@44171bd2df65:/src# thumbor --port=8000 --conf=/usr/src/app/thumbor.conf -l DEBUG
2023-03-11 16:22:52 root:DEBUG thumbor starting at 0.0.0.0:8000
2023-03-11 16:22:52 asyncio:DEBUG Using selector: EpollSelector
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: response.count:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: response.none_smart:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: response.none_smart:0
2023-03-11 16:23:07 thumbor:DEBUG [RESULT_STORAGE] getting from /data/thumbor/result_storage/default/16/10/fc6d87db0af8192db5a48d82b016d4c28918
2023-03-11 16:23:07 thumbor:DEBUG [RESULT_STORAGE] image not found at /data/thumbor/result_storage/default/16/10/fc6d87db0af8192db5a48d82b016d4c28918
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: result_storage.incoming_time:0
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: result_storage.miss:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: storage.miss:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: original_image.fetch.200.i_f1g_fr:59
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: original_image.fetch.200.i_f1g_fr:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: original_image.status.200:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: original_image.status.200.i_f1g_fr:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: original_image.response_bytes:38698
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: iptc_passthrough_create_image.time:0
2023-03-11 16:23:07 thumbor:DEBUG creating tempfile for i.f1g.fr/media/cms/509x286_crop/2022/11/21/76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg in /data/thumbor/storage/c8/a265b39d5bec5b40ca5a3714d34fbc04be5aae.e65963ced7774178a74318dfcc414161...
2023-03-11 16:23:07 thumbor:DEBUG moving tempfile /data/thumbor/storage/c8/a265b39d5bec5b40ca5a3714d34fbc04be5aae.e65963ced7774178a74318dfcc414161 to /data/thumbor/storage/c8/a265b39d5bec5b40ca5a3714d34fbc04be5aae...
2023-03-11 16:23:07 thumbor:DEBUG No image format specified. Retrieving from the image extension: .jpg.
2023-03-11 16:23:07 thumbor:DEBUG Content Type of image/jpeg detected.
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: iptc_passthrough_read.time:0
2023-03-11 16:23:07 tornado.access:INFO 200 GET /unsafe/filters:quality(60)/i.f1g.fr/media/cms/509x286_crop/2022/11/21/76bde3fc961f0fa8733756922d1e2ed06311d804ec38b89dc60d6ba36d30e046.jpg (10.201.2.1) 99.44ms
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: response.time:98
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: response.time.200:98
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: response.status.200:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: response.not_smart.count:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: response.not_smart.latency:98
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: response.format.jpg:1
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: response.time.jpg:98
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: response.bytes.jpg:32132
2023-03-11 16:23:07 thumbor:DEBUG [RESULT_STORAGE] putting at /data/thumbor/result_storage/default/16/10/fc6d87db0af8192db5a48d82b016d4c28918 (/data/thumbor/result_storage/default/16/10)
2023-03-11 16:23:07 thumbor:DEBUG METRICS: inc: result_storage.bytes_written:32132
2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: result_storage.outgoing_time:0

2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: iptc_passthrough_create_image.time:0

...

2023-03-11 16:23:07 thumbor:DEBUG METRICS: timing: iptc_passthrough_read.time:0