clip-gaze
An art analysis tool powered by CLIP.
Motivation
Diffusion models (such as Stable Diffusion) used OpenAI's CLIP in order to perform textual analysis of their training data. Precisely what these machine learning systems actually learned from their training data is opaque. This tool helps us understand how CLIP, and therefore the models that use CLIP, see images.
What does it do?
Given an image and a series of (text) phrases it calculates the relative likelihood of each phrase to be a good description of the image. Note that this is not the same thing as "given a text phrase, calculate the accuracy of that phrase".
An example
Let's show it the painting "Brücke über die Marne bei Creteil" by Cézanne.
If we download the 2,175 × 1,713 pixel version of the painting and open it (e.g. using PIL.Image.open from the package pillow) as image we can then pass it to the gaze command.
# Assuming you have already saved the image
import clip_gaze
clip_gaze.gaze(image, clip_gaze.MOVEMENTS)
Just looking at the highest probability outputs for clip_gaze.ARTISTS_BY_TRAINING_PREVALENCE, clip_gaze.Movements, and clip_gaze.SURFACES we see:
{'artist': ['by paul cézanne (82%)'], 'movement': ['tonalism movement (16%)'], 'surface': ['on canvas (86%)']}
So the terms "by paul cézanne", "tonalism movement", and "on canvas" are the most likely to describe the input image.
For more examples and fuller documentation please see the project's page on github.
How does it work?
CLIP is a tool provided by OpenAI that calculates the similarity between an image and some text. This is a machine learning system trained on an enormous amount of data, and that data will contain biases (intentional and unintentional). It is not a source of truth, but a useful tool to give you ideas about where to search next.
This tool works by downloading CLIP onto your computer and running it locally.
This is not an easy task for all computers, especially older ones.
See the "Arguments for gaze" section of the project's README for a way to change memory load.
Biases
This software is built on a machine learning system, and the biases in this tool come in two parts:
- CLIP itself comes with its own biases, and we refer the user to OpenAI's own work on explaining and mitigating that bias
- The lists of chosen phrases
The lists used in this software are primarily from Wikipedia and from the training data that CLIP used. Neither of these sources are perfect, and care should be taken when using this software to account for these biases where possible. Although the lists are long (e.g. the list of 6000 artists) there are no claims of completeness or relative importance made.
Metadata
Release files for clip-gaze 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| clip-gaze-0.3.0.tar.gz | 117.2 kB | Without provenance | Details for clip-gaze-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| clip_gaze-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for clip_gaze-0.3.0-py3-none-any.whl |
Total release size: 255.9 kB
Release files / clip-gaze-0.3.0.tar.gz
| Download URL | clip-gaze-0.3.0.tar.gz |
|---|---|
| Size | 117.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
07589aad151b01e85d91e7e964a51b46f772e0806bb578aae427c9b0ee5b6ff8
|
|
BLAKE2b-256 checksum How to use checksums |
2b1a9f8fb828c0db139723bb0895975c2adac84af1d9dea367af9fc71e39c233
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.10
|
Release files / clip_gaze-0.3.0-py3-none-any.whl
| Download URL | clip_gaze-0.3.0-py3-none-any.whl |
|---|---|
| Size | 138.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5fa0ae0f65fff025f24e57cf6fc0afb87acb33dbc7a23b346a9e53531069ce24
|
|
BLAKE2b-256 checksum How to use checksums |
1227ad200aaa383b8ddbb59d0d71ab7ce7d5d141258117996bde036a1bee1b78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.10
|
Security for clip-gaze 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.