ruclip 0.0.1rc5

RuCLIP: Zero-shot image classification models for Russian language

RuCLIP

Zero-shot image classification model for Russian language

---

RuCLIP (Russian Contrastive Language–Image Pretraining) is a multimodal model for obtaining images and text similarities and rearranging captions and pictures. RuCLIP builds on a large body of work on zero-shot transfer, computer vision, natural language processing and multimodal learning. This repo has the prototypes model of OpenAI CLIP's Russian version following this paper.

Models

• ruclip-vit-base-patch32-224 🤗
• ruclip-vit-base-patch16-224 🤗
• ruclip-vit-large-patch14-224 🤗
• ruclip-vit-large-patch14-336 ☁️SberCloud only
• ruclip-vit-base-patch32-384 🤗
• ruclip-vit-base-patch16-384 🤗
• ruclip-vit-large-patch14-384 ☁️SberCloud only ️

Performance

We have evaluated the performance on the following datasets:

Dataset	Metric Name	ruclip-vit-base-patch32-224	ruclip-vit-base-patch16-224	ruclip-vit-large-patch14-224	ruclip-vit-large-patch14-336 ☁️SberCloud only	ruclip-vit-base-patch32-384	ruclip-vit-base-patch16-384	ruclip-vit-large-patch14-384 ☁️SberCloud only
Food101	acc	0.505	0.552	0.597	0.712	-	-	-
CIFAR10	acc	0.818	0.810	0.878	0.906	-	-	-
CIFAR100	acc	0.504	0.496	0.511	0.591	-	-	-
Birdsnap	acc	0.115	0.117	0.172	0.213	-	-	-
SUN397	acc	0.452	0.462	0.484	0.523	-	-	-
Stanford Cars	acc	0.433	0.487	0.559	0.659	-	-	-
DTD	acc	0.380	0.401	0.370	0.408	-	-	-
MNIST	acc	0.447	0.464	0.337	0.242	-	-	-
STL10	acc	0.932	0.932	0.934	0.956	-	-	-
PCam	acc	0.501	0.505	0.520	0.554	-	-	-
CLEVR	acc	0.148	0.128	0.152	0.142	-	-	-
Rendered SST2	acc	0.489	0.527	0.529	0.539	-	-	-
ImageNet	acc	0.375	0.401	0.426	0.488	-	-	-
FGVC Aircraft	mean-per-class	0.033	0.043	0.046	0.075	-	-	-
Oxford Pets	mean-per-class	0.560	0.595	0.604	0.546	-	-	-
Caltech101	mean-per-class	0.786	0.775	0.777	0.835	-	-	-
Flowers102	mean-per-class	0.401	0.388	0.455	0.517	-	-	-
HatefulMemes	roc-auc	0.564	0.516	0.530	0.519	-	-	-

Supported by