Skip to main content

AllenNLP integration for Shiba: Japanese CANINE model

Project description

Allennlp Integration for Shiba

CI Release Python PyPI

allennlp-shiab-model is a Python library that provides AllenNLP integration for shiba-model.

SHIBA is an approximate reimplementation of CANINE [1] in raw Pytorch, pretrained on the Japanese wikipedia corpus using random span masking. If you are unfamiliar with CANINE, you can think of it as a very efficient (approximately 4x as efficient) character-level BERT model. Of course, the name SHIBA comes from the identically named Japanese canine.


Installing the library and dependencies is simple using pip.

pip install allennlp-shiba


This library enables users to specify the in a jsonnet config file. Here is an example of the model in jsonnet config file:

    "dataset_reader": {
        "tokenizer": {
            "type": "shiba",
        "token_indexers": {
            "tokens": {
                "type": "shiba",
    "model": {
        "shiba_embedder": {
            "type": "basic",
            "token_embedders": {
                "shiba": {
                    "type": "shiba",
                    "eval_model": true,



Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

allennlp-shiba-0.1.1.tar.gz (9.6 kB view hashes)

Uploaded source

Built Distribution

allennlp_shiba-0.1.1-py3-none-any.whl (11.4 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page