Skip to main content

The multi-lingual DALL-E for story visualization model as described in 'Mudrik, N., Charles, A., “Multi-Lingual DALL-E Storytime”. Arxiv. (2022). https://arxiv.org/abs/2212.11985'

Project description

multi-lingual-storytelling-dall-e

Mudrik, N., Charles, A., “Multi-Lingual DALL-E Storytime”. Arxiv. (2022). https://arxiv.org/abs/2212.11985

While recent advancements in artificial intelligence (AI) language models demonstrate cutting-edge performance when working with English texts, equivalent models do not exist in other languages or do not reach the same performance level. This undesired effect of AI advancements increases the gap between access to new technology from different populations across the world. This unsought bias mainly discriminates against individuals whose English skills are less developed, e.g., non-English speakers children. Following significant advancements in AI research in recent years, OpenAI has recently presented DALL-E: a powerful tool for creating images based on English text prompts. While DALL-E is a promising tool for many applications, its decreased performance when given input in a different language, limits its audience and deepens the gap between populations. An additional limitation of the current DALL-E model is that it only allows for the creation of a few images in response to a given input prompt, rather than a series of consecutive coherent frames that tell a story or describe a process that changes over time. Here, we present an easy-to-use automatic DALL-E storytelling framework that leverages the existing DALL-E model to enable fast and coherent visualizations of non-English songs and stories, pushing the limit of the one-step-at-a-time option DALL-E currently offers. We show that our framework is able to effectively visualize stories from non-English texts and portray the changes in the plot over time. It is also able to create a narrative and maintain interpretable changes in the description across frames. Additionally, our framework offers users the ability to specify constraints on the story elements, such as a specific location or context, and to maintain a consistent style throughout the visualization.

How to use?

one time pre-steps:

  1. create an account in openai and store your api key. You will have to type your key later.
  2. create a free Google workspace account, create credetials, and download the associated json file. (https://cloud.google.com/apis)
  3. rename the json file to "translation.json"

using the package:

  1. pip install the package
  2. import the package
  3. run song2images([name of text file], parameters)
  4. You will find the set of images in the folder 'images' under the current directory
  5. in order to create a figure of subplots, run _create_subplots_fig(path of saved images, num_columns (int), path_save = [where to save the subplots fig?], title = [name of file? without .png]) _
  6. in order to save to gif, run create_gif(path of images, name_save = [how to call the gif?], path_save = [where to save the gif?])

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

multi-lingual-storytelling-dall-e-0.0.1.tar.gz (24.2 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file multi-lingual-storytelling-dall-e-0.0.1.tar.gz.

File metadata

File hashes

Hashes for multi-lingual-storytelling-dall-e-0.0.1.tar.gz
Algorithm Hash digest
SHA256 71622f1e9180749e2aea9835fbaf3f5ca47813b18f74ae7aa790d0abcf2e19f7
MD5 baf0e8053997033aa7cb79e9d71d06aa
BLAKE2b-256 cd04dd7a1cb54b6dc137ffcca31d02605cc1f6466bd0375709134aa314f4e813

See more details on using hashes here.

File details

Details for the file multi_lingual_storytelling_dall_e-0.0.1-py3-none-any.whl.

File metadata

File hashes

Hashes for multi_lingual_storytelling_dall_e-0.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 a0be94a9a0dba939113f282419b4b1fb456e9594d7eddccaa56c275bbaa1cb15
MD5 65bd3162e4a5ea6d2b4ad01e7cbc40ba
BLAKE2b-256 6a3d892bd1d9ab543eb8b2189b6b43c5e4f19bdd163f0042920ac3cc2a31a6a1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page