Skip to main content

A multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation.

Project description

MMKit-Features: Multimodal Feature Extraction Toolkit

Traditional knowledge graphs (KGs) are usually comprised of entities, relationships, and attributes. However, they are not designed to effectively store or represent multimodal data. This limitation prevents them from capturing and integrating information from different modes of data, such as text, images, and audio, in a meaningful and holistic way.

The MMKit-Features project proposes a multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation.

Project Goal

  • To extract, store, and fuse various multimodal features from multimodal datasets efficiently;
  • To achieve generative adversarial network(GAN)-based multimodal knowledge representation dynamically in multimodal knowledge graphs;
  • To provide a common deep learning-based architecture to enhance multimodal knowledge reasoning in real life.

Installation

You can install this toolkit using our PyPi package.

    pip install mmkit-features

Design Science Framework

Multimodal Computational Sequence

Figure 1: Multimodal Computational Sequence

GAN-based Multimodal Concept Generation

Figure 2: GAN-based Multimodal Concept Generation

Modalities

  1. Text/Language modality
  2. Image modality
  3. Video modality
  4. Audio modality
  5. Cross-modality among above

Usage

A toy example showing how to build a multimodal feature (MMF) library is here:

from mmkfeatures.fusion.mm_features_lib import MMFeaturesLib
from mmkfeatures.fusion.mm_features_node import MMFeaturesNode
import numpy as np
if __name__ == "__main__":
    # 1. create an empty multimodal features library with root and dataset names
    feature_lib = MMFeaturesLib(root_name="test features",dataset_name = "test_features")
    # 2. set short names for each dimension for convenience
    feature_lib.set_features_name(["feature1","feature2","feature3"])
    # 3. set a  list of content IDs
    content_ids = ["content1","content2","content3"]
    # 4. according to IDs, assign a group of features with interval to corresponding content ID
    features_dict = {}
    for id in content_ids:
        mmf_node = MMFeaturesNode(id)
        mmf_node.set_item("name",str(id))
        mmf_node.set_item("features",np.array([[1,2,3]]))
        mmf_node.set_item("intervals",np.array([[0,1]]))
        features_dict[id] = mmf_node
    # 5. set the library's data
    feature_lib.set_data(features_dict)
    # 6. save the features to disk for future use
    feature_lib.save_data("test6_feature.csd")
    # 7. check structure of lib file with the format of h5py
    feature_lib.show_structure("test6_feature.csd")
    # 8. have a glance of features content within the dataset
    feature_lib.show_sample_data("test6_feature.csd")
    # 9. Finally, we construct a simple multimodal knowledge base. 

Further instructions on the toolkit refers to here.

Applications

Here are some examples of using our work in real life with codes and documents.

1. Multimodal Features Extractors

2. Multimodal Feature Library (MMFLib)

3. Multimodal Knowledge Bases

4. Multimodal Indexing and Querying

Credits

The project includes some source codes from various open-source contributors. Here is a list of their contributions.

  1. A2Zadeh/CMU-MultimodalSDK
  2. aishoot/Speech_Feature_Extraction
  3. antoine77340/video_feature_extractor
  4. jgoodman8/py-image-features-extractor
  5. v-iashin/Video Features

License

The mmkit-features project is provided by Donghua Chen with MIT license.

Citation

Please cite our project if the project is used in your research.

Chen, D. (2023). MMKit-Features: Multimodal Features Extraction Toolkit (Version 0.0.2) [Computer software]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mmkit-features-0.0.1a4.tar.gz (112.7 kB view details)

Uploaded Source

Built Distribution

mmkit_features-0.0.1a4-py3-none-any.whl (145.3 kB view details)

Uploaded Python 3

File details

Details for the file mmkit-features-0.0.1a4.tar.gz.

File metadata

  • Download URL: mmkit-features-0.0.1a4.tar.gz
  • Upload date:
  • Size: 112.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.11

File hashes

Hashes for mmkit-features-0.0.1a4.tar.gz
Algorithm Hash digest
SHA256 470cbf12d6ccf99c2717ddf57d3824f4b095b95ade69323636a538e5d58e2eda
MD5 1014f5bf615e3d294ce16b14b740ed98
BLAKE2b-256 2e3f67a9ac82cf3d09643f2d82151c1fb2e4ae66b70914dcaf4c469dc712b17f

See more details on using hashes here.

Provenance

File details

Details for the file mmkit_features-0.0.1a4-py3-none-any.whl.

File metadata

  • Download URL: mmkit_features-0.0.1a4-py3-none-any.whl
  • Upload date:
  • Size: 145.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.11

File hashes

Hashes for mmkit_features-0.0.1a4-py3-none-any.whl
Algorithm Hash digest
SHA256 452f53f5eecd81ec5980e49fda180dbe330990228fa1573f92e84eb96167cb5a
MD5 c2ff91f366c63867ff384dd2616a69ce
BLAKE2b-256 f5f4a8187704e18a8ee928aa55c3ea672556e73a56d98343e71bef1285766aac

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page