Skip to main content

WML AI Model Managers

A package with several AI classes making it easy to train data based on PyTorch.

Initialization

To initialize:

# Make sure the test and train data come from the same dataset
myai = WMLTextModelManagerOne(
    model_file_name="AmazonReviewFull.pkl",
    dataloader_info ={
      "datapipe_fn":datasets.AmazonReviewFull,
      "vocab_folder_path":"data/AmazonReviewFull",
      "get_dataset":True
    }
  )

Training

To train:

  myai.download_train_and_test_data()
  myai.load_model_from_scratch()
  myai.train()
  myai.save_model_to_pickle()

Chat

To chat:

myai =  WMLTextModelManagerOne(
    model_file_name="AmazonReviewFull.pkl",
    dataloader_info ={
      "datapipe_fn":datasets.AmazonReviewFull,
      "vocab_folder_path":"data/AmazonReviewFull",
      "get_dataset":False
    },
  )

myai.download_train_and_test_data()
myai.load_model_from_file()
# the default is 150 but we increased to 500 the limit should be myai.batch_size * myai.block_size but see if you can get larger than this
myai.chat_with_model(500)

Class Initialization Properties

  • device: Specifies whether to use CUDA, CPU, or lets it be determined by available computer hardware.
  • dataloader_info: Important information about the pytorch dataset the model manager needs to retrieve the dataset
    • datapipe_fn: One of the various pytorch datapipes which can be found here https://pytorch.org/text/stable/datasets.html. You can implement your own custom datapipe fn as the model is looking for an IterDataPipe, make sure your custom fn have train and test splits.
    • vocab_folder_path: Where the vocab file for the dataset will be stored. it will store test-vocab.txt and train-vocab.txt.
    • get_dataset: Set to false if communicating with chatbot set to true if doing a training session.
  • max_iters: Number of iterations for the entire training run.
  • n_embd: Number of embeddings.
  • n_head: Number of heads for multihead attention.
  • n_layer: Number of layers (linear, activation, output functions).
  • dropout: Number of values to turn to zero to prevent the model from memorization.
  • model_file_name: The name of the file to save the model to (currently supports only pickle files, so please save as .pkl).
  • reporting_loss.
  • learning_rate.
  • block_size: Amount of characters in a section of text that represents 1 batch.
  • batch_size: Amount of batches the model gets to learn in 1 iteration during the training session. A training session is like a child going through pre-K through college, and each grade is 1 iteration. At the end, the model should be able to generalize (converge) well, like an adult who can meaningfully contribute to society.

Changelog

v0.0.2:

  • Corrected issues with the dataloader where the model_manager would not receive the training and test dataloaders.

v0.0.3:

  • Changed the default block and batch sizes so beginners can feel more tangible results.
  • Refactored

v1.0.0:

  • instead of providing a dataloader argument to the modelManger provide dataloader_info according to the example above

v1.0.1

  • abstracted more fns into

v1.1.0

  • changed from load_model_from_file to load_model_from_pickle_file
  • changed from save_model_to_pickle to save_model_to_pickle_file

v1.1.1

  • fixed a bug with v1.1.0

Release files for wml-ai-model-managers 1.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for wml-ai-model-managers 1.1.1
File Size Uploaded
wml_ai_model_managers-1.1.1.tar.gz 15.6 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for wml-ai-model-managers 1.1.1
File Interpreter ABI Platform
wml_ai_model_managers-1.1.1-py3-none-any.whl Python 3 none any Details

Total release size: 41.4 kB

Release files / wml_ai_model_managers-1.1.1.tar.gz

Download URL wml_ai_model_managers-1.1.1.tar.gz
Size 15.6 kB
Tags Source
SHA-256 checksum
How to use checksums
f177b0eb3f14126355758acc722ccbc6032915f871757e644260b9c8bf8a1fd5
BLAKE2b-256 checksum
How to use checksums
dc3d5375bf409f7f37fb5fa269072b3f7600ad1b53b056f72125e391b6c41077
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.5

Release files / wml_ai_model_managers-1.1.1-py3-none-any.whl

Download URL wml_ai_model_managers-1.1.1-py3-none-any.whl
Size 25.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6c01e683b44d89fb0d571b49a4b8e487aac471c02279db0eb7c6daceeb66c99f
BLAKE2b-256 checksum
How to use checksums
02f716cf536b2459eb6e5560d320923a4b6b4ff62c4bf81b72d482de6d4b824c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.5

Release history Release notifications | RSS feed

This release

1.1.1 This release

2 release files

1.1.0

2 release files

1.0.1

2 release files

1.0.0

2 release files

0.0.3

2 release files

0.0.2

2 release files

0.0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page