Skip to main content

Benchmark performance of **any model** deployed on **Amazon SageMaker** or available on **Amazon Bedrock** or deployed by you on an AWS service of choice (such as Amazon EKS or Amazon EC2) a.k.a **Bring your own endpoint**.

Project description

Foundation Model benchmarking tool (FMBench)

Benchmark any Foundation Model (FM) on any AWS service [Amazon SageMaker, Amazon Bedrock, Amazon EKS, Bring your own endpoint etc.]

Amazon SageMaker | Amazon Bedrock

PyPI Version

A key challenge with FMs is the ability to benchmark their performance in terms of inference latency, throughput and cost so as to determine which model running with what combination of the hardware and serving stack provides the best price-performance combination for a given workload.

Stated as business problem, the ask is “What is the dollar cost per transaction for a given generative AI workload that serves a given number of users while keeping the response time under a target threshold?

But to really answer this question, we need to answer an engineering question (an optimization problem, actually) corresponding to this business problem: “What is the minimum number of instances N, of most cost optimal instance type T, that are needed to serve a workload W while keeping the average transaction latency under L seconds?

W: = {R transactions per-minute, average prompt token length P, average generation token length G}

This foundation model benchmarking tool (a.k.a. FMBench) is a tool to answer the above engineering question and thus answer the original business question about how to get the best price performance for a given workload. Here is one of the plots generated by FMBench to help answer the above question (This foundation model benchmarking tool (a.k.a. FMBench) is a tool to answer the above engineering question and thus answer the original business question about how to get the best price performance for a given workload. Here is one of the plots generated by FMBench to help answer the above question (the instance types in the legend have been blurred out on purpose, you can find them in the actual plot generated on running FMBench).

business question

Models benchmarked

Configuration files are available in the configs folder for the following models in this repo.

Model SageMaker g4dn/g5/p3 SageMaker Inf2 SageMaker P4 SageMaker P5 Bedrock On-demand throughput Bedrock provisioned throughput
Anthropic Claude-3 Sonnet
Anthropic Claude-3 Haiku
Mistral-7b-instruct
Mistral-7b-AWQ
Mixtral-8x7b-instruct
Llama3-8b instruct
Llama3-70b instruct
Llama2-13b chat
Llama2-70b chat
Amazon Titan text lite
Amazon Titan text express
Cohere Command text
Cohere Command light text
AI21 J2 Mid
AI21 J2 Ultra
distilbert-base-uncased

New in this release

v1.0.45

  1. Analytics for results across multiple runs.
  2. Llama3-70b config files for g5.48xlarge instances.

v1.0.44

  1. Endpoint metrics (CPU/GPU utilization, memory utiliztion, model latency) and invocation metrics (including errors) for SageMaker Endpoints.
  2. Llama3-8b config files for g6 instances.

v1.0.42

  1. Config file for running Llama3-8b on all instance types except p5.
  2. Fix bug with business summary chart.
  3. Fix bug with deploying model using a DJL DeepSpeed container in the no S3 dependency mode.

v1.0.40

  1. Make it easy to run in the Amazon EC2 without any dependency on Amazon S3 dependency mode.

v1.0.39

  1. Add an internal FMBench website.

v1.0.38

  1. Support for running FMBench on Amazon EC2 without any dependency on Amazon S3.
  2. Llama3-8b-Instruct config file for ml.p5.48xlarge.

v1.0.37

  1. g5/p4d/p5/inf2/trn1 specific config files for Llama3-8b-Instruct.
    1. p4d config file for both vllm and lmi-dist.

v1.0.36

  1. Fix bug at higher concurrency levels (20 and above).
  2. Support for instance count > 1.

v1.0.35

  1. Support for Open-Orca dataset and corresponding prompts for Llama3, Llama2 and Mistral.

v1.0.34

  1. Don't delete endpoints for the bring your own endpoint case.
  2. Fix bug with business summary chart.

v1.0.32

  1. Report enhancements: New business summary chart, config file embedded in the report, version numbering and others.

  2. Additional config files: Meta Llama3 on Inf2, Mistral instruct with lmi-dist on p4d and p5 instances.

v1.0.31

  1. Meta Llama3 benchmarking on Amazon SageMaker.

v1.0.29

  1. Support for Amazon Bedrock. Benchmark models available on Bedrock, both on-demand throughput and provisioned throughput.

v1.0.28

  1. Support for HuggingFace datasets as well as bring your own datasets, more here.

  2. Support for external endpoints. No longer limited to Amazon SageMaker endpoints, more here.

  3. Bring your own Amazon SageMaker endpoints. If you have an already deployed SageMaker endpoint you can now test it with FMBench.

  4. Added config files for Mistral-7B-Instruct, Mistral-7B-Instruct-v0.2-AWQ, huggingface-tc-distilbert-base-uncased (from SageMaker JumpStart), meta-textgenerationneuron-llama-2-70b-f (on AWS Inferentia2).

Key Features

  1. Benchmark any model on any serving stack as long as it can be deployed on Amazon SageMaker.

  2. Bring your own script for model deployment if the model is not natively available via Amazon SageMaker JumpStart.

  3. Bring your own tokenizer for your model, configure any inference container parameters you need.

  4. Auto-generated reports comparing and contrasting different serving options.

Installation

  1. Launch the AWS CloudFormation template included in this repository using one of the buttons from the table below. The CloudFormation template creates the following resources within your AWS account: Amazon S3 buckets, Amazon IAM role and an Amazon SageMaker Notebook with this repository cloned. A read S3 bucket is created which contains all the files (configuration files, datasets) required to run FMBench and a write S3 bucket is created which will hold the metrics and reports generated by FMBench. The CloudFormation stack takes about 5-minutes to create.

    AWS Region Link
    us-east-1 (N. Virginia)
  2. Once the CloudFormation stack is created, navigate to SageMaker Notebooks and open the fmbench-notebook.

  3. On the fmbench-notebook open a Terminal and run the following commands.

    conda create --name fmbench_python311 -y python=3.11 ipykernel
    source activate fmbench_python311;
    pip install -U fmbench
    

Steps to run

  1. Now you are ready to fmbench with the following command line. We will use a sample config file placed in the S3 bucket by the CloudFormation stack for a quick first run.

    1. We benchmark performance for the Llama2-7b model on a ml.g5.xlarge and a ml.g5.2xlarge instance type, using the huggingface-pytorch-tgi-inference inference container. This test would take about 30 minutes to complete and cost about $0.20.

    2. It uses a simple relationship of 750 words equals 1000 tokens, to get a more accurate representation of token counts use the Llama2 tokenizer (instructions are provided in the next section). It is strongly recommended that for more accurate results on token throughput you use a tokenizer specific to the model you are testing rather than the default tokenizer. See instructions provided later in this document on how to use a custom tokenizer.

      account=`aws sts get-caller-identity | jq .Account | tr -d '"'`
      fmbench --config-file s3://sagemaker-fmbench-read-${account}/configs/llama2/7b/config-llama2-7b-g5-quick.yml
      
  2. The generated reports and metrics are available in the sagemaker-fmbench-write-<replace_w_your_aws_account_id> bucket. The metrics and report files are also downloaded locally and in the results directory (created by FMBench) and the benchmarking report is available as a markdown file called report.md in the results directory. You can view the rendered Markdown report in the SageMaker notebook itself or download the metrics and report files to your machine for offline analysis.

License

MIT-0

Documentation

The official documentation is available in the GitHub repo.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fmbench-1.0.45.tar.gz (569.2 kB view hashes)

Uploaded Source

Built Distribution

fmbench-1.0.45-py3-none-any.whl (686.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page