fmbench 1.0.45

Benchmark performance of **any model** deployed on **Amazon SageMaker** or available on **Amazon Bedrock** or deployed by you on an AWS service of choice (such as Amazon EKS or Amazon EC2) a.k.a **Bring your own endpoint**.

Foundation Model benchmarking tool (FMBench)

Benchmark any Foundation Model (FM) on any AWS service [Amazon SageMaker, Amazon Bedrock, Amazon EKS, Bring your own endpoint etc.]

Amazon SageMaker | Amazon Bedrock

A key challenge with FMs is the ability to benchmark their performance in terms of inference latency, throughput and cost so as to determine which model running with what combination of the hardware and serving stack provides the best price-performance combination for a given workload.

Stated as business problem, the ask is “What is the dollar cost per transaction for a given generative AI workload that serves a given number of users while keeping the response time under a target threshold?”

But to really answer this question, we need to answer an engineering question (an optimization problem, actually) corresponding to this business problem: “What is the minimum number of instances N, of most cost optimal instance type T, that are needed to serve a workload W while keeping the average transaction latency under L seconds?”

W: = {R transactions per-minute, average prompt token length P, average generation token length G}

This foundation model benchmarking tool (a.k.a. FMBench) is a tool to answer the above engineering question and thus answer the original business question about how to get the best price performance for a given workload. Here is one of the plots generated by FMBench to help answer the above question (This foundation model benchmarking tool (a.k.a. FMBench) is a tool to answer the above engineering question and thus answer the original business question about how to get the best price performance for a given workload. Here is one of the plots generated by FMBench to help answer the above question (the instance types in the legend have been blurred out on purpose, you can find them in the actual plot generated on running FMBench).

Models benchmarked

Configuration files are available in the configs folder for the following models in this repo.

Model	SageMaker g4dn/g5/p3	SageMaker Inf2	SageMaker P4	SageMaker P5	Bedrock On-demand throughput	Bedrock provisioned throughput
Anthropic Claude-3 Sonnet					✅	✅
Anthropic Claude-3 Haiku					✅
Mistral-7b-instruct	✅		✅	✅	✅
Mistral-7b-AWQ				✅
Mixtral-8x7b-instruct					✅
Llama3-8b instruct	✅	✅	✅		✅
Llama3-70b instruct	✅	✅	✅		✅
Llama2-13b chat	✅	✅	✅		✅
Llama2-70b chat	✅	✅	✅		✅
Amazon Titan text lite					✅
Amazon Titan text express					✅
Cohere Command text					✅
Cohere Command light text					✅
AI21 J2 Mid					✅
AI21 J2 Ultra					✅
distilbert-base-uncased	✅

New in this release

v1.0.45

1. Analytics for results across multiple runs.
2. Llama3-70b config files for g5.48xlarge instances.

v1.0.44

1. Endpoint metrics (CPU/GPU utilization, memory utiliztion, model latency) and invocation metrics (including errors) for SageMaker Endpoints.
2. Llama3-8b config files for g6 instances.

v1.0.42

1. Config file for running Llama3-8b on all instance types except p5.
2. Fix bug with business summary chart.
3. Fix bug with deploying model using a DJL DeepSpeed container in the no S3 dependency mode.

v1.0.40

1. Make it easy to run in the Amazon EC2 without any dependency on Amazon S3 dependency mode.

v1.0.39

1. Add an internal FMBench website.

v1.0.38

1. Support for running FMBench on Amazon EC2 without any dependency on Amazon S3.
2. Llama3-8b-Instruct config file for ml.p5.48xlarge.

v1.0.37

1. g5/p4d/p5/inf2/trn1 specific config files for Llama3-8b-Instruct.
   1. p4d config file for both vllm and lmi-dist.

v1.0.36

1. Fix bug at higher concurrency levels (20 and above).
2. Support for instance count > 1.

v1.0.35

1. Support for Open-Orca dataset and corresponding prompts for Llama3, Llama2 and Mistral.

v1.0.34

1. Don't delete endpoints for the bring your own endpoint case.
2. Fix bug with business summary chart.

v1.0.32

1. Report enhancements: New business summary chart, config file embedded in the report, version numbering and others.

2. Additional config files: Meta Llama3 on Inf2, Mistral instruct with lmi-dist on p4d and p5 instances.

v1.0.31

1. Meta Llama3 benchmarking on Amazon SageMaker.

v1.0.29

1. Support for Amazon Bedrock. Benchmark models available on Bedrock, both on-demand throughput and provisioned throughput.

v1.0.28

1. Support for HuggingFace datasets as well as bring your own datasets, more here.

2. Support for external endpoints. No longer limited to Amazon SageMaker endpoints, more here.

3. Bring your own Amazon SageMaker endpoints. If you have an already deployed SageMaker endpoint you can now test it with FMBench.

4. Added config files for Mistral-7B-Instruct, Mistral-7B-Instruct-v0.2-AWQ, huggingface-tc-distilbert-base-uncased (from SageMaker JumpStart), meta-textgenerationneuron-llama-2-70b-f (on AWS Inferentia2).

Key Features

1. Benchmark any model on any serving stack as long as it can be deployed on Amazon SageMaker.

2. Bring your own script for model deployment if the model is not natively available via Amazon SageMaker JumpStart.

3. Bring your own tokenizer for your model, configure any inference container parameters you need.

4. Auto-generated reports comparing and contrasting different serving options.

Installation

1. Launch the AWS CloudFormation template included in this repository using one of the buttons from the table below. The CloudFormation template creates the following resources within your AWS account: Amazon S3 buckets, Amazon IAM role and an Amazon SageMaker Notebook with this repository cloned. A read S3 bucket is created which contains all the files (configuration files, datasets) required to run FMBench and a write S3 bucket is created which will hold the metrics and reports generated by FMBench. The CloudFormation stack takes about 5-minutes to create.

   AWS Region	Link
   us-east-1 (N. Virginia)

2. Once the CloudFormation stack is created, navigate to SageMaker Notebooks and open the fmbench-notebook.

3. On the fmbench-notebook open a Terminal and run the following commands.

   conda create --name fmbench_python311 -y python=3.11 ipykernel
   source activate fmbench_python311;
   pip install -U fmbench
   

Steps to run

1. Now you are ready to fmbench with the following command line. We will use a sample config file placed in the S3 bucket by the CloudFormation stack for a quick first run.

   1. We benchmark performance for the Llama2-7b model on a ml.g5.xlarge and a ml.g5.2xlarge instance type, using the huggingface-pytorch-tgi-inference inference container. This test would take about 30 minutes to complete and cost about $0.20.

   2. It uses a simple relationship of 750 words equals 1000 tokens, to get a more accurate representation of token counts use the Llama2 tokenizer (instructions are provided in the next section). It is strongly recommended that for more accurate results on token throughput you use a tokenizer specific to the model you are testing rather than the default tokenizer. See instructions provided later in this document on how to use a custom tokenizer.

      account=`aws sts get-caller-identity | jq .Account | tr -d '"'`
      fmbench --config-file s3://sagemaker-fmbench-read-${account}/configs/llama2/7b/config-llama2-7b-g5-quick.yml
      

2. The generated reports and metrics are available in the sagemaker-fmbench-write-<replace_w_your_aws_account_id> bucket. The metrics and report files are also downloaded locally and in the results directory (created by FMBench) and the benchmarking report is available as a markdown file called report.md in the results directory. You can view the rendered Markdown report in the SageMaker notebook itself or download the metrics and report files to your machine for offline analysis.

License

MIT-0

Documentation

The official documentation is available in the GitHub repo.