Skip to main content

Client SDK for BlindAI Confidential Inference Server

Project description

BlindAI

Mithril Security – BlindAI

Website | Cloud | Github | Documentation | Blog | Docker Hub | LinkedIn | Twitter | Discord

Fast, accessible and privacy friendly AI deployment 🚀🔒

BlindAI is a confidential AI inference server. Like regular AI inference solutions, BlindAI helps AI engineers serve models for end-users to benefit from their predictions, but with an added privacy layer. Data sent by users to the AI model is kept confidential at all times, from the transfer to the analysis. This way, users can benefit from AI models without ever having to expose their data in clear to anyone: neither the AI service provider, nor the Cloud provider (if any), can see the data.

Confidentiality is assured by using special hardware-enforced Trusted Execution Environments. To read more about those, read our blog series here.

Our solution comes in two parts:

  • A secure inference server to serve AI models with privacy guarantees, developed using the Rust Programming Language. 🦀🦀
  • A Python client SDK to securely consume the remote AI models.

Table of content

Motivation

Today, most AI tools offer no privacy by design mechanisms, so when data is sent to be analysed by third parties, the data is exposed to malicious usage or potential leakage.

We illustrate it below with the use of AI for voice assistants. Audio recordings are often sent to the Cloud to be analysed, leaving conversations exposed to leaks and uncontrolled usage without users’ knowledge or consent.

Currently, even though data can be sent securely with TLS, some stakeholders in the loop can see and expose data : the AI company renting the machine, the Cloud provider or a malicious insider.

Before / after BlindAI

By using BlindAI, data remains always protected as it is only decrypted inside a Trusted Execution Environment, called an enclave, whose contents are protected by hardware. While data is in clear inside the enclave, it is inaccessible to the outside thanks to isolation and memory encryption. This way, data can be processed, enriched, and analysed by AI, without exposing it to external parties.

Gradio Demo

You can test and see how BlindAI secures AI application through our hosted demo of GPT2, built using Gradio.

In this demo, you can see how BlindAI works and make sure that your data is protected. Thanks to the attestation mechanism, even before sending data, our Python client will check that:

  • We are talking to a secure enclave with the hardware protection enabled.
  • The right code is loaded inside the enclave, and not a malicious one.

You can find more on secure enclaves attestation here.

GPT2 demo

Quick tour

BlindAI allows you to easily and quickly deploy your AI models with privacy, all in Python. To interact with an AI model hosted on a remote secure enclave, we provide the blindai.client API. This client will:

  • check that we are talking to a genuine secure enclave with the right security features
  • upload an AI model that was previously converted to ONNX
  • query the model securely

BlindAI is configured by default to connect to our managed Cloud backend to make it easy for users to upload and query models inside our secure enclaves. Even though we managed users AI models, thanks to the protection provided by the use of secure enclaves, data and models sent to our Cloud remain private.

You can also deploy BlindAI on your own infra.

Installing BlindAI

BlindAI can easily be installed from PyPI:

pip install blindai

This package is enough for the deployment and querying of models on our managed infrastructure. For on-premise deployment, you will have to deploy our Docker images.

Querying a GPT2

We can see how it works with our GPT2 model for text generation. It is already loaded inside our managed Cloud, so we will simply need to query it. We will be using the transformers library for tokenizing.

import blindai
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
example = "I like the Rust programming language because"

def get_example_inputs(example, tokenizer):
  # Detailled tokenizing here https://gist.github.com/dhuynh95/4357aec425bd30fbb41db0bc6ce0f8b2
  ...

input_list = get_example_inputs([example], tokenizer)

# Connect to a remote model. If security checks fail, an exception is raised
with blindai.Connection() as client:
  # Send data to the GPT2 model
  response = client.predict("gpt2", input_list)

example = tokenizer.decode(response.output[0].as_torch(), skip_special_tokens=True)

# We can see how GPT2 completed our sentence 🦀
>>> example
"I like the Rust programming language because it's easy to write and maintain."

Uploading a ResNet18

The model in the GPT2 example had already been loaded by us, but BlindAI also allows you to upload your own models to our managed Cloud solution.

You can find a Colab notebook showing how to deploy and query a ResNet18 on BlindAI.

To be able to upload your model to our Cloud, you will need to first register to get an API key.

Once you have the API key, you just have to provide it to our backend.

import torch
import blindai

# Get the model and export it locally in ONNX format
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)
dummy_inputs = torch.zeros(1,3,224,224)
torch.onnx.export(model, dummy_inputs, "resnet18.onnx")

# Upload the ONNX file along with specs and model name
with blindai.connect(api_key=...) as client:
    client.upload_model(
      model="resnet18.onnx",
    )

The first block of code pulls a model from PyTorch Hub, and export it in ONNX format. Because tracing is used, we need to provide a dummy input for the model to know the shape of inputs used live.

Before uploading, we need to provide information on the expected inputs and outputs.

Finally, we can connect to the managed backend and upload the model. You can provide a model name to know which one to query, for instance with model_name="resnet18". Because we have already uploaded a model with the name "resnet18", you should not try to upload a model with that exact name as it is already taken on our main server.

Querying a ResNet18

Now we can consume this model securely. We can now have a ResNet18 analyze an image of our dog, without showing the image of the dog in clear.

Dog to analyze

We will first pull the dog image, and preprocess it before sending it our enclave. The code is similar to PyTorch ResNet18 example:

# Source: https://pytorch.org/hub/pytorch_vision_resnet/
import blindai
import urllib
from PIL import Image
from torchvision import transforms

# Download an example image from the pytorch website
url, filename = ("https://github.com/pytorch/hub/raw/master/images/dog.jpg", "dog.jpg")
try: urllib.URLopener().retrieve(url, filename)
except: urllib.request.urlretrieve(url, filename)

# sample execution (requires torchvision)
input_image = Image.open(filename)
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(input_image)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

Now we that we have the input tensor, we simply need to send it to the pre-uploaded ResNet18 model inside our secure enclave:

with blindai.connect() as client:
  # Send data to the GPT2 model
  response = client.predict("resnet18", input_batch)

>>> response.output[0].as_numpy().argmax()

On-premise deployment

If you do not wish to use our managed Cloud, it is possible to deploy BlindAI yourself. We provide Docker images to help you with the deployment.

To run our solution with the full security features, you will need access to the proper hardware.

You can still try our solution without the right hardware through our simulation version. Be careful, as the simulation provides no added security guarantees, and is meant for tests only.

Which part of the AI workflow do we cover?

Position AI toolkit

BlindAI is currently a solution for AI model deployment. We suppose the model has already been trained and wants to be deployed but requires privacy guarantees for the data owners sending data to the model. We focus mostly on deep learning models, though inference of random forests can be covered by BlindAI.

This scenario often comes up once you have been able to train a model on a specific dataset, most likely on premise, like on biometric, medical or financial data, and now want to deploy it at scale as a Service to your users.

BlindAI can be seen as a variant of current serving solutions, like Nvidia Triton, Torchserve, TFserve, Kserve and so on. We provide the networking layer and the client SDK to consume the service remotely and securely, thanks to our secure AI backend.

Models covered by BlindAI

Here is a list of models BlindAI supports, the use cases it unlocks and articles to provide more context on each case. The articles are in preparation and we welcome all contributions to show how BlindAI can be used to deploy AI models with confidentiality!

Model name Model family Link to model Example use case Article Link to the notebook Inference time (ms) Hardware
DistilBERT BERT https://huggingface.co/docs/transformers/model_doc/distilbert Sentiment analysis Deploy Transformers models with confidentiality https://github.com/mithril-security/blindai/blob/master/examples/distilbert/BlindAI-DistilBERT.ipynb 28.435 Intel(R) Xeon(R) Platinum 8370C
COVID-Net-CXR-2 2D CNN https://github.com/lindawangg/COVID-Net Chest XRAY analysis for COVID detection Confidential medical image analysis with COVID-Net and BlindAI https://github.com/mithril-security/blindai/blob/master/examples/covidnet/BlindAI-COVID-Net.ipynb To be announced To be announced
Wav2vec2 Wav2vec https://huggingface.co/docs/transformers/model_doc/wav2vec2 Speech to text Build a privacy-by-design voice assistant with BlindAI https://github.com/mithril-security/blindai/blob/master/examples/wav2vec2/BlindAI-Wav2vec2.ipynb 617.04 Intel(R) Xeon(R) Platinum 8370C
Facenet Resnet https://github.com/timesler/facenet-pytorch Facial recognition To be announced https://github.com/mithril-security/blindai/blob/master/examples/facenet/BlindAI-Facenet.ipynb 47.135 Intel(R) Xeon(R) Platinum 8370C
YoloV5 Yolo https://github.com/ultralytics/yolov5 Object detection To be announced To be announced To be announced To be announced
Word2Vec Word2Vec https://spacy.io/usage/embeddings-transformers Document search To be announced To be announced To be announced To be announced
Neural Random Forest Random Forest https://arxiv.org/abs/1604.07143 Credit scoring To be announced To be announced To be announced To be announced
M5 network 1D CNN https://arxiv.org/pdf/1610.00087.pdf Speaker recognition To be announced To be announced To be announced To be announced

We will publish soon the scripts to run the benchmarks.

Documentation

To learn more about our project, do not hesitate to read our documentation.

What you can do with BlindAI

  • Easily deploy state-of-the-art models with confidentiality. Run any ONNX model, from BERT for text to ResNets for images, and much more.
  • Provide guarantees to third parties, for instance clients or regulators, that you are indeed providing data protection, through code attestation.
  • Explore different scenarios from confidential Sentiment analysis, to medical imaging with our pool of examples.

What you cannot do with BlindAI

  • Our solution aims to be modular but we have yet to incorporate tools for generic pre/post processing. Specific pipelines can be covered but will require additional handwork for now.
  • We do not cover training and federated learning yet, but if this feature interests you do not hesitate to show your interest through the roadmap or Discord channel.
  • The examples we provide are simple, and do not take into account complex mechanisms such as secure storage of confidential data with sealing keys, advanced scheduler for inference requests, or complex key management scenarios. If your use case involves more than what we show, do not hesitate to contact us for more information.

Current hardware support

Our solution currently leverages Intel SGX enclaves to protect data.

If you want to deploy our solution with real hardware protection and not only simulation, you can either deploy it on premise with the right hardware specs, or rent a machine adapted for Confidential Computing in the Cloud.

You can go to Azure Confidential Computing VMs to try, with our guides available here for deployment on DCsv2 and DCsv3.

What next

We intend to cover AMD SEV and Nitro Enclave in the future, which would make our solution available on GCP and AWS.

While we only cover deployment for now, we will start working on covering more complex pre/post processing pipelines inside enclaves, and training with Nvidia secure GPUs. More information about our roadmap can be found here.

FAQ

Q: How do I make sure data that I send is protected

A: We leverage secure enclaves to provide end-to-end protection. This means that even while your data is sent to someone else for them to apply an AI on it, your data remains protected thanks to hardware memory isolation and encryption.

We provide some information in our workshop Reconcile AI and privacy with Confidential Computing.

You can also have a look on our series Confidential Computing explained.

Q: Why should I trust you?

A: The client and server are open source. You can find them on our GitHub.

Thanks to this, you can verify yourself that the client does implement the right encryption mechanisms.

In addition, the server side code can be verified thanks to remote attestation. Our server code is open, and any version will generate a specific hash that you can reproduce yourself. You can compile BlindAI and get the hash of the code yourself.

Then you will just need to verify that this trustful source is indeed loaded in the remote server by using remote attestation.

Q: How much slowdown should we expect when using BlindAI?

A: We will provide a detailled benchmark soon. Usually you should see a negligeable slowdown with some simple models, and we have observed up to 30-40% slowdown for complex models.

Q: What is the maximal data/model size with BlindAI?

A: With the latest Intel Xeon Icelake 3rd Gen, the enclaves can now protect up to 1TB of code and data. This means that most models, even the biggest ones, can be made confidential with our solution.

Q: What do I need to do to use BlindAI?

A: The general workflow of BlindAI is described here. Basically you need to export your model in ONNX, upload it to the server and then you can send data to be analyzed securely.

Q: Can I use Python script with BlindAI?

A: We only support ONNX models for now, but most of the time preprocessing or postprocessing workflows can be expressed using ONNX operators. In that case you just have to include it in your model before exporting it to ONNX. You can see example for instance in the Wav2vec2 example.

Q: Do you do training or federated learning?

A: We will cover this topic soon. We have multy party learning framework leveraging secure enclave in development. You should learn about it in the near future.

Telemetry

BlindAI collects anonymous data regarding general usage, this allows us to understand how you are using the project. We only collect data regarding the execution mode (Hardware/Software) and the usage metrics.

This feature can be easily disabled, by settin up the environment variable BLINDAI_DISABLE_TELEMETRY to 1.

You can find more information about the telemetry in our documentation.

Disclaimer

BlindAI is still in alpha and is being actively developed. It is provided as is, use it at your own risk.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

blindai-0.5.6-cp310-cp310-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.10Windows x86-64

blindai-0.5.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ x86-64

blindai-0.5.6-cp39-cp39-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.9Windows x86-64

blindai-0.5.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.9manylinux: glibc 2.17+ x86-64

blindai-0.5.6-cp38-cp38-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.8Windows x86-64

blindai-0.5.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.8manylinux: glibc 2.17+ x86-64

blindai-0.5.6-cp37-cp37m-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.7mWindows x86-64

blindai-0.5.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.17+ x86-64

blindai-0.5.6-cp36-cp36m-win_amd64.whl (2.1 MB view details)

Uploaded CPython 3.6mWindows x86-64

blindai-0.5.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6mmanylinux: glibc 2.17+ x86-64

File details

Details for the file blindai-0.5.6-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: blindai-0.5.6-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.0

File hashes

Hashes for blindai-0.5.6-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 238079afd38ef1eb70cb181f40802d5a68132ee39ba290ca094409c4ebf9892b
MD5 393ef3928c368014ddbf2055fe305a00
BLAKE2b-256 dcbc6dfa05e976a58a2f400c963ced095f5e288ffe8f402605e5aa6f3d00ecaf

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blindai-0.5.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f7d34ccc7a97aa7461dd5a0769c60a55f96726f1dd9dacf1f67abeb0c25612e6
MD5 25cb946dde6b2c6745fbefd463aaaaa0
BLAKE2b-256 5a72c839a83b639a34288984098dc9b22976ce50e57ccf0adcc24191e3aedb51

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: blindai-0.5.6-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.0

File hashes

Hashes for blindai-0.5.6-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 25737f8f6608f9d09d26a55f6518e0e90dfe0366c94557d433536a684ff3c88e
MD5 0dd9618bfae342926569a751aa8e8cf4
BLAKE2b-256 262722d43dc43aaae3ff269a68ff8adf869ce27746211d5becf73b25ebf304b5

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blindai-0.5.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 89a0424c44241ea3539cc8288010f402fa9cd90018d99b440d59f7997cd845fa
MD5 3433594fc0e4876d677db6b235160c71
BLAKE2b-256 0a33969bb5ab931f3abfc2a1199904dce916ffdba76e449038283acdb41bc8a7

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: blindai-0.5.6-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.0

File hashes

Hashes for blindai-0.5.6-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 b76d85707e60af317a3db533e1458d2df896f673a868d2d108444ae9c526f1e9
MD5 6ab94f9db4d33914519ef87ef9e61f3f
BLAKE2b-256 e56e4fd4707af97da9630b32351fc6cdb48c745ba117111752950ddca371f7aa

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blindai-0.5.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b7a0db8f0e7c1106c25025926e040b67df16e7bbd70ed8503bacb426beeb807e
MD5 cbda5d52ceae2d1618466078d2450a65
BLAKE2b-256 288ce47cf4ac3379a5bb0f83856f33288c68ced8a28917f6cf627d7ae07a8016

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: blindai-0.5.6-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.0

File hashes

Hashes for blindai-0.5.6-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 797c53b9460b8f21bd02181a285e0d33f8a26db962f81759fee153d338063064
MD5 b3d26a9180eb7a79b76897a12e99e5bd
BLAKE2b-256 75f946090b9143a6ccdf26d96309bc7674d7fcbf3022989bc325949d77a1ced9

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blindai-0.5.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 32fa0aee2cb05686708ab9db08aaa6b9ea812f39f949edbcb03d0f9305fcf6e1
MD5 4a9e5d4cdfbe29ab2d9d7af0ecc53123
BLAKE2b-256 bb9feb397da9098b3d0e42c2d791c8f0e59ed4ec02122b8b8825444b53a05265

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: blindai-0.5.6-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.10.0

File hashes

Hashes for blindai-0.5.6-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 d5e97945c99aba3781d500d6bea896cecd4ac2efff413302978dbc518c6b90f2
MD5 a625a7963040f2df8cdc43c4a99cfdfb
BLAKE2b-256 e7bbe8df7fedd83fe91da4dc0fcfed22918acab07bffbc628e3032411f4d1804

See more details on using hashes here.

File details

Details for the file blindai-0.5.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for blindai-0.5.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ecb47ca058ffc15bc89d0f2072073337b694646ea48b3a1780e9af9e7c4f56f9
MD5 689938cab900bf087235a10d292e14d6
BLAKE2b-256 7c29626581bcfe8944e20592c1298e9ea0a04051fff1d83e7234bbb9e3140ac5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page