Skip to main content

RedBrick AI and AWS Sagemaker integration

Project description

redbrick-sagemaker

This package is an integration between RedBrick AI and AWS sagemaker to allow end-to-end Active Learning on computer vision datasets.

The objective of Active Learning is to label your data in order of information gain to your model. Following this strategy can drastically reduce the amount of data you have to label by only labeling those images that help your model improve.

This package will help you run a full end-to-end process where you will be able to iteratively label your dataset and train your model in true Active Learning fashion.

Setup

Install the redbrick_sagemaker package

pip install redbrick_sagemaker

Create an s3 bucket

You will need to create an s3 bucket where your training and model files will get stored. Please follow this tutorial to create an s3 bucket through the CLI, SDK or AWS console.

NOTE: Create your bucket with the sagemaker part of the name, so that you can conditionally give it access. For example - redbrick-sagemaker-bucket.

Create sagemaker role

NOTE: We reccommend you run the redbrick_sagemaker package within a Sagemaker Notebook instance. If you do this, you won't have to create a Sagemaker Execution Role.

If you're running redbrick_sagemaker outside of a Sagemaker Notebook, you need to create a Sagemaker Execution role to allow sagemaker to perform operations on your behalf. Please see this tutorial for creating a AmazonSageMakerFullAccess role. After creating the role, make a note of the ARN.

Use

Standard RedBrick AI set up:

api_key="TODO"
org_id="TODO"
project_id="TOOD"

# The bucket where sagemaker will read/write predictions and training input/outputs.
s3_bucket_name="TODO"
s3_bucket_prefix="TODO"

# OPTIONAL: Add the sagemaker execution role you created here.
# only required if you are running redbrick_sagemaker outside of an AWS sagemaker notebook instance.
# If runnning inside a sagemaker notebook, set role=None
role="TODO"

Create a RedBrick AI Active Learning object:

import redbrick_sagemaker

active_learner = redbrick_sagemaker.ActiveLearner(
    api_key,
    org_id,
    project_id,
    s3_bucket=bucket,
    s3_bucket_prefix=bucket_prefix,
    iam_role=role
)

Begin an Active Learning cycle. Running this for the first time will start a hyperparameter optimization job to train your model.

active_learner.run()

Check on the status of your hyperparameter job.

active_learner.describe()

You can have a detailed view on the progress of your hyperparameter tuning job on your AWS console.

Sagemaker AWS console.

Once your hyperparameter job is complete, you can re-run to perform inference and update Active Learning priorities.

active_learning.run()

If your hyperparameter job is still processing, but there is a model job that has completed, you can force run an inference.

active_learning.run(force_run=True)

If you want to run training, and inference in one go synchronously, you can simply do:

active_learning.run(wait=True)

Please see the flowchart below for an explanation of the different states and flows.

RedBrick Sagemaker active learning flow.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

redbrick_sagemaker-0.0.4.tar.gz (19.5 kB view details)

Uploaded Source

Built Distribution

redbrick_sagemaker-0.0.4-py3-none-any.whl (21.2 kB view details)

Uploaded Python 3

File details

Details for the file redbrick_sagemaker-0.0.4.tar.gz.

File metadata

  • Download URL: redbrick_sagemaker-0.0.4.tar.gz
  • Upload date:
  • Size: 19.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.2 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for redbrick_sagemaker-0.0.4.tar.gz
Algorithm Hash digest
SHA256 41b497f678c03706b52cfd4c66fdefb303f2268a7ff31a8af4a48b1a91d782a4
MD5 81e9ea35c3d1053ad512f06936697fce
BLAKE2b-256 1db7dbfe9114bb23589e0f6abb8f36d041e59e60e6894573d3bdef29550f7134

See more details on using hashes here.

File details

Details for the file redbrick_sagemaker-0.0.4-py3-none-any.whl.

File metadata

  • Download URL: redbrick_sagemaker-0.0.4-py3-none-any.whl
  • Upload date:
  • Size: 21.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.2 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for redbrick_sagemaker-0.0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 b485c289d46ed1a644c936a4718b54bbce6101f9a538f0298ec510f7a20dfdfb
MD5 ff46f2a9b616732af1b56403f6cc8ab2
BLAKE2b-256 52e30cde8187fda1cb956a9b1b085f9b7250b8ee2eddb3a75fedc2657388b987

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page