mypy-boto3-builder 0.4.0

Builder for mypy-boto3.

mypy_boto3

mypy-friendly type annotations for boto3.

Based on boto3_type_annotations by @alliefitter.

• mypy_boto3
  • Installation
  • Usage
    • Setup your IDE
    • VSCode
    • PyCharm
    • Other IDEs
    • Explicit type annotations
    • Pylint compatibility
  • How to build
    • Locally
    • With Docker image
  • Differences from boto3-type-annotations
  • Thank you
  • Sub-modules
    • Examples
    • List of all sub-modules

Installation

# install `boto3` type annotations
# for ec2, s3, rds, lambda, sqs, dynamo and cloudformation
pip install boto3-stubs[essential]

# install annotations for other services
pip install boto3-stubs[acm, apigateway]

# or install annotations for all services
# WARNING: this requires ~170 MB of space
pip install boto3-stubs[all]

Usage

• Install mypy and optionally enable it in your IDE
• Install boto3
• Now imports from boto3 and boto3.session have correct types and code autocomplete

import boto3

client = boto3.client("s3")

# IDE autocomplete suggests function name and arguments here
client.create_bucket(Bucket="bucket")

# (mypy) error: Missing positional argument "Key" in call to "get_object" of "Client"
client.get_object(Bucket="bucket")

# (mypy) error: Argument "Key" to "get_object" of "Client" has incompatible type "None"; expected "str"
client.get_object(Bucket="bucket", Key=None)

resource = boto3.Session(region_name="us-west-1").resource("s3")

# IDE autocomplete suggests function name and arguments here
bucket = resource.Bucket("bucket")

# (mypy) error: Unexpected keyword argument "key" for "upload_file" of "Bucket"
bucket.upload_file(Filename="my.txt", key="my-txt")

Setup your IDE

VSCode

• Install Official Python extension
• Install mypy
• Activate mypy checking in settings: "python.linting.mypyEnabled": true
• Install boto3-stubs with boto3 services you use
• Use explicit type annotations because function overload is not fully supported yet

PyCharm

• Install mypy plugin
• Install mypy
• Set path to mypy in mypy plugin settings
• Install boto3-stubs with boto3 services you use
• Use explicit type annotations because function overload is not fully supported yet

Official mypy plugin does not work for some reason at least for me. If you know how to setup it correctly, please let me know.

Do not install all services, it slows down PyCharm. Install only the ones you use.

Other IDEs

Please help me to extend this section.

Explicit type annotations

mypy correctly resolves types from boto3-stubs, but auto-complete in your IDE probably does not fully support overloaded functions, so methods and arguments auto-complete will not be very useful.

To help IDE to resolve types correctly, you can set types explicitly.

import boto3

from mypy_boto3.ec2 import Client, ServiceResource
from mypy_boto3.ec2.waiters import BundleTaskCompleteWaiter
from mypy_boto3.ec2.paginators import DescribeVolumesPaginator

session = boto3.session.Session(region_name="us-west-1")

ec2_client: Client = boto3.client("ec2", region_name="us-west-1")

ec2_resource: ServiceResource = session.resource("ec2")

bundle_task_complete_waiter: BundleTaskCompleteWaiter = ec2_client.get_waiter("bundle_task_complete")

describe_volumes_paginator: DescribeVolumesPaginator = ec2_client.get_paginator("describe_volumes")

# ec2_client, ec2_resource, bundle_task_complete_waiter and describe_volumes_paginator
# now have correct type so IDE automoplete for methods, arguments and return types
# works as expected

Pylint compatibility

It is totally safe to use TYPE_CHECKING flag in order to avoid boto3-stubs dependency in production. However, there is an issue in pylint that it complains about undefined variables. To fix it, set all types to Any in non-TYPE_CHECKING mode.

import boto3
from typing import Any, TYPE_CHECKING

if TYPE_CHECKING:
    from mypy_boto3.ec2 import Client, ServiceResource
    from mypy_boto3.ec2.waiters import BundleTaskCompleteWaiter
    from mypy_boto3.ec2.paginators import DescribeVolumesPaginator
else:
    Client = Any
    ServiceResource = Any
    BundleTaskCompleteWaiter = Any
    DescribeVolumesPaginator = Any

...

How to build

Locally

mypy-boto3 is built for the latest version of boto3. If you need type annotations for another version of boto3, you can use mypy-boto3-builder.

# Install preferred version of `boto3`
pip install boto3==1.10.18 botocore==1.13.18

# Install `mypy-boto3-builder`
pip install mypy-boto3-builder[black]

# Set output path to any directory
OUTPUT_PATH=`pwd`/output

# Build all packages
# You can specify required services explicitly like
# mypy_boto3_builder ${OUTPUT_PATH} -s ec2 s3
mypy_boto3_builder ${OUTPUT_PATH}

# Install custom `mypy-boto3` service packages
PACKAGES=${OUTPUT_PATH}/mypy_boto3_*
for PACKAGE in $PACKAGES
do
    cd ${PACKAGE}
    python setup.py install
done

# Install custom `mypy-boto3` and `boto3-stubs` packages
cd ${OUTPUT_PATH}/master_package
python setup.py install
cd ${OUTPUT_PATH}/boto3_stubs_package
python setup.py install

With Docker image

• Install Docker
• Pull latest mypy_boto3_builder version and tag it

docker pull docker.pkg.github.com/vemel/mypy_boto3/mypy_boto3_builder:latest
docker tag docker.pkg.github.com/vemel/mypy_boto3/mypy_boto3_builder:latest mypy_boto3_builder

• Generate stubs in output directory

mkdir output

# generate stubs for all services
docker run -v `pwd`/output:/output -ti mypy_boto3_builder

# generate stubs for s3 service
docker run -v `pwd`/output:/output -ti mypy_boto3_builder -s s3

# generate stubs for a specific boto3 version
docker run -e BOTO3_VERSION=1.10.18 BOTOCORE_VERSION=1.13.18 -v `pwd`/output:/output -ti mypy_boto3_builder

• Install packages from output directory as described above

Differences from boto3-type-annotations

• mypy compatibility
• Fully type annotated
• No need to set types explicitly (depends on your IDE)
• Generated types for return values and arguments
• Added ServiceResource sub-collections
• Support service-specific sub-modules
• Modules documentation
• Type annotations for return structures
• Correct annotations for client.get_waiter and client.get_paginator
• Helper functions for IDEs with no overload support (Hi, VSCode!)

Thank you

• Guys behind boto3-type-annotations, this package is based on top of their work
• black developers for awesome formatting tool
• mypy for doing all dirty work for us

Sub-modules

Examples

You can install any sub-modules using pip

# pip install boto3-stubs[<submodule_name>, ...]

# install ec2, s3 and sqs type annotations
pip install boto3-stubs[s3, ec2, sqs]

# install type annotations for all boto3 services
pip install boto3-stubs[all]

List of all sub-modules

• all - Type annotations for all boto3 services.
• essential - Type annotations for ec2, s3, rds, lambda, sqs, dynamodb and cloudformation services.
• acm - Type annotations for boto3 acm service.
• acm-pca - Type annotations for boto3 acm-pca service.
• alexaforbusiness - Type annotations for boto3 alexaforbusiness service.
• amplify - Type annotations for boto3 amplify service.
• apigateway - Type annotations for boto3 apigateway service.
• apigatewaymanagementapi - Type annotations for boto3 apigatewaymanagementapi service.
• apigatewayv2 - Type annotations for boto3 apigatewayv2 service.
• application-autoscaling - Type annotations for boto3 application-autoscaling service.
• application-insights - Type annotations for boto3 application-insights service.
• appmesh - Type annotations for boto3 appmesh service.
• appstream - Type annotations for boto3 appstream service.
• appsync - Type annotations for boto3 appsync service.
• athena - Type annotations for boto3 athena service.
• autoscaling - Type annotations for boto3 autoscaling service.
• autoscaling-plans - Type annotations for boto3 autoscaling-plans service.
• backup - Type annotations for boto3 backup service.
• batch - Type annotations for boto3 batch service.
• budgets - Type annotations for boto3 budgets service.
• ce - Type annotations for boto3 ce service.
• chime - Type annotations for boto3 chime service.
• cloud9 - Type annotations for boto3 cloud9 service.
• clouddirectory - Type annotations for boto3 clouddirectory service.
• cloudformation - Type annotations for boto3 cloudformation service.
• cloudfront - Type annotations for boto3 cloudfront service.
• cloudhsm - Type annotations for boto3 cloudhsm service.
• cloudhsmv2 - Type annotations for boto3 cloudhsmv2 service.
• cloudsearch - Type annotations for boto3 cloudsearch service.
• cloudsearchdomain - Type annotations for boto3 cloudsearchdomain service.
• cloudtrail - Type annotations for boto3 cloudtrail service.
• cloudwatch - Type annotations for boto3 cloudwatch service.
• codebuild - Type annotations for boto3 codebuild service.
• codecommit - Type annotations for boto3 codecommit service.
• codedeploy - Type annotations for boto3 codedeploy service.
• codepipeline - Type annotations for boto3 codepipeline service.
• codestar - Type annotations for boto3 codestar service.
• codestar-notifications - Type annotations for boto3 codestar-notifications service.
• cognito-identity - Type annotations for boto3 cognito-identity service.
• cognito-idp - Type annotations for boto3 cognito-idp service.
• cognito-sync - Type annotations for boto3 cognito-sync service.
• comprehend - Type annotations for boto3 comprehend service.
• comprehendmedical - Type annotations for boto3 comprehendmedical service.
• config - Type annotations for boto3 config service.
• connect - Type annotations for boto3 connect service.
• cur - Type annotations for boto3 cur service.
• datapipeline - Type annotations for boto3 datapipeline service.
• datasync - Type annotations for boto3 datasync service.
• dax - Type annotations for boto3 dax service.
• devicefarm - Type annotations for boto3 devicefarm service.
• directconnect - Type annotations for boto3 directconnect service.
• discovery - Type annotations for boto3 discovery service.
• dlm - Type annotations for boto3 dlm service.
• dms - Type annotations for boto3 dms service.
• docdb - Type annotations for boto3 docdb service.
• ds - Type annotations for boto3 ds service.
• dynamodb - Type annotations for boto3 dynamodb service.
• dynamodbstreams - Type annotations for boto3 dynamodbstreams service.
• ec2 - Type annotations for boto3 ec2 service.
• ec2-instance-connect - Type annotations for boto3 ec2-instance-connect service.
• ecr - Type annotations for boto3 ecr service.
• ecs - Type annotations for boto3 ecs service.
• efs - Type annotations for boto3 efs service.
• eks - Type annotations for boto3 eks service.
• elasticache - Type annotations for boto3 elasticache service.
• elasticbeanstalk - Type annotations for boto3 elasticbeanstalk service.
• elastictranscoder - Type annotations for boto3 elastictranscoder service.
• elb - Type annotations for boto3 elb service.
• elbv2 - Type annotations for boto3 elbv2 service.
• emr - Type annotations for boto3 emr service.
• es - Type annotations for boto3 es service.
• events - Type annotations for boto3 events service.
• firehose - Type annotations for boto3 firehose service.
• fms - Type annotations for boto3 fms service.
• forecast - Type annotations for boto3 forecast service.
• forecastquery - Type annotations for boto3 forecastquery service.
• fsx - Type annotations for boto3 fsx service.
• gamelift - Type annotations for boto3 gamelift service.
• glacier - Type annotations for boto3 glacier service.
• globalaccelerator - Type annotations for boto3 globalaccelerator service.
• glue - Type annotations for boto3 glue service.
• greengrass - Type annotations for boto3 greengrass service.
• groundstation - Type annotations for boto3 groundstation service.
• guardduty - Type annotations for boto3 guardduty service.
• health - Type annotations for boto3 health service.
• iam - Type annotations for boto3 iam service.
• importexport - Type annotations for boto3 importexport service.
• inspector - Type annotations for boto3 inspector service.
• iot - Type annotations for boto3 iot service.
• iot-data - Type annotations for boto3 iot-data service.
• iot-jobs-data - Type annotations for boto3 iot-jobs-data service.
• iot1click-devices - Type annotations for boto3 iot1click-devices service.
• iot1click-projects - Type annotations for boto3 iot1click-projects service.
• iotanalytics - Type annotations for boto3 iotanalytics service.
• iotevents - Type annotations for boto3 iotevents service.
• iotevents-data - Type annotations for boto3 iotevents-data service.
• iotthingsgraph - Type annotations for boto3 iotthingsgraph service.
• kafka - Type annotations for boto3 kafka service.
• kinesis - Type annotations for boto3 kinesis service.
• kinesis-video-archived-media - Type annotations for boto3 kinesis-video-archived-media service.
• kinesis-video-media - Type annotations for boto3 kinesis-video-media service.
• kinesisanalytics - Type annotations for boto3 kinesisanalytics service.
• kinesisanalyticsv2 - Type annotations for boto3 kinesisanalyticsv2 service.
• kinesisvideo - Type annotations for boto3 kinesisvideo service.
• kms - Type annotations for boto3 kms service.
• lakeformation - Type annotations for boto3 lakeformation service.
• lambda - Type annotations for boto3 lambda service.
• lex-models - Type annotations for boto3 lex-models service.
• lex-runtime - Type annotations for boto3 lex-runtime service.
• license-manager - Type annotations for boto3 license-manager service.
• lightsail - Type annotations for boto3 lightsail service.
• logs - Type annotations for boto3 logs service.
• machinelearning - Type annotations for boto3 machinelearning service.
• macie - Type annotations for boto3 macie service.
• managedblockchain - Type annotations for boto3 managedblockchain service.
• marketplace-entitlement - Type annotations for boto3 marketplace-entitlement service.
• marketplacecommerceanalytics - Type annotations for boto3 marketplacecommerceanalytics service.
• mediaconnect - Type annotations for boto3 mediaconnect service.
• mediaconvert - Type annotations for boto3 mediaconvert service.
• medialive - Type annotations for boto3 medialive service.
• mediapackage - Type annotations for boto3 mediapackage service.
• mediapackage-vod - Type annotations for boto3 mediapackage-vod service.
• mediastore - Type annotations for boto3 mediastore service.
• mediastore-data - Type annotations for boto3 mediastore-data service.
• mediatailor - Type annotations for boto3 mediatailor service.
• meteringmarketplace - Type annotations for boto3 meteringmarketplace service.
• mgh - Type annotations for boto3 mgh service.
• mobile - Type annotations for boto3 mobile service.
• mq - Type annotations for boto3 mq service.
• mturk - Type annotations for boto3 mturk service.
• neptune - Type annotations for boto3 neptune service.
• opsworks - Type annotations for boto3 opsworks service.
• opsworkscm - Type annotations for boto3 opsworkscm service.
• organizations - Type annotations for boto3 organizations service.
• personalize - Type annotations for boto3 personalize service.
• personalize-events - Type annotations for boto3 personalize-events service.
• personalize-runtime - Type annotations for boto3 personalize-runtime service.
• pi - Type annotations for boto3 pi service.
• pinpoint - Type annotations for boto3 pinpoint service.
• pinpoint-email - Type annotations for boto3 pinpoint-email service.
• pinpoint-sms-voice - Type annotations for boto3 pinpoint-sms-voice service.
• polly - Type annotations for boto3 polly service.
• pricing - Type annotations for boto3 pricing service.
• qldb - Type annotations for boto3 qldb service.
• qldb-session - Type annotations for boto3 qldb-session service.
• quicksight - Type annotations for boto3 quicksight service.
• ram - Type annotations for boto3 ram service.
• rds - Type annotations for boto3 rds service.
• rds-data - Type annotations for boto3 rds-data service.
• redshift - Type annotations for boto3 redshift service.
• rekognition - Type annotations for boto3 rekognition service.
• resource-groups - Type annotations for boto3 resource-groups service.
• resourcegroupstaggingapi - Type annotations for boto3 resourcegroupstaggingapi service.
• robomaker - Type annotations for boto3 robomaker service.
• route53 - Type annotations for boto3 route53 service.
• route53domains - Type annotations for boto3 route53domains service.
• route53resolver - Type annotations for boto3 route53resolver service.
• s3 - Type annotations for boto3 s3 service.
• s3control - Type annotations for boto3 s3control service.
• sagemaker - Type annotations for boto3 sagemaker service.
• sagemaker-runtime - Type annotations for boto3 sagemaker-runtime service.
• savingsplans - Type annotations for boto3 savingsplans service.
• sdb - Type annotations for boto3 sdb service.
• secretsmanager - Type annotations for boto3 secretsmanager service.
• securityhub - Type annotations for boto3 securityhub service.
• serverlessrepo - Type annotations for boto3 serverlessrepo service.
• service-quotas - Type annotations for boto3 service-quotas service.
• servicecatalog - Type annotations for boto3 servicecatalog service.
• servicediscovery - Type annotations for boto3 servicediscovery service.
• ses - Type annotations for boto3 ses service.
• shield - Type annotations for boto3 shield service.
• signer - Type annotations for boto3 signer service.
• sms - Type annotations for boto3 sms service.
• sms-voice - Type annotations for boto3 sms-voice service.
• snowball - Type annotations for boto3 snowball service.
• sns - Type annotations for boto3 sns service.
• sqs - Type annotations for boto3 sqs service.
• ssm - Type annotations for boto3 ssm service.
• stepfunctions - Type annotations for boto3 stepfunctions service.
• storagegateway - Type annotations for boto3 storagegateway service.
• sts - Type annotations for boto3 sts service.
• support - Type annotations for boto3 support service.
• swf - Type annotations for boto3 swf service.
• textract - Type annotations for boto3 textract service.
• transcribe - Type annotations for boto3 transcribe service.
• transfer - Type annotations for boto3 transfer service.
• translate - Type annotations for boto3 translate service.
• waf - Type annotations for boto3 waf service.
• waf-regional - Type annotations for boto3 waf-regional service.
• workdocs - Type annotations for boto3 workdocs service.
• worklink - Type annotations for boto3 worklink service.
• workmail - Type annotations for boto3 workmail service.
• workmailmessageflow - Type annotations for boto3 workmailmessageflow service.
• workspaces - Type annotations for boto3 workspaces service.
• xray - Type annotations for boto3 xray service.