botoplier 0.0.1rc2

A powerful yet simple way to use AWS APIs, built on boto3

botoplier 🚀

Enhance your AWS API interactions using botoplier, a Python library built on top of boto3 and botocore, facilitating simplified API operations through automated fetch, session management, and data handling features.

Features

• Response un-nesting: Specify the nested attribute you're interested in, and let botoplier pull it from the results for you.
• Seamless pagination: Pagination is handled transparently - all results will be fetched when pagination is required.
• Multi-session management: Convenient management of multiple AWS sessions.
• Asyncio support: Can be used synchronously or asynchronously with asyncio [1].
• Simple It's not complex!
• • Way under 1K LOC. You can read it all in an hour.

[1] : botocore still does not support asynchronous I/O per-se, so the concurrency support relies on asyncio thread pools at this time.

Installation

Botoplier is distributed on PyPI, and can be installed with pip, poetry, pdm, or any other Python package manager:

pip install botoplier

smart_query

Basic examples

from botoplier.sync.smart_query import smart_query

# Describe all EC2 instances across multiple sessions and regions
result = smart_query("ec2", "describe_instances")

NB: The API names are kebab-cased while the operation names are snake_cased, which is in line with boto3.

Result shape and un-nesting

The smart_query function automatically processes and simplifies the response structure from AWS services. It neatly organizes deeply nested data into a more readable and usable format.

Pagination detection

smart_query automatically detects and handles API pagination, ensuring that all data across pages is fetched and returned as a single cohesive response.

Multi-session / Parallelism

botoplier was initially developed not just to reduce boilerplate around boto3 calls, but also to simplify querying several accounts at once.

Creating / Configuring multi-session

You can create and configure multi-sessions to simultaneously work across different AWS accounts and regions:

from botoplier.sync.sessions import make_sessions

sessions = make_sessions({"account1": "123456789012"}, ["us-west-2", "us-east-1"], {"account1": "roleName"})

The above assumes that the "ambient" AWS account is able to assume the provided roles in the matching accounts.

make_sessions produces a dictionary keyed by "session key" strings, for each of which a DecoratedSession is attached.

The sessions are cached on-disk for as long as the credentials are valid.

Bring your own sessions

DecoratedSession

A DecoratedSession is essentially a regular botocore session, with a smidge of extra information that allows the recompute "session keys" from it. A "session key" is a unique string identifier for the session, which is somewhat human readable, and machine parsable, like "eu-west-3-prd".

Using your own sessions

It's quite possible your AWS authentication setup is more complicated than ours, and that our default make_sessions() helper is inappropriate as a result. You can bring your own sessions by passing any dict[SessionKey, DecoratedSession].

from botoplier.types import DecoratedSession

# the botocore_session argument is key here
# NB: DecoratedSession inherits from boto3.Session, so you may use any of its init parameters
ds = DecoratedSession(botocore_session=..., region_name="eu-west-3")
ds.set_arn_from_sts()  # You can also do this manually using .set_arn()

Parallel calls with gather_dict

Use gather_dict to await a dictionary of future results. It works nicely with session dictionaries such as the ones provided by make_sessions like so:

from botoplier.asyncio import smart_query, make_sessions, gather_dict


async def in_async():
    sessions = await make_sessions({"account1": "123456789012"}, ["us-west-2", "us-east-1"], {"account1": "roleName"})
    results = await gather_dict({
        sk: smart_query("s3", "list_buckets", session=session)
        for sk, session in sessions.items()
    })

smart_query, make_sessions and gather_dict is the botoplier trifecta. There is not much more to it - it's just boto3, but nicer.

Environment variables

Changing the environment variable prefix

If you're writing a tool that uses botoplier, it is sometimes more expedient to change the recognized prefix for environment variables, which defaults to BOTOPLIER.

In your src/__init__.py, or before any botoplier import:

import botoplier.config

botoplier.config.PREFIX = "MYTOOL"

CLI Usage

Botoplier ships with a simple CLI to easily inspect the output shape of AWS APIs.

Discovering default output shape

$ AWS_DEFAULT_REGION=eu-west-3 botoplier output-shape ec2 describe_host_reservations

DescribeHostReservationsResult (structure)
└── HostReservationSet: HostReservationSet (list)
    └── HostReservation (structure)
        ├── Count: Integer (integer)
        ├── CurrencyCode: CurrencyCodeValues (string)
        ├── Duration: Integer (integer)
        ├── End: DateTime (timestamp)
        ├── HostIdSet: ResponseHostIdSet (list)
        │   └── String (string)
        ├── HostReservationId: HostReservationId (string)
        ├── HourlyPrice: String (string)
        ├── InstanceFamily: String (string)
        ├── OfferingId: OfferingId (string)
        ├── PaymentOption: PaymentOption (string)
        ├── Start: DateTime (timestamp)
        ├── State: ReservationState (string)
        ├── Tags: TagList (list)
        │   └── Tag (structure)
        │       ├── Key: String (string)
        │       └── Value: String (string)
        └── UpfrontPrice: String (string)

Tree size: 21
First branching node:  .HostReservationSet[]

The output here indicates the default un-nesting smart_query will adopt when calling this API. This expression means a list of HostReservation dicts will be returned, avoiding the need to manually read through the HostReservationSet field present at the top level of the response.

Discovering the shape with a hint

Doing a similar invocation with an extra argument explores the shape when giving a different unnest= parameter.

$ AWS_DEFAULT_REGION=eu-west-3 botoplier output-shape ec2 describe_host_reservations Tags

DescribeHostReservationsResult (structure)
└── HostReservationSet: HostReservationSet (list)
    └── HostReservation (structure)
        ├── Count: Integer (integer)
        ├── CurrencyCode: CurrencyCodeValues (string)
        ├── Duration: Integer (integer)
        ├── End: DateTime (timestamp)
        ├── HostIdSet: ResponseHostIdSet (list)
        │   └── String (string)
        ├── HostReservationId: HostReservationId (string)
        ├── HourlyPrice: String (string)
        ├── InstanceFamily: String (string)
        ├── OfferingId: OfferingId (string)
        ├── PaymentOption: PaymentOption (string)
        ├── Start: DateTime (timestamp)
        ├── State: ReservationState (string)
        ├── Tags: TagList (list)
        │   └── Tag (structure)
        │       ├── Key: String (string)
        │       └── Value: String (string)
        └── UpfrontPrice: String (string)

Tree size: 21
First branching node:  .HostReservationSet[]
Key Tags lookup path:  .HostReservationSet[].Tags 

Note the Key Tags lookup path at the end. One can understand that by passing unnest="Tags" to smart_query, one will receive a list of list of tag dicts as a result.

Contributing

We value contributions from the community! Please read our Contributing Guidelines and our Code of Conduct for information on how to get involved.

Quick setup guide

We use mise and PDM for local development.

With these installed:

mise install
pdm install
pdm run botoplier --help # Test the installation of the CLI
pdm build # Build the package wheel and sdist
pdm run pytest # Run the tests
# etc...

We use pre-commit hooks to ensure code quality.

Supported versions

We currently support Python 3.9, 3.10, 3.11 and 3.12.

We may drop the support for a Python version before its end of life, to keep the codebase up to date with the latest Python features: i.e.: we will endeavor to support either the last 3 or 4 stable Python releases.

We don't plan to support earlier versions or different runtimes, but contributions are welcome.

Roadmap

Syntax and advance type-hinting tricks are areas we'd like to improve.

The multi-region / multi-account support takes some of the tedium away. We're open to improving this aspect further should an interested party come up with something.

Outside of this, we're happy with the limited scope of this library and are unlikely to accept contributions widening its scope.

TODO

• Replace jq by jmespath for un-nesting
• Rename smart_query module to query module, to avoid confusion with smart_query function
• Proper online documentation, with published object inventory