tap-aws-cost-explorer

tap-aws-cost-explorer is a Singer tap for AWS Cost Explorer, built with the Meltano Singer SDK.

It extracts daily or monthly AWS costs, grouped by account, service, usage type, region, cost allocation tag or cost category. Run against an AWS Organizations management account, it covers every member account in the organization.

This is the MeltanoLabs variant, forked from albert-marrero/tap-aws-cost-explorer by Albert Marrero.

Installation

uv tool install meltanolabs-tap-aws-cost-explorer

Prerequisites

The tap needs AWS credentials that can call:

To cover every account in an organization, use credentials (or role_arn) for the organization's management account. Cost Explorer there returns costs for every member account, split out by the LINKED_ACCOUNT dimension. Credentials for a member account only return that account's costs, and can't call organizations:ListAccounts, so deselect the accounts stream.

Cost Explorer API requests are billed by AWS, at $0.01 per request at the time of writing. The tap makes one request per page of results per stream, so each selected stream adds to the cost of a run.

Settings

A full list of settings and capabilities is available by running:

tap-aws-cost-explorer --about

Streams

Each cost* stream has one row per time period, group and metric:

The accounts stream lists the accounts in the organization, with their id, arn, email, name, state, status, joined_method and joined_timestamp. Join it to linked_account to name the accounts in cost reports.

Custom reports

Cost Explorer can group a query by at most two things, so each report is its own stream. Add a report for any other combination of dimensions, cost allocation tags (TAG) or cost categories (COST_CATEGORY):

config:
  reports:
  - name: cost_by_team
    group_by:
    - type: TAG
      key: team
    - type: DIMENSION
      key: SERVICE
    # Optional: a Cost Explorer filter expression, as in the GetCostAndUsage API.
    filter:
      Dimensions:
        Key: RECORD_TYPE
        Values: [Usage]

Group columns are named after the key in snake case, with tags prefixed tag_ and cost categories cost_category_, so this report has tag_team and service columns. Costs that aren't tagged or categorized have a null tag or cost category value. A tag must be activated as a cost allocation tag before Cost Explorer can group by it.

Incremental replication

Every cost* stream is incremental on time_period_start. Each run starts lookback_days before the bookmark, because AWS revises recent amounts as usage is processed and while a month's bill is finalized. Rows read again replace the earlier ones by primary key. With MONTHLY granularity, each run starts on the 1st of the month.

To backfill, reset the stream's state and set start_date to the first day you need, within the last 14 months.

Usage

You can run tap-aws-cost-explorer by itself or in a pipeline using Meltano.

tap-aws-cost-explorer --version
tap-aws-cost-explorer --help
tap-aws-cost-explorer --config CONFIG --discover > ./catalog.json

Developer resources

Initialize your development environment

uv sync

Create and run tests

The tests in tests/test_streams.py run against stubbed AWS responses. The standard SDK tests in tests/test_core.py call the live API, and run only when TAP_AWS_COST_EXPLORER_ACCESS_KEY and TAP_AWS_COST_EXPLORER_SECRET_KEY are set.

uv run pytest

Testing with Meltano

uv tool install meltano
meltano install
meltano invoke tap-aws-cost-explorer --version
meltano run tap-aws-cost-explorer target-jsonl

See the dev guide for more on how to use the SDK to develop taps and targets.

License

MIT