cloud-harvester 0.2.0

Cloud-agnostic resource harvesting with a unified resource model.

Cloud Harvester

Cloud-agnostic harvesting for AWS and Azure inventories. The collect() entry point fans out to built-in collectors across compute, containers/serverless, networking and edge, storage, databases, identity/security, and observability; limit scope with providers or inject your own boto3/Azure clients.

Every record is normalized into a Resource dataclass with fields like id, provider, kind, resource (service), name, region, status, network_id, subnetwork_id, tags, and the raw source payload for downstream use. Resources also include cloud-agnostic graph fields: scope, placements, relationships, and addresses.

Quickstart

import boto3
from azure.identity import ClientSecretCredential
from cloud_harvester import collect

# AWS: static credentials (replace with real values)
aws_session = boto3.Session(
    aws_access_key_id="FAKEAWSACCESSKEY123",
    aws_secret_access_key="FAKEAWSSECRETKEY456",
)

# Azure: service principal credentials (replace with real values)
azure_credential = ClientSecretCredential(
    tenant_id="00000000-0000-0000-0000-000000000000",
    client_id="11111111-1111-1111-1111-111111111111",
    client_secret="fake-azure-client-secret",
)
azure_subscription_id = "22222222-2222-2222-2222-222222222222"

# Collect from both providers with injected sessions/credentials
result = collect(
    providers=["aws", "azure"],
    aws_session=aws_session,
    azure_credential=azure_credential,
    azure_subscription_id=azure_subscription_id,
)

for res in result.resources:
    print(res.to_dict())

for err in result.errors:
    print(err.provider, err.collector, err.region, err.error_code, err.error_message)

collect() returns a CollectionResult with two fields:

• resources: List[Resource] — every resource successfully gathered.
• errors: List[CollectorError] — one entry per collector invocation that raised. Each entry carries the provider, collector name, region (AWS only; None for global/Azure collectors), an error_code parsed from the underlying exception when possible (botocore ClientError.response["Error"]["Code"] for AWS, azure.core.exceptions.HttpResponseError.error.code for Azure), and the raw error_message. Failures are still logged at ERROR, but the structured list is the canonical surface for callers that want to show users what went wrong (e.g. an AccessDenied on a specific collector/region).

AWS collection scans all enabled regions by default. To limit AWS scope, pass an explicit region list:

result = collect(
    providers=["aws"],
    aws_session=aws_session,
    aws_regions=["eu-west-1", "us-east-1"],
)

If aws_regions is omitted, Cloud Harvester discovers and scans all enabled AWS regions. You can also set CLOUD_HARVESTER_AWS_REGIONS to a comma-separated list.

Collectors run concurrently by default. To tune API pressure, pass max_workers or set CLOUD_HARVESTER_MAX_WORKERS:

result = collect(
    providers=["aws"],
    aws_session=aws_session,
    max_workers=8,
)

Development

Run the checks with:

make fix
make test

Logging

Cloud Harvester uses Python's standard logging module. Enable INFO logs to see provider progress, AWS region discovery, per-region collector progress, collector result counts, and collector failures:

import logging

logging.basicConfig(level=logging.INFO)

Credentials

• AWS: In the AWS console, create or reuse an IAM role/user with read permissions. Minimum managed policies to attach:

  • ReadOnlyAccess
  • AmazonEC2ReadOnlyAccess
  • AmazonEKSMCPReadOnlyAccess Generate access keys, then either:
  • Export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN (if temporary credentials), or
  • Store them in an AWS_PROFILE and point AWS_PROFILE/CLOUD_HARVESTER_AWS_PROFILE at it. Optionally set CLOUD_HARVESTER_AWS_REGIONS to limit collection to specific regions. If it is omitted, Cloud Harvester discovers and scans all enabled account regions.

• Azure: Create an App Registration (service principal) in Microsoft Entra ID and assign it the required RBAC roles on your subscription (Reader, Security Reader, Key Vault Reader). Capture:

  • tenant_id, client_id, client_secret from the service principal
  • subscription_id for the target subscription If Azure AD collectors are needed, add Microsoft Graph app permissions (e.g., Directory.Read.All) and have an admin grant consent.
    Either set AZURE_SUBSCRIPTION_ID / AZURE_TENANT_ID (or CLOUD_HARVESTER_*) or pass a ClientSecretCredential created from these values.