AWS Resource Inventory

A comprehensive AWS multi-service scanner with tag-based filtering, parallel processing, advanced logging capabilities, and optimization features. This tool enables efficient discovery and analysis of AWS resources across multiple regions and services with intelligent caching, rich output formats, and detailed AWS API tracing.

Features

• Scans eight AWS services — EC2, S3, ECS, EFS, VPC, RDS, ELB, and Auto Scaling — across multiple regions concurrently, with or without tag filters.
• Discovers resources from 100+ AWS services via the Resource Groups Tagging API (--all-services, requires a tag filter).
• Table, JSON, and Markdown output; results are rendered in the terminal and written to a file for further processing.
• Result caching with a 10-minute TTL (--cache / --no-cache).
• Configurable parallelism per region (--max-workers) and per service (--service-workers), dry-run preview, continuous refresh mode, and graceful Ctrl+C handling.
• Debug traces (--debug), full AWS API tracing (--verbose), and custom log files (--log-file) — see Logging Architecture.

🏗️ Supported AWS Services

📚 Architecture Details: For detailed information about the scanning architecture and service implementation patterns, see Architecture Documentation.

📋 Prerequisites

Before installing the AWS Resource Inventory, ensure you have the following dependencies:

Required Software

• Python 3.10+ - Runtime environment
• Poetry - Python dependency management and packaging
• pip - Python package installer
• AWS CLI - AWS command line interface for authentication
• pre-commit - Git hooks framework (for development)

AWS Configuration

• Valid AWS credentials configured via:
  • AWS SSO (aws sso login)
  • AWS CLI (aws configure)
  • Environment variables
  • IAM roles (for EC2/containers)

Required IAM Permissions

The scanner is read-only. This policy covers every API call it makes — the eight service scanners, the credential check (sts), and the tag scan (tag:GetResources, only exercised with --tag-key/--tag-value/ --all-services):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AwsResourceInventoryReadOnly",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity",
                "ec2:Describe*",
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:GetBucketTagging",
                "ecs:List*",
                "ecs:Describe*",
                "elasticloadbalancing:Describe*",
                "elasticfilesystem:Describe*",
                "rds:Describe*",
                "autoscaling:Describe*",
                "tag:GetResources"
            ],
            "Resource": "*"
        }
    ]
}

🚀 Installation

Automated Setup (Recommended)

The easiest way to get started is using our automated setup script:

# Clone the repository
git clone https://github.com/ishuar/aws-resource-inventory.git
cd aws-resource-inventory

# Run the automated setup script
./setup.sh

The setup script will:

1. ✅ Check and install Python 3.10+
2. ✅ Install Poetry (Python dependency manager)
3. ✅ Install pre-commit (Git hooks framework)
4. ✅ Set up pre-commit hooks
5. ✅ Install all project dependencies via Poetry
6. ✅ Run verification tests
7. ✅ Install AWS CLI (if not present)
8. ✅ Provide AWS configuration guidance

Manual Installation

If you prefer manual installation:

# Install dependencies (macOS with Homebrew)
brew install python3 poetry pre-commit awscli

# Install project dependencies
poetry install

# Set up pre-commit hooks
pre-commit install --install-hooks

# Verify installation
./run_quick_tests.sh

💻 Usage

Command Structure

All commands follow this pattern:

poetry run aws-inventory [GLOBAL OPTIONS] COMMAND [COMMAND OPTIONS]

Global options apply to all commands: --verbose / -v (AWS API tracing, use with --debug) and --log-file / -l (custom log file path). The main command is scan.

Basic Commands

# Display help and available commands
poetry run aws-inventory --help

# Display scan command help and options
poetry run aws-inventory scan --help

# Basic scan with default settings (all supported services)
poetry run aws-inventory scan --regions us-east-1,eu-west-1,eu-central-1,us-west-2

# Scan specific services
poetry run aws-inventory scan --service ec2

# Scan specific regions
poetry run aws-inventory scan --regions us-east-1,eu-west-1

Debug and Logging Options

# Enable debug mode for detailed execution traces
poetry run aws-inventory scan --debug --regions us-east-1

# Enable verbose AWS API tracing (requires --debug)
poetry run aws-inventory --verbose scan --debug --service ec2

# Custom log file for debug output
poetry run aws-inventory --log-file /tmp/my-scan.log scan --debug --regions us-east-1

# Combine verbose logging with custom log file
poetry run aws-inventory --verbose --log-file /tmp/aws-api-trace.log scan --debug --service ec2 --service s3

# Debug with dry run (no actual scanning)
poetry run aws-inventory --verbose scan --debug --dry-run --service vpc

Service-Specific Scanning

# Scan only EC2 resources
poetry run aws-inventory scan --service ec2

# Scan multiple services
poetry run aws-inventory scan --service ec2 --service s3 --service vpc

# Scan all built-in services in specific regions
poetry run aws-inventory scan --regions us-east-1,us-west-2

# Combine service and region filtering
poetry run aws-inventory scan --service ec2 --regions eu-central-1,eu-west-1

# Scan ALL AWS services using Resource Groups API (requires tags)
poetry run aws-inventory scan --all-services --tag-key Environment --tag-value Production

Tag-Based Filtering

# Filter by environment tag
poetry run aws-inventory scan --tag-key Environment --tag-value Production

# Filter by application tag
poetry run aws-inventory scan --tag-key app --tag-value web-server

# Filter by cost center in specific regions
poetry run aws-inventory scan --regions us-east-1 --tag-key CostCenter --tag-value Engineering

Output Formats

# Default table format (human-readable)
poetry run aws-inventory scan --format table

# JSON format for programmatic processing
poetry run aws-inventory scan --format json --output results.json

# Markdown format for documentation
poetry run aws-inventory scan --format md --output report.md

# Export filtered results to JSON
poetry run aws-inventory scan --tag-key Environment --tag-value Production --format json --output prod-resources.json

Advanced Options

# Dry run (preview without execution)
poetry run aws-inventory scan --dry-run --service ec2

# Disable caching for fresh data
poetry run aws-inventory scan --no-cache

# Configure worker threads for performance
poetry run aws-inventory scan --max-workers 10 --service-workers 6

# Continuous refresh mode with custom interval
poetry run aws-inventory scan --refresh --refresh-interval 30 --service ec2

# Debug mode with performance timing
poetry run aws-inventory --verbose scan --debug --max-workers 1 --service ec2

Real-World Examples

# Production infrastructure audit with comprehensive logging
poetry run aws-inventory --verbose --log-file prod-audit.log scan \
    --debug --tag-key Environment --tag-value Production \
    --format json --output production-audit.json

# Regional compliance check with detailed tracing
poetry run aws-inventory --verbose --log-file compliance-trace.log scan \
    --debug --regions eu-west-1,eu-central-1 \
    --service ec2 --format md --output eu-compliance-report.md

# Application-specific resource discovery across all AWS services
poetry run aws-inventory scan \
    --all-services --tag-key Application --tag-value MyApp \
    --format table --regions us-east-1

# Development environment troubleshooting with verbose logging
poetry run aws-inventory --verbose --log-file dev-debug.log scan \
    --debug --regions us-west-2 \
    --tag-key Environment --tag-value Development \
    --no-cache --dry-run

# Performance analysis with sequential processing
poetry run aws-inventory --verbose --log-file perf-analysis.log scan \
    --debug --max-workers 1 --service-workers 1 \
    --service ec2 --service s3 --regions us-east-1

🔧 Configuration

AWS Profile Configuration

# Set AWS profile
export AWS_PROFILE=your-profile-name

# Login via SSO
aws sso login --profile $AWS_PROFILE

# Verify credentials
aws sts get-caller-identity

Logging Configuration

The scanner features a comprehensive logging system with multiple configuration options:

# Global logging options (apply to all commands)
poetry run aws-inventory --verbose --log-file /path/to/logfile.log scan --debug

# Debug modes explained:
# --debug: Enable debug mode with rich console output and file logging
# --verbose: Enable AWS API tracing (requires --debug)
# --log-file: Custom log file path (default: .debug_logs/aws_scanner_debug_TIMESTAMP.log)

Logging levels: normal runs show progress and results; --debug adds execution traces, timing, and caller context; --verbose together with --debug adds full AWS API tracing including HTTP requests and responses.

Performance Tuning

• --max-workers (1-20) controls region-level parallelism; --service-workers (1-10) controls service-level parallelism.
• --cache speeds up repeated scans (10-minute TTL); --no-cache forces fresh data.

📁 Project Structure

aws-resource-inventory/
├── aws_scanner.py               # Scan orchestration across regions
├── cli.py                       # Command-line interface (typer)
├── setup.sh                     # Automated setup script
├── run_quick_tests.sh           # Smoke-test script
├── pyproject.toml               # Project configuration and dependencies
│
├── services/                    # One scanner module per AWS service
│   ├── registry.py              # Single source of truth: service → scanner + output processor
│   ├── ec2_service.py           # Instances, volumes, security groups, AMIs, snapshots
│   ├── s3_service.py            # Buckets (region-filtered, tag-enriched)
│   ├── ecs_service.py           # Clusters, services, task definitions, capacity providers
│   ├── efs_service.py           # File systems
│   ├── elb_service.py           # Load balancers, target groups, listeners, rules
│   ├── rds_service.py           # DB instances, clusters, snapshots (incl. Aurora)
│   ├── vpc_service.py           # VPC networking components
│   └── autoscaling_service.py   # ASGs, launch configurations, launch templates
│
├── aws_scanner_lib/             # Core library modules
│   ├── engine.py                # Shared scanning engine: pagination, concurrency, error policy
│   ├── records.py               # Resource — the typed record every output consumes
│   ├── clients.py               # The only boto3 client factory (pooling, adaptive retries)
│   ├── scan.py                  # Region/service scan orchestration + caching hooks
│   ├── outputs.py               # Table / JSON / Markdown output processing
│   ├── resource_groups_utils.py # Resource Groups Tagging API path (--all-services, tags)
│   ├── cache.py                 # Result caching with TTL
│   └── logging.py               # Unified logging with AWS API tracing
│
├── scripts/
│   └── e2e-diff.sh              # Before/after functional comparison against real AWS
│
├── docs/                        # Documentation
│   ├── adr/                     # Architecture decision records
│   ├── Architecture.md          # System architecture and design patterns
│   └── LOGGING_ARCHITECTURE.md  # Logging system documentation
│
└── tests/                       # Test suite — runs with ZERO AWS credentials (moto)

🧪 Testing

# Run the full suite (coverage is reported automatically)
poetry run pytest

# Run one test file
poetry run pytest tests/test_engine.py

# Quick smoke checks (help, dry-run, formats)
./run_quick_tests.sh

After merging a change that touches scan behaviour, verify against real AWS with the before/after comparison script:

scripts/e2e-diff.sh                        # compares origin/main~1 vs origin/main
scripts/e2e-diff.sh --tag-key team         # exercise the tag-scan path too

🐛 Troubleshooting

Common Issues

1. AWS Credentials: Ensure AWS credentials are properly configured

   aws sts get-caller-identity
   

2. Python Version: Verify Python 3.10+ is installed

   python3 --version
   

3. Dependencies: Reinstall dependencies if needed

   poetry install --no-cache
   

4. Permissions: Ensure your AWS user/role has necessary permissions for the services you're scanning

Debug and Logging Troubleshooting

The advanced logging system provides powerful debugging capabilities:

# Basic debug information
poetry run aws-inventory scan --debug --dry-run

# Verbose AWS API tracing for troubleshooting
poetry run aws-inventory --verbose --log-file debug-trace.log scan --debug --service ec2

# Check debug log files (automatically created)
ls .debug_logs/

# Monitor real-time logging
tail -f .debug_logs/aws_scanner_debug_*.log

# Filter AWS API calls
grep -E "(boto|botocore|HTTP)" .debug_logs/aws_scanner_debug_*.log

Common logging issues: no AWS API logs usually means one of --debug or --verbose is missing (both are required, and --verbose goes before scan); for sequential, readable debugging use --max-workers 1; if log files are missing, check permissions on the .debug_logs/ directory.

📌 Quick Reference

Most Common Commands

# Quick scan with basic output
poetry run aws-inventory scan --regions us-east-1

# Debug mode with detailed logging
poetry run aws-inventory scan --debug --regions us-east-1

# Full AWS API tracing (development/troubleshooting)
poetry run aws-inventory --verbose --log-file trace.log scan --debug --service ec2

# Tag-based filtering across all AWS services
poetry run aws-inventory scan --all-services --tag-key Environment --tag-value Production

# Production audit with comprehensive logging
poetry run aws-inventory --verbose --log-file audit.log scan --debug \
    --tag-key Environment --tag-value Production --format json --output audit.json

Flag Combinations Guide

📖 Documentation

The project includes comprehensive documentation covering all aspects of the system:

• Architecture Documentation — system design, component interactions, and patterns
• Architecture Decision Records — why things are built the way they are
• Shell Completion — tab-completion setup for the CLI
• Logging Architecture — the logging system: API tracing, configuration, integration patterns, troubleshooting, performance

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Install development dependencies: Use ./setup.sh
4. Set up pre-commit hooks: pre-commit install --install-hooks
5. Make your changes and run tests: poetry run pytest
6. Test logging changes: poetry run aws-inventory --verbose scan --debug --dry-run
7. Commit with a conventional-commit message (titles feed the release changelog): git commit -m 'feat: add amazing feature'
8. Push to the branch: git push origin feature/amazing-feature
9. Open a Pull Request

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with Typer for CLI interface
• Styled with Rich for beautiful console output
• Powered by Boto3 for AWS integration
• Managed with Poetry for dependency management

Made with ❤️ for the AWS community