Wilma - AWS Bedrock Security Configuration Checker

Enhanced with GenAI-Specific Security Features

A comprehensive security auditing tool for AWS Bedrock that combines traditional cloud security best practices with cutting-edge GenAI security capabilities. Perfect for organizations adopting generative AI while maintaining enterprise security standards.

Key Features for GenAI Security

GenAI-Specific Security Checks

• Prompt Injection Detection: Identifies vulnerabilities to prompt manipulation attacks
• Data Privacy Compliance: Detects PII exposure risks in model interactions
• Model Poisoning Detection: Monitors for signs of compromised training data
• Cost Anomaly Detection: Alerts on unusual usage patterns indicating potential abuse
• Guardrail Validation: Ensures content filtering and safety measures are in place

User-Friendly Design

• Standard Mode (default): Clear explanations with both simple and technical details
• Learning Mode: Educational content about each security check

Prerequisites & Setup

AWS Authentication Setup

This tool requires AWS credentials to access your Bedrock resources.

Quick Setup via AWS Console

1. Create IAM User

   • Go to AWS IAM Console
   • Click "Users" → "Create user"
   • Name it wilma-security-checker
   • Select "Programmatic access"

2. Set Permissions

   • Choose "Attach existing policies directly"
   • Either use "PowerUserAccess" OR create a custom policy with:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:List*",
                "bedrock:Get*",
                "bedrock:Describe*",
                "iam:ListPolicies",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetEventSelectors",
                "logs:DescribeLogGroups",
                "ec2:DescribeVpcEndpoints",
                "s3:GetBucketEncryption",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

3. Save Credentials
   • Download CSV or copy Access Key ID and Secret Access Key
   • WARNING: You won't see the secret key again!

Configure Your Local Machine

Option 1: AWS CLI (Recommended)

aws configure
# Enter your Access Key ID
# Enter your Secret Access Key
# Enter default region: us-east-1
# Enter default output: json

Option 2: Environment Variables

export AWS_ACCESS_KEY_ID="your-access-key-here"
export AWS_SECRET_ACCESS_KEY="your-secret-key-here"
export AWS_DEFAULT_REGION="us-east-1"

Option 3: AWS Profile

# Add to ~/.aws/credentials
[bedrock-checker]
aws_access_key_id = your-access-key
aws_secret_access_key = your-secret-key

# Use with: wilma --profile bedrock-checker

Supported AWS Regions

Bedrock is available in:

• US East (N. Virginia) - us-east-1
• US West (Oregon) - us-west-2
• Asia Pacific (Singapore) - ap-southeast-1
• Asia Pacific (Tokyo) - ap-northeast-1
• Europe (Frankfurt) - eu-central-1
• Europe (Ireland) - eu-west-1

Security Best Practices

• Never commit AWS credentials to git
• Use IAM roles when on AWS infrastructure
• Apply least privilege permissions
• Rotate access keys every 90 days
• Enable MFA on your AWS account

Quick Start

Installation

Option 1: Install from PyPI (Recommended)

pip install wilma

# Run directly from command line
wilma

Option 2: Install from Source

git clone https://github.com/ethanolivertroy/wilma.git
cd wilma
pip install -e .

# Or manually install dependencies
pip install -r requirements.txt
wilma

Usage

# Run security check (default mode)
wilma

# Learning mode - understand the security checks
wilma --learn

# Output as JSON for CI/CD integration
wilma --output json

# Use specific AWS profile
wilma --profile production

# Check specific region
wilma --region us-west-2

Testing Locally

Want to test Wilma with real AWS resources? We've got you covered!

Use the included demo script to create sample AWS Bedrock resources with intentional security issues:

# Install Wilma first
pip install -e .

# Create demo resources (with security issues)
python scripts/demo_setup.py --setup --region us-east-1

# Run Wilma to detect the issues
python scripts/demo_setup.py --test

# Clean up all demo resources
python scripts/demo_setup.py --cleanup

# Or do all three steps at once
python scripts/demo_setup.py --all --confirm

What the demo creates:

• S3 bucket without encryption (HIGH risk)
• S3 bucket without versioning (MEDIUM risk)
• S3 bucket without Block Public Access (CRITICAL risk)
• Knowledge Base without proper tags (LOW risk)
• IAM role with permissions for testing

Cost: Minimal (usually free tier eligible, < $0.10)

Important: Remember to run cleanup to avoid ongoing charges!

Current Security Check Coverage

Knowledge Bases (RAG) - 12 checks:

• S3 bucket public access validation (CRITICAL)
• S3 bucket encryption verification (HIGH)
• Vector store encryption (OpenSearch, Aurora, RDS) (HIGH)
• Vector store access control (CRITICAL)
• PII pattern detection in configurations (HIGH)
• Prompt injection pattern detection (HIGH)
• S3 versioning validation (MEDIUM)
• IAM role permission audit (HIGH)
• Chunking configuration review (LOW)
• CloudWatch logging validation (MEDIUM)
• Resource tagging compliance (LOW)
• Embedding model access control (MEDIUM)

Coming Soon:

• AWS Bedrock Agents security (10 checks)
• Advanced Guardrails validation (11 checks)
• Model Fine-Tuning security (11 checks)

Example Output

AWS Bedrock Security Check
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Good News: 3 security best practices are properly configured
Critical: 1 high-risk issue needs immediate attention
Attention Needed: 2 medium-risk issues found

CRITICAL ISSUES:
─────────────────────────
1. Policy allows unrestricted access to ALL Bedrock operations
   Where: IAM Policy: BedrockAdminPolicy
   Risk Score: 9/10

   What this means: This is like giving someone admin access to all your AI models
   Technical details: Policy contains wildcard actions (bedrock:*) with no resource restrictions

   To fix this, run:
   > aws iam create-policy-version --policy-arn arn:aws:iam::123456789012:policy/BedrockAdminPolicy --policy-document file://restricted-policy.json --set-as-default

Security Checks Performed

Traditional Security

• IAM permission auditing
• Encryption validation
• Network security (VPC endpoints)
• Audit logging configuration
• Resource tagging compliance

GenAI-Specific Security

• Prompt injection vulnerability assessment
• PII detection in model configurations
• Model access pattern analysis
• Usage anomaly detection setup
• Real-time threat monitoring

Knowledge Bases (RAG) Security - NEW!

• S3 Data Source Security: Public access blocking, encryption, versioning
• Vector Store Security: Encryption and access control for OpenSearch/Aurora/RDS
• PII Detection: Pattern-based scanning of configurations and metadata
• Prompt Injection Detection: Identifies suspicious patterns in KB content
• Access Control: IAM role and policy validation
• Configuration Review: Chunking strategies, logging, and tagging compliance

Architecture

The tool is designed with modularity and extensibility in mind:

wilma/
├── Security Checks
│   ├── Traditional AWS Security
│   └── GenAI-Specific Security
├── Reporting Modes
│   ├── Standard (default)
│   └── Learning
└── Output Formats
    ├── Human-readable text
    └── JSON for automation

DevSecOps Integration

CI/CD Pipeline Example

# .github/workflows/bedrock-security.yml
name: Bedrock Security Audit
on: [push, pull_request]

jobs:
  security-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Run Bedrock Security Audit
        run: |
          pip install wilma
          wilma --output json > security-report.json
          
          # Fail the build if critical issues found
          if [ $? -eq 2 ]; then
            echo "Critical security issues detected!"
            cat security-report.json
            exit 1
          fi

Risk Scoring System

The tool uses a simple 1-10 risk scoring system:

• 9-10: Critical - Immediate action required
• 7-8: High - Address within 24 hours
• 4-6: Medium - Plan remediation
• 1-3: Low - Best practice improvements

GenAI Threat Model Coverage

Based on OWASP Top 10 for LLMs and MITRE ATLAS:

Threat Category	Coverage	Detection Method
Prompt Injection	Yes	Pattern matching & guardrail checks
Data Poisoning	Yes	Training source validation
Model Theft	Yes	Access pattern analysis
PII Leakage	Yes	Content scanning
Denial of Service	Yes	Cost & rate monitoring
Supply Chain	Partial	Basic model source verification

Learning Mode

To understand the security concepts and checks performed:

wilma --learn

This explains:

• Prompt injection detection techniques
• PII pattern recognition
• Model access control principles
• Audit logging importance
• Network security for AI
• Cost monitoring for abuse detection

Contributing

We welcome contributions! Areas of interest:

• Additional GenAI attack patterns
• Integration with more AWS services
• Support for other cloud providers
• Enhanced remediation automation

Development Setup

# Clone and install in development mode
git clone https://github.com/ethanolivertroy/aws-bedrock-security-config-check.git
cd aws-bedrock-security-config-check
make install-dev

# Run tests
make test

# Format code
make format

Releasing New Versions

Releases are automated via GitHub Actions. See RELEASING.md for details.

Required IAM Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:List*",
                "bedrock:Get*",
                "bedrock:Describe*",
                "iam:ListPolicies",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetEventSelectors",
                "logs:DescribeLogGroups",
                "ec2:DescribeVpcEndpoints",
                "s3:GetBucketEncryption",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Why This Tool Stands Out

1. Dual Focus: Combines traditional cloud security with GenAI-specific risks
2. Accessibility: Beginner-friendly without sacrificing technical depth
3. Actionable: Provides exact commands to fix issues
4. Educational: Learning mode helps teams understand GenAI security
5. Automated: JSON output enables CI/CD integration
6. Comprehensive: Covers the full spectrum of Bedrock security concerns

Troubleshooting

Common Issues

"Unable to locate credentials"

# Check if AWS CLI is configured
aws configure list

# If not configured, run:
aws configure

"You must specify a region"

# Set default region
export AWS_DEFAULT_REGION=us-east-1

# Or specify in command
wilma --region us-east-1

"Access Denied" errors

Ensure your IAM user/role has the required permissions listed in the IAM Permissions section above.

Testing without AWS Account

While the main security checker requires AWS credentials, you can:

1. Use the learning mode to understand security concepts: wilma --learn
2. Review the documentation in this README
3. Use the tool with read-only IAM credentials to explore safely

"command not found: wilma" after pip install

If pip installed the package but the command isn't found:

# Option 1: Add Python user bin to PATH
echo 'export PATH="$HOME/Library/Python/3.11/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

# Option 2: Use the full path
~/Library/Python/3.11/bin/wilma

# Option 3: Create an alias
echo 'alias wilma="$HOME/Library/Python/3.11/bin/wilma"' >> ~/.zshrc
source ~/.zshrc

# Option 4: Install with pipx (recommended for CLI tools)
pipx install wilma

Project History

Wilma evolved from my earlier AWS Bedrock Security Configuration Checker project. After extensive use and feedback from the community, I rebuilt it from the ground up with a focus on:

• Cleaner Architecture: Modular design replacing the original monolithic structure
• Professional Output: Text-based status indicators instead of emojis for better terminal compatibility
• Better Maintainability: Separated concerns with dedicated modules for different security checks
• Enhanced Usability: Streamlined installation and CLI experience

The rebranding to "Wilma" represents this fresh start while maintaining the core security-first approach that made the original tool valuable.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

See the LICENSE file for details.

---

Built by ET for the GenAI security community