cs-image-system-aws-runtime-plugin

The aws runtime type. A runtime is where images are baked and instances run; this package makes Amazon EC2 one. It registers one runtime model (AwsCloudBuilderModel), one runtime builder (AwsCloudBuilder) and one version checker (AwsCLIVersionChecker, for the aws CLI). At load it discovers the account's VPCs, subnets and security groups with boto3 and checks the declared networking against them. During a run it resolves vendor AMIs by owner and filter, emits the amazon-ebs packer source for every image baked here, bakes and reaches instances through AWS Systems Manager (SSM), answers the state query from describe-images / describe-instances, reads and (for one bounded task at a time) sets an instance's power state, tags released AMIs, and disposes of AMIs together with their snapshots.

What it registers

The entry point is declared in pyproject.toml:

[project.entry-points."cs_image_system.plugins.runtime"]
aws_plugin = "cs_image_system.aws_runtime.main:initialize"

initialize() in main.py returns an AwsRuntimePluginMetadata whose services map the key aws to three classes and whose builders_for_models binds the model to its builder.

Class	csis_name()	csis_classifier()	Registered under
AwsCloudBuilderModel (aws_runtime_models.py)	aws	VCT.CLOUD_BUILDER_MODEL	cloud_builder_model and, because the loader re-registers cloud models as runtimes, runtime_builder_model
AwsCloudBuilder (aws_runtime_builders.py)	aws	VCT.RUNTIME_BUILDER	runtime_builder
AwsCLIVersionChecker (aws_runtime_builders.py)	aws-cli	VCT.VERSION_CHECKER	version_checker, since stage 48.1 (2026-09-17)

A YAML entry selects the model and builder with type: aws under runtime_builders:. There are no type aliases; aws is the only type key. An entry's own aliases: list gives that runtime extra names (the fixture's aws-east2-runtime also answers to aws, amazon and us-east-2), and other entries refer to it by any of them.

The version checker is selected differently: validate and every run look up a checker for each entry of cfg/executables.yml under the entry's name first, then its type (see validate.py). An executable named aws-cli therefore gets this checker, which runs <binary> --version and reads the version from the regex aws-cli\/([\d\.]+)\s on the first output line (aws-cli/2.32.0 Python/...). An entry named anything else with binary: aws falls to the generic checker.

Present in the package but not registered as a service:

• AwsProviderSpecificImage (aws_provider_specific_image.py), reached through the builder's provider_specific_image_class() hook.

The AWS section of the bootstrap (stage 70 step 4), through the cs_image_system.bootstrap entry-point group: aws = cs_image_system.aws_runtime.bootstrap:aws_section. It owns the questions and the terraform for this cloud's one-time initialisation -- the GitHub OIDC identity provider, the READ-ONLY and WRITE roles of CI_SETUP.md 3.3 with their trust and permission documents, and the state bucket and the Session Manager instance profile when the account lacks them (tfmodules/bootstrap_aws). The section is wanted by default where the tree declares an AWS runtime or an S3 state backend. Its defaults come from the raw cfg/runtime-builders.yml and cfg/state-backends.yml (read as YAML, never loaded; a starter's REPLACE-ME values and its example account id are no defaults), and every "does it already exist?" question is a probe of the account as the runtime's profile (aws iam get-open-id-connect-provider, get-role, get-instance-profile, s3api head-bucket): found, absent, or -- with no session -- unanswerable, which --quiet refuses by name. A role that exists is adopted with an import block in the root, and the subjects it already trusted that are not this repository's are read from its trust document and kept (another repository may share the role); the root's state binds to the tree's declared S3 backend when the bucket exists. See OPERATIONS.md, "The bootstrap".

Models

AwsCredentials

Extends CredentialsBase in credentials.py. The base declares no fields; each provider narrows it. Field names are boto3 Session keyword arguments and pass straight through.

Field	Type	Default	Meaning
profile_name	str | None	None	A named profile in the operator's AWS configuration. The fixture uses noaa.
aws_access_key_id	str | None	None	Key id; carries a template such as {{ ENV['AWS_ACCESS_KEY_ID_east2'] }}, never a literal.
aws_secret_access_key	str | None	None	Secret; same rule.
aws_session_token	str | None	None	Session token; same rule.

as_dict() omits unset fields, so an absent key never reaches boto3 as None. The shared model config forbids extra keys, so a misspelled key inside credentials: is a validation error naming the field path. No credential value belongs in the configuration tree: these fields name a profile or read the environment.

AwsCloudNetworkingModel

Extends CloudNetworkingConfig in cloud_builder.py, which is RuntimeNetworkingModel in runtime.py under another name.

Fields it adds:

Field	Type	Default	Meaning
addl_security_groups	list[str]	[]	Existing groups every launched instance also wears, unmodified. Checked against the account at load and the only list counted toward the limit of five below. The instance module's vpc_security_group_ids lists them after the generated group; the EFS storage builder uses them as the mount targets' client groups.
ssh_ingress_security_group_ids	list[str]	[]	Groups whose members may reach port 22 on launched instances. The generated instance security group's ingress references only these groups, never a CIDR range. Not checked against the account.
_model_id	str | None	None (not an init field)	Back-reference to the owning runtime model.

Base fields it inherits:

Field	Type	Default	Meaning
name	str	the network id	Empty or default falls back to network.
network	str	default	The VPC id. default resolves at load to the account's default VPC.
subnets	list[RuntimeSubnetModel]	required, non-empty	Each has name, subnet_id, is_default, public, cidr, availability_zone, config. The first entry with is_default: true is the default subnet; without one the model refuses to load.
availability_zones	list[RuntimeAvailabilityZoneModel]	[]	name, is_default. The default zone, when declared, becomes the packer source's availability_zone.

It strips name and refuses one that is empty afterwards (only a whitespace-only name can reach that check, because the base already substitutes the network id for an empty or default name). get_subnet_id() raises when no default subnet exists. It does not accept security groups by name, only by id, and it refuses more than five groups in addl_security_groups.

There is no security_group_ids field any more (stage 63). Until 2026-09-25 the model accepted it, checked its ids against the account and counted them toward the limit of five, but no emitter ever read it, so a group listed there reached neither the packer source nor the instance and storage roots. It was removed rather than wired in, because addl_security_groups already is the list that reaches the roots. Declaring it is now an unknown-key refusal at load (the shared model config forbids extra keys), and the error names the key; move the ids into addl_security_groups. The accessor get_security_group_ids() went with it; get_addl_security_groups() remains.

AwsCloudBuilderModel

Extends CloudBuilderModel in cloud_builder.py, which extends RuntimeBuilderModel in runtime.py, which extends BuilderModel and NameTyped in builder_model.py.

Fields it adds:

Field	Type	Default	Meaning
account_id	str | None	None	The AWS account. Reported by runtime describe; usable in templates as {{ this.account_id }}. Nothing else reads it: the account boto3 acts in is the profile's.
credentials	AwsCredentials	empty	Narrows the base CredentialsBase.
state_configuration	str	DEFAULT	Foreign key to a state backend. The second rung of every storage and instance root's backend resolution on this runtime: a root whose own state_configuration is default inherits this one; default here falls through to the default backend (stage 46.2, orchestrator.py).
ena_support	bool | None	None	When declared, written into the amazon-ebs packer source as ena_support = true|false, so the baked AMI's Elastic Network Adapter flag is set explicitly; unset, nothing is written and packer's default applies (stage 63; until 2026-09-25 accepted and read by nothing).
sriov_support	bool | None	None	When declared, written into the amazon-ebs packer source as sriov_support = true|false (enhanced networking through the Intel SR-IOV interface); unset, nothing is written and packer's default applies (stage 63; until 2026-09-25 accepted and read by nothing).
iam_instance_profile	str | None	None	Instance profile written into the packer source for build instances when session_instance_profile is unset (it is also the SSM fallback then). When both are set the SSM profile wins (stage 63 item 9, decided 2026-09-24; until then this one, written after the SSM block, replaced it). Never attached to launched instances.
session_mechanism	str | None	None	ssm (any case, surrounding whitespace ignored) is the only value. Any other string is an error the first time the builder's session_mechanism() is called, which is at generation.
session_instance_profile	str | None	None	The SSM-capable instance profile given to launched instances (through the instance plugin) and to build instances (through the packer source).
ssh_username	str	DEFAULT	The bake's SSH user for images on this runtime that name none more specifically: step 4 of the bake-user order (CONFIGURATION 5.1.1). Until stage 63 it overrode every entry.
networking	AwsCloudNetworkingModel | None	None	Narrows the base type and makes it optional. Missing networking logs a warning at load; image generation then fails (the packer source reads the VPC and subnet from it).
vpc_map	dict	discovered	Not an init field. {vpc_id: {"subnets": [...]}} from the account.
default_vpc_id	str | None	discovered	Not an init field.
all_security_groups	dict[str, dict]	discovered	Not an init field. {group_id: description} from the account.

Base fields it inherits, and what this runtime does with them:

Field	From	Default	Meaning here
name, type, description, aliases	NameTyped	name/type required	type is aws.
executable, is_default, config, gitignore, tags	BuilderModel		tags merge into every image's tags; is_default makes this the runtime default resolves to.
region	CloudBuilderModel	required	boto3 region_name, the packer region, the aws provider block, --region on the release and storage-transition commands.
default_machine_type	RuntimeBuilderModel	required	Instance type when neither an OS-builder runtime entry nor an image names one; the instance type of a launched instance that declares no machine_type.
default_image_builder	RuntimeBuilderModel	DEFAULT	The image builder an OS-builder runtime entry with image_builder: default resolves to.
default_owners	RuntimeBuilderModel	None	Joins every vendor-image query's owner list on this runtime, after the OS builder's owners and before the entry's own. Read since stage 63 item 22: the lookup in os_builder_runtime_config.py keyed the runtime by the entry's image_builder name and never found it; it now takes the image builder's runtime.
default_config_username	RuntimeBuilderModel	None	The bake's SSH user when nothing more specific names one: step 5 of the bake-user order (CONFIGURATION 5.1.1), read since stage 63 item 22.
ephemeral	RuntimeBuilderModel	False	See "Retention and the cycle".
retention_keep	RuntimeBuilderModel	None	Builds kept per series on this runtime when the image declares no retention.
on_failure, teardown_after	RuntimeBuilderModel	None	Runtime-level defaults for ephemeral instances.

It overrides finalize() to run update_networking() after the base finalization, and csis_name() to return aws. It carries the class attribute type = "aws".

Not supported: a top-level profile: key (the modelled key is credentials.profile_name; unknown keys are rejected), a parameters: key (retired; the load names the replacement), any session mechanism other than ssm, and a run without a region.

The builder

AwsCloudBuilder extends CloudBuilderBase in builder_base_cloud.py, an empty specialisation of RuntimeBuilderBase in builder_base_runtime.py. The runtime builder writes no files of its own: generate_items_during defers to the base and returns nothing. Everything it contributes reaches generated/ through the image, instance and release builders that call its hooks. In the order a run reaches them:

When	Hook	What it does
Configuration load	AwsCloudBuilderModel.finalize()	Runs update_networking() (network discovery below).
Resolution	provider_specific_image_class()	Returns AwsProviderSpecificImage.
Resolution	query_provider_image(subconfig)	Finds the vendor AMI for one OS-builder runtime entry (image query below). Returns (ami_id, owner, raw_result) or None.
Resolution	query_provider_image_by_id(subconfig, image_id)	Stage 63: the AMI an entry pins with image_id, looked up with DescribeImages by id (aws_utils._query_by_ami_id) instead of the query. Same return shape; None when AWS does not know the id.
Image generation	packer_source_type()	amazon-ebs.
Image generation	packer_source_blocks(...)	The data "amazon-ami" lookup and the source "amazon-ebs" block for one image.
Image generation	session_mechanism()	ssm or None; raises on any other declared value.
Image generation	session_agent_commands(os_family)	Installs and enables the SSM agent on a base image; dpkg path for debian/ubuntu, yum path otherwise. Empty when no mechanism is declared.
Image generation	session_verify_commands(os_family)	Assertions that the agent is present and enabled. Empty when no mechanism is declared.
Image generation	default_bake_user(family), bake_ssh_username(image), bake_finalize_commands()	default_bake_user: the vendor AMI's user, admin for debian, ubuntu for ubuntu, ec2-user otherwise (the last step of the bake-user order). bake_ssh_username keeps the base None, so the ansible provisioner keeps its own user on AWS; no closing provisioner.
After a bake	build_id_from_artifact(artifact_id)	Base default: the packer manifest's <region>:<ami> becomes the AMI id.
After a bake	retag_image(ami, tags)	The packer plugin stamps the resolved csis_parent and csis_fingerprint onto every recorded build.
Instance generation	session_instance_profile()	The instance profile the instance module attaches when the mechanism is ssm; None otherwise.
Instance verification	verify_instance(name, expected_build, expect_mounts, timeout)	Over SSM: waits for the launch completion marker /var/lib/csis/launch-applied, counts mounts under /mnt, compares the booted AMI with the expected build.
Instance operations	run_session_command(name, script, timeout)	AWS-RunShellScript through SSM on the RUNNING instance whose Name tag matches; polls the invocation to completion.
Power state (stage 57)	can_query_instance_power_state(), query_instance_power_state(name)	True; EC2's State.Name mapped onto the system's vocabulary (pending -> starting, running, stopping/shutting-down -> stopping, stopped, terminated -> absent, no instance -> absent, anything else -> unknown); None only when the cloud could not be asked.
Power state (stage 57)	can_set_instance_power_state(), start_instance(name, timeout), stop_instance(name, timeout)	True; start_instances / stop_instances followed by the EC2 instance_running / instance_stopped waiter (5 s delay, timeout // 5 attempts). Only running_for_task calls them.
Identity (stage 58)	can_query_instance_identity(), query_instance_identity(name)	True; {"instance_id": InstanceId, "provider_hostname": PrivateDnsName} of the named instance whatever its power state, or None.
Release	release_commands(build_id, tags)	One aws ec2 create-tags --region ... --resources <ami> --tags Key=..,Value=.. executable, with --profile when credentials.profile_name is set. Runs only under --no-dry-run and only when the global apply_release is true.
State reconciliation	retag_image(image_id, tags)	create-tags on one of the system's own AMIs (lineage relabel, lineage restamp).
Retention	dispose_image(build_id)	Deregisters the AMI and deletes every EBS snapshot in its block device mappings. A missing AMI returns False (already gone).
State query	query_images(series)	Every AMI owned by self that carries a csis_series tag: {image_id, name, state, created, tags}.
State query	can_query_instance_boot_image(), query_instance_boot_image(name)	True; the ImageId of the pending or running instance whose Name tag matches, or None. Read-only, never fatal.
empty --runtime	inventory()	Not implemented on AWS; the base raises NotImplementedError, so the emptiness check refuses this runtime.

Every instance lookup above is by the EC2 Name tag, which the instance module sets to the instance's declared name (tags = merge({ Name = var.name }, var.tags) in tfmodules/aws_instance/main.tf). Two lookups differ on purpose: _running_instance filters on pending/running (the boot-image query, the session command), _named_instance takes any state and prefers a non-terminated record (the power-state and identity queries, start and stop), because EC2 keeps a terminated record visible for about an hour after the machine is gone.

Credentials

AwsCloudBuilderModel.self_to_aws_client_config() is the one conversion point: credentials.as_dict() plus region_name from region. Every boto3 call in the plugin builds boto3.Session(**config) from it, so a profile_name selects a profile and an empty credentials: lets boto3's default chain run (environment variables, AWS_PROFILE, an instance role). The same profile name is written into the packer source (profile = "noaa" on both the data lookup and the source), passed as --profile to the release command, and read by the state-backend, storage and instance plugins for their provider blocks, backend files and transition scripts, so packer, terraform and the plugin's own calls all authenticate the same way.

Network discovery at load

update_networking() in aws_runtime_models.py calls get_vpc_map_and_default_vpc_id() in aws_utils.py, which issues describe_vpcs, describe_subnets, describe_route_tables and describe_security_groups once each and builds:

• vpc_map: for every VPC, a subnets list of {subnet_id, cidr, is_public, tags, config}. A subnet is public when its effective route table (its own association, else the VPC's main table) has a 0.0.0.0/0 or ::/0 route whose gateway id starts with igw- or vgw-.
• default_vpc_id: the VPC with IsDefault.
• all_security_groups: every group by id.

Then it validates the declaration: network: default becomes the default VPC (an error when the account has none); the VPC must exist; every declared subnets[].subnet_id must be one of that VPC's subnets (stage 63, below); every id in addl_security_groups must exist; more than five of them is an error. The declared subnets[].public flag is the operator's statement and is not overwritten by the discovered is_public.

The subnet check (stage 63) compares each declared subnet_id with the subnets the account listed for the declared VPC in vpc_map. When a declared id is not among them the load is refused with Subnet <id> (<name>) in networking configuration for AWS cloud builder <runtime> is not in VPC <vpc>. Until 2026-09-25 the declared subnet ids were not checked at all, so a subnet from another VPC (or a typo) passed the load and failed only at the first bake or apply. The check makes a claim only when the account's answer lists subnets for the VPC: a VPC listed without any subnets (an empty VPC, or a session that cannot see them) leaves every declared subnet unchecked rather than refusing all of them. A load therefore needs a live AWS session even for a dry run, and so does every command that loads the configuration for another runtime (the GCE cycle included).

The image query

query_provider_image() takes one OS-builder runtime entry (OSBuilderBaseImageBuilderSubconfig in os_builder_runtime_config.py). remap_for_image_query() builds a describe_images request:

• Owners is the entry's get_owners(): the OS builder's owners, then the runtime's default_owners, then the entry's own, deduplicated in order (the runtime's were never found before stage 63 item 22).
• query.filters keys are mapped through AWS_DI_MAP (snake_case to EC2 filter names, tag:<key> kept), booleans lowercased, and state: available forced.
• Keys the map does not know go to a post-query exact-match filter on the result dictionaries.
• The rewrite works on a deep copy of the entry's query (stage 63). get_query() hands back the entry's own nested mappings, and until 2026-09-25 the rewrite injected state: available into the OS builder entry's own filters mapping on every resolution, so the loaded model changed each time an image was resolved. The entry is now left exactly as declared.

query_image() in aws_utils.py runs that DescribeImages query (paging through describe_images), keeps every match that agrees with the post-query filter, and returns the newest by CreationDate, or None when nothing matches. Its docstring says so since stage 63; the unreachable raise that used to follow the return is gone. The builder then reads the owner straight from the AMI record, ImageOwnerAlias (for example amazon) else OwnerId (the account id), and returns (ImageId, owner, raw); the base registers an AwsProviderSpecificImage in the resolved state. A record with neither field raises Could not get owner alias or owner id .... Before stage 63 a second, unreachable Could not get owner branch followed that one; it was removed on 2026-09-25. No match returns None, and the base's resolve step then fails the run.

When the entry declares image_id (stage 63), the base's resolve step does not call query_provider_image() at all: it calls query_provider_image_by_id(entry, image_id), which runs DescribeImages with ImageIds=[<id>] through _query_by_ami_id in aws_utils.py, with the runtime's own client configuration (profile and region). The entry's owners and query are not consulted. The owner is read from the record the same way, ImageOwnerAlias else OwnerId (else self), and the result is (ImageId, owner, raw). An id AWS answers InvalidImageID.NotFound or InvalidImageID.Malformed for, or one that returns no image, gives None, and the base stops resolution with OS builder <name>: image_id '<id>' on runtime <rt> is not known to the provider; any other API refusal raises AMIQueryError. An AMI id is regional, so a pinned id must exist in this runtime's region. Pinning makes the base bake reproducible: the same vendor AMI every time, where the query takes the newest match.

Stage 63 also removed helpers from aws_utils.py that nothing called: get_ami_owner, get_ami_ssh_user, remap_for_aws, _query_by_name and _build_ami_filters. _query_by_ami_id (an exact lookup by AMI id) was kept, and since stage 63 it is what query_provider_image_by_id() calls.

AwsProviderSpecificImage.get_query_assets() gives the packer data "amazon-ami" body:

• resolved: filters = { image-id = "<ami>" } plus owners = ["<owner>"] when known;
• deferred (an image a build produces later): owners = ["self"], filters = { name = "<pattern>*", root-device-type = "ebs", virtualization-type = "hvm", architecture = "<arch or x86_64>" }, most_recent = true.

tests/test_aws_provider_specific_image.py pins both shapes.

The state query and instance hooks

query_images, query_instance_boot_image, query_instance_power_state, query_instance_identity, start_instance, stop_instance, verify_instance, run_session_command, retag_image and dispose_image all go through aws_utils.ec2_client() / aws_utils.aws_client("ssm", ...), which build a client from the same session config. The state query in state_query.py reads query_images to classify missing, foreign and changed images, and asks query_instance_boot_image only because can_query_instance_boot_image() is True. When that answers None it asks query_instance_power_state before calling the silence unavailable: a stopped machine becomes a note in the report, not drift and not a failure to answer (stage 57). verify_instance is the deferred verification step of the instance lifecycle; run_session_command is how the system runs a script on an instance (the unmount before a detach, the post-bake tests, the AltNames write of stage 58). query_instance_identity feeds the generation ledger and the alias pass in provider_aliases.py.

Retention and the cycle

The closing retention lifecycle in retention.py runs dispose image --retention at execution time. For each series on this runtime the number of builds kept is, in order: 0 when the runtime is ephemeral, the image's retention.keep, the runtime's retention_keep, else everything. Builds beyond that are disposed through dispose_image(), except released builds and builds an instance is pinned to or was launched from, which are reported as retention debt and kept. Declared storages are never touched by retention.

ephemeral: true on an AWS runtime means nothing baked there survives a successful run. The fixture's AWS runtimes are not ephemeral and declare no retention_keep, so every AMI baked on them stands until an explicit dispose image. on_failure and teardown_after on the runtime are the defaults for its ephemeral instances: keep leaves a failed instance standing and fails the run; teardown tears it down and still fails the run; teardown_after: "2h" lets a later run tear a standing failure down.

Cost posture

A run on an AWS runtime may leave standing: every AMI it baked and the snapshots behind them (kept by default), every released AMI (exempt from retention), the declared storages (managed by the storage plugins), every non-ephemeral instance, and an ephemeral instance whose verification failed under on_failure: keep. The packer build instance is terminated by packer; force_deregister = !var.release lets a non-release rebuild replace an AMI of the same name. Because inventory() is not implemented here, empty --runtime cannot prove an AWS runtime empty; the state query and query_images are the checks. An instance the operator switched off is left off: the system never starts one except for a bounded task (verification, the post-bake tests) and stops it again afterwards.

Emission

The plugin's output is visible in the golden emission under tests/fixtures/v2_golden/generated.

• pckr-ebs-ans-image-generation-source-basic-rh-10-block-000.pkr.hcl: the data "amazon-ami" lookup (filters, owners, profile) and the source "amazon-ebs" block written by amazon_ebs_source() in aws_packer_source.py: source_ami, ami_name, instance_type (the runtime entry's machine type, else the runtime default), region, vpc_id, subnet_id (the default subnet), the merged lineage tags, profile, ena_support and sriov_support only when the runtime declares them (stage 63; the fixture declares neither, so the golden source carries neither), and, because the runtime declares session_mechanism: ssm, ssh_interface = "session_manager", iam_instance_profile, associate_public_ip_address = false, ssh_timeout = "15m" and a user_data script that installs the SSM agent on vendor images that lack it. ssh_username is what cs_image_system.base.bake_user.resolve_bake_user answers (the image's entry, the chain root's entry, config_username, this runtime's ssh_username and default_config_username, then the family's vendor user: admin for debian, ubuntu for ubuntu, ec2-user otherwise). launch_block_device_mappings uses the source AMI's real RootDeviceName (else /dev/xvda for debian and amazon roots, /dev/sda1 otherwise), gp3, and volume_size from lineage.bake_disk_size(ctx, image, runtime) (stage 63), the one rule the input fingerprint also hashes: this runtime declares no default_disk_size, so for a base image it is the OS builder entry's default_primary_disk_size for this runtime when declared, else the OS builder's default_primary_disk_size (200); for an instance image its own primary_disk_size. The fixture's base images bake with volume_size = 200.
• pckr-ebs-ans-image-generation-block-000-build.pkr.hcl: the # debug session mechanism (ssm) shell provisioner per base image (session_agent_commands) and the # verify: SSM agent baked lines inside the in-bake verification provisioner (session_verify_commands). The amazon entry in the neighbouring -plugins.pkr.hcl comes from the image builder's required_plugins, not from this package.
• open-tofu-instance-generation.tf and open-tofu-instance-generation-instance-test.tf: written by the instance plugin from this model. provider "aws" { region, profile, alias }; an aws_security_group whose port-22 ingress lists ssh_ingress_security_group_ids and no CIDR; the instance module's subnet_id (the default subnet), vpc_security_group_ids (the generated group, then addl_security_groups), and iam_instance_profile (session_instance_profile). The .tfbackend.hcl beside them carries the same profile, written by the state-backend plugin.
• run-release.sh: carries the aws ec2 create-tags line from release_commands() when a release is declared. The golden declares none, so it is empty of them.
• Runtime facts: cs-image-system runtime describe aws-east2-runtime (runtime_facts.py) reports runtime, type, region, account_id, default_machine_type, ephemeral, retention_keep, the images, storages, instances and ephemeral_instances declared on it, the builders bound to it and its emission directories. It is a command, not a file.

Example configuration

From tests/fixtures/config/cfg/runtime-builders.yml, the default runtime:

runtime_builders:
  - name: aws-east2-runtime
    is_default: true
    aliases:
      - aws
      - amazon
      - us-east-2
    type: aws
    account_id: "514190660293"
    description: |
      AWS runtime builder for the {{ this.region }} region account {{ this.account_id }}.
    credentials:
      profile_name: noaa
    tags:
      Project: MyProject
      Environment: Development
    region: us-east-2
    # Debug sessions and bakes connect through SSM: no public IP, no SSH from outside.
    session_mechanism: ssm
    session_instance_profile: AmazonSSMRoleForInstancesQuickSetup
    default_machine_type: t2.micro
    default_image_builder: pckr-ebs-ans
    networking:
      # network is the VPC id
      network: vpc-0c78d0d63b7a100df
      # SSH reaches instances only through the session relay's group; no CIDR ingress
      ssh_ingress_security_group_ids:
        - sg-03015ec107ae5f81a
      # existing groups every instance also wears (never modified here)
      addl_security_groups:
        - sg-03015ec107ae5f81a
      subnets:
        - name: east2-az1-private
          subnet_id: subnet-09f79018af845358a
          is_default: true
        - name: east2-az2-private
          subnet_id: subnet-00075cfbfcbc8f2cf
        - name: east2-az1-public
          subnet_id: subnet-01e56d2f9c1c95cc8
          public: true
        - name: east2-az2-public
          subnet_id: subnet-0c127b762e075a94d
          public: true

The same file declares two more type: aws runtimes (aws-east1, west1) with network: default and no session mechanism; they show the minimum an AWS runtime needs: name, type, region, credentials, default_machine_type and a networking block with one default subnet. The live tree adds availability_zone to each subnet so that validate can refuse a zone mismatch between an instance, its zonal storages and its runtime's default subnet.

Prerequisites and integration

Everything below exists outside the system. The plugin creates none of it; the rule in docs/OPERATIONS.md ("The system generates no IAM changes; IAP, SSM and impersonation are operator wiring") applies to all of it.

An AWS account and a region. The plugin finds the region in the runtime's region field. It acts in whatever account the credentials resolve to; account_id is a fact for runtime describe and templates, not a check.

Credentials. Found through credentials: on the runtime, converted by self_to_aws_client_config():

• credentials.profile_name names a profile in ~/.aws/config (fixture and live: noaa, an sso-session profile; aws sso login --profile noaa opens it). This is the operator's path.
• credentials.aws_access_key_id / aws_secret_access_key / aws_session_token carry {{ ENV[...] }} templates for static keys.
• An empty credentials: lets boto3's default chain run: AWS_ACCESS_KEY_ID and friends, AWS_PROFILE, an instance role.

botocore drops the environment credential provider when a profile is named, so a named profile must itself carry credentials: in CI the workflow writes a [noaa] shim into ~/.aws/credentials from the federated keys (the [noaa] profile shim in docs/OPERATIONS.md). preflight reads the same field (else AWS_PROFILE) to find the SSO cache under ~/.aws/sso/cache (CSIS_AWS_DIR overrides the directory) and reports the token's expiry; static keys have no readable expiry.

Because update_networking() runs at configuration load, the session must be live for every load, dry runs and GCE-only work included. An sso-session profile's token renews itself only while the browser portal session lives; a session that lapses mid-run costs the legs that were running (2026-09-08, and again 2026-09-21).

IAM permissions for the caller. What the plugin itself calls, all with the session above: ec2:DescribeVpcs, DescribeSubnets, DescribeRouteTables, DescribeSecurityGroups (every load); ec2:DescribeImages (resolution, the state query, disposal); ec2:DescribeInstances (every instance lookup); ec2:CreateTags (retag, and the release command through the CLI); ec2:DeregisterImage and ec2:DeleteSnapshot (disposal); ec2:StartInstances, ec2:StopInstances (the bounded power-state exception); ssm:SendCommand and ssm:GetCommandInvocation (verification, unmounts, post-bake tests, the alias write). Packer's bake and the tofu roots need their own permissions on top, including iam:PassRole for the instance profile and ssm:StartSession for the Session Manager communicator; the CI table in docs/OPERATIONS.md (AWS_ROLE_ARN read-only, AWS_APPLY_ROLE_ARN write) lists the two roles the live job uses.

Network. Found through networking:: the VPC (network, or default for the account's default VPC), the subnets by id, the security groups by id. The fixture's VPC has no internet gateway, which shapes everything else: bakes and instances go on the default (private) subnet with associate_public_ip_address = false, egress is the subnet's NAT, and both packer and the operator reach machines through SSM. The default subnet must route to the SSM endpoints and to the package repositories the bootstrap downloads from (s3.<region>.amazonaws.com); SSH ingress is by security-group reference only (ssh_ingress_security_group_ids, the Okta gateway's group), and the gateway relay requires the target to wear that same group (addl_security_groups).

An SSM-capable instance profile. Found in session_instance_profile (live: AmazonSSMRoleForInstancesQuickSetup), attached to launched instances by the instance plugin and to build instances by the packer source. Without it, session_mechanism: ssm still bakes the agent but packer keeps the SSH communicator and instances get no profile. The role behind it needs the SSM managed policy; the caller needs iam:PassRole on it.

Tools. Found through cfg/executables.yml (executables.yml):

• aws (the entry named aws-cli, fixture floor >=2.32): the release command and the storage builders' transition scripts run it; its version is read by AwsCLIVersionChecker at validate and at every run.
• packer with the amazon plugin: declared on the image builder's required_plugins, not here; the source this package emits needs it.
• The AWS Session Manager plugin for packer's ssh_interface = "session_manager": installed on the machine that bakes (CI installs it before the performing step). Not declared in the tree; packer fails without it.
• tofu with the hashicorp/aws provider for the instance and storage roots, declared on those builders.
• Python: boto3>=1.43, botocore>=1.43, python-hcl2>=8.1, pydantic>=2.13, and the cs-image-system-hashicorp-utils package for the HCL formatter (pyproject.toml).

Vendor images. Found through the OS builders' owners and query (fixture: AlmaLinux 10 from owner 764336703387, the Debian images by family name). The account must be allowed to see and launch them (Allowed AMIs, Marketplace subscriptions where they apply).

Instances reachable by name. The instance module tags every machine Name=<declared name>; every lookup in this plugin depends on that tag and on the SSM agent being registered on the machine (baked in by session_agent_commands, bootstrapped by the source's user_data on a vendor image that lacks it). The launch script writes /var/lib/csis/launch-applied as its last act; verification waits for it.

Nothing from the identity side. Enrollment, OPA and sft are the identity plugins' concern; this plugin only runs their scripts on the machine over SSM when asked.

Configuration reference

Every field the plugin reads from the YAML it owns (the type: aws entry under runtime_builders:). The base's own fields are included because this plugin is what reads them on AWS. Types are the model's; default in a value column means the literal string default.

The runtime entry

Field	Type	Default	Meaning
name	str	required	The runtime's name; the value other builders' runtime: and the lineage records carry.
type	str	required	aws.
description	str or null	null	Free text; templates such as {{ this.region }} resolve.
aliases	list[str]	[]	Extra names the runtime answers to.
is_default	bool	false	The runtime default resolves to; at most one per class.
tags	mapping[str, str]	{}	Merged into every image's tags, so they reach the AMI and the packer source.
region	str	required	boto3 region_name; the packer region; the provider block; --region on the CLI commands the plugin and the storage builders emit.
default_machine_type	str	required	The bake instance type when neither the OS-builder runtime entry (default_machine_type) nor the image's runtime entry (machine_type) names one; the instance type of a launched instance without machine_type.
default_image_builder	str	default	What an OS-builder runtime entry with image_builder: default resolves to.
credentials	AwsCredentials	{}	See the model above. An unknown key is a validation error naming its path.
credentials.profile_name	str or null	null	The profile. Reaches boto3, the packer source, the release command, the provider blocks and the backend files.
credentials.aws_access_key_id, aws_secret_access_key, aws_session_token	str or null	null	Static keys through {{ ENV[...] }} templates; reach boto3 only.
account_id	str or null	null	Reported by runtime describe; a template value. A bare number is coerced to a string.
state_configuration	str	default	The backend a storage or instance root on this runtime inherits when its own is default; default here means the default backend.
session_mechanism	str or null	null	ssm bakes the SSM agent into base images, makes packer connect through Session Manager, attaches session_instance_profile to instances, and makes every instance operation possible (verification, unmount, post-bake tests, aliases). Any other value is refused at generation.
session_instance_profile	str or null	null	The instance profile for launched instances and (when iam_instance_profile is unset) build instances. Read only when the mechanism is ssm.
iam_instance_profile	str or null	null	The build instances' profile in the packer source when session_instance_profile is unset; with both set the SSM profile wins (stage 63 item 9). Never reaches launched instances.
ssh_username	str	default	The bake SSH user for images that name none more specifically (step 4 of the bake-user order, CONFIGURATION 5.1.1).
networking	mapping or null	null	See below. Null loads with a warning and fails at image generation.
ephemeral	bool	false	Nothing baked here survives a successful run: the closing retention disposes every build on this runtime.
retention_keep	int or null	null	Builds kept per series when the image declares no retention; null keeps all.
on_failure	str or null	null	Default failure policy for ephemeral instances here: keep or teardown.
teardown_after	str or null	null	Default grace for ephemeral instances here: <number> then m, h or d.
default_owners	list[str] or null	null	Owners added to every vendor-image query on this runtime (stage 63 item 22).
default_config_username	str or null	null	The bake SSH user when no image entry, OS entry, config_username or runtime ssh_username names one (stage 63 item 22).
ena_support	bool or null	null	Declared true or false: written as ena_support = true|false in the amazon-ebs packer source of every image baked here. Null: not written; packer's default applies (stage 63; until 2026-09-25 accepted and read by nothing).
sriov_support	bool or null	null	Declared true or false: written as sriov_support = true|false in the same source. Null: not written; packer's default applies (stage 63; until 2026-09-25 accepted and read by nothing).
executable, config, gitignore			Accepted from BuilderModel; this plugin reads none of them.
profile, parameters, runtime_classifier, executables			Refused (unknown keys); the fixture's comments record the replacements.

networking

Field	Type	Default	Meaning
name	str	the network id	A label; stripped, and refused when empty afterwards.
network	str	default	The VPC id; default resolves at load to the account's default VPC. Written as the packer vpc_id, the instance root's data "aws_vpc" id and the EFS builder's vpc_id.
subnets	list	required, at least one	Exactly one entry with is_default: true; the load refuses none.
subnets[].name	str	the subnet id	Label.
subnets[].subnet_id	str	required	The subnet id. The default one is the packer subnet_id and the instance module's subnet_id; the EFS builder mounts on every non-public one (one per zone). Checked at load against the subnets the account lists for the declared VPC, and refused when it is not one of them (stage 63; until 2026-09-25 not checked). A VPC the account lists without subnets makes no claim.
subnets[].is_default	bool	false	The subnet bakes and instances use.
subnets[].public	bool	false	The operator's statement; not overwritten by discovery. The EFS builder skips public subnets.
subnets[].cidr	str or null	null	Informational.
subnets[].availability_zone	str or null	null	Declared, never inferred; validate refuses a zone incompatible with an instance's zonal storages. The EFS builder uses it to pick one subnet per zone.
subnets[].config	mapping	{}	Accepted, not read.
availability_zones	list	[]	{name, is_default}; the default one becomes the packer availability_zone.
addl_security_groups	list[str]	[]	Checked to exist at load; the only list counted toward the limit of five; appended to the instance's vpc_security_group_ids; the EFS mount targets' client groups.
ssh_ingress_security_group_ids	list[str]	[]	Not checked at load. The generated instance group's port-22 source; when empty the ingress is the VPC's own CIDR block instead.
security_group_ids			Refused (unknown key) since stage 63. Until 2026-09-25 it was checked and counted but reached no emitter; put the ids in addl_security_groups.

What other entries contribute

These are not this plugin's fields, but the packer source reads them:

Where	Field	Effect on AWS
OS builder runtimes[] entry	default_machine_type	The bake instance type for that base (the fixture bakes on t3.medium because a t2.micro OOMs).
OS builder runtimes[] entry	ssh_username	The bake user for that OS on this runtime (step 2 of the bake-user order); unset, the order continues down to the chain root family's vendor user.
OS builder runtimes[] entry	owners, query	The describe_images request.
OS builder runtimes[] entry	image_id	Stage 63: an AMI id that replaces the query; looked up by id with DescribeImages.
OS builder runtimes[] entry	default_primary_disk_size	Stage 63: the base image's volume_size on this runtime when declared.
OS builder	default_primary_disk_size	The base image's volume_size when the entry declares none (200 by default).
image runtimes[] entry	machine_type	The bake instance type for that instance image.
image	primary_disk_size	The root volume size of an instance-image bake, in GB.
instance	machine_type	The launched instance type; the runtime default otherwise.

Variations

• A queried vs a pinned vendor AMI. Without image_id on the OS builder's entry, resolution runs the describe_images query and takes the newest match, so the next base bake may start from a newer vendor AMI. With image_id (stage 63) the query is skipped and that one AMI is looked up by id; it must exist in this runtime's region, and the id is part of the fingerprint's vendor source.

• Where the bake disk comes from. volume_size is lineage.bake_disk_size (stage 63): this runtime has no default_disk_size, so a base image takes its entry's default_primary_disk_size, else the OS builder's (200). Until 2026-09-25 the fingerprint hashed the entry's value (then defaulting to 100) while the bake used the OS builder's 200; the fingerprint now hashes what is baked, so every AWS base image's recorded fingerprint moved once and reads DUE until lineage restamp --runtime <rt> --commit records the new one.

• session_mechanism: ssm declared, with a profile. The packer source gets ssh_interface = "session_manager", the profile as iam_instance_profile, associate_public_ip_address = false, ssh_timeout = "15m" and the agent-installing user_data; base images get the agent provisioner and its verification lines; launched instances get the profile; verification, unmounts, post-bake tests and alias writes work. Not declared: the source keeps packer's SSH communicator with associate_public_ip_address = true and no user_data, no agent provisioner is emitted, instances get no profile, and every session-based operation fails with SSM's TargetNotConnected / InvalidInstanceId. Declared without any profile: the agent is baked and verified, but the source keeps the SSH communicator (the SSM block needs a profile).

• iam_instance_profile set beside session_instance_profile. Both build and launched instances wear session_instance_profile (the SSM profile wins, stage 63 item 9); iam_instance_profile applies to build instances only when the SSM profile is unset.

• networking.network: default resolves to the account's default VPC at load and is refused when the account has none. A VPC id must be one the session can describe. Either way, a declared subnet in that VPC loads and a subnet id from another VPC (or a mistyped one) is refused at load (stage 63; it used to pass and fail at the first bake or apply); a VPC the account lists without subnets leaves the declared subnets unchecked.

• Security groups in addl_security_groups are checked, counted (at most five) and worn by every launched instance; the same ids under security_group_ids are an unknown-key refusal at load (stage 63; until 2026-09-25 accepted, checked and counted, but never emitted).

• ena_support / sriov_support declared (true or false) are written into the amazon-ebs source as that value; not declared, the source carries neither line and packer's defaults decide (stage 63; until 2026-09-25 the fields were accepted and read by nothing, whatever their value).

• ephemeral: true keeps nothing baked here past a successful run; false (the fixture) keeps everything until retention_keep, an image's retention or an explicit dispose image.

• A pinned parent (parent_policy: pinned with a pin, or a rebuild) writes filters = { image-id = "<pinned>" }, owners = ["self"] into the data lookup; an unpinned or deferred parent writes the name-pattern / most-recent lookup from get_query_assets(); a resolved vendor image writes its exact id and owner.

• An entry's ssh_username wins over the runtime's (stage 63 item 23; the runtime's used to win); the OS builder's config_username wins over the runtime's too. With nothing named anywhere, an image takes admin (debian root), ubuntu (ubuntu root) or ec2-user.

• A vendor query that resolved gives the bake the real RootDeviceName; a deferred source falls back by family (/dev/xvda for debian and amazon, /dev/sda1 otherwise).

• Dry run versus real run. A dry run still loads (network discovery runs), still resolves vendor AMIs, still emits the source and provider blocks, and still runs the state query's read-only hooks; it never bakes, never tags, never disposes, never starts or stops a machine, and the release and dispose commands print their plan. Only --no-dry-run runs the runner scripts that call aws ec2 create-tags and dispose image.

• apply_release on (global config) adds the create-tags executables to the release runner; off, release records the release in meta-state and tags nothing in the cloud.

• apply_instances on for this runtime lets the instance root apply, after which the plugin's instance hooks have something to find; off, the lookups answer "absent" and the stage-58 alias pass does nothing.

• A stopped instance is a note in the state report, skips verification and login proof unless the task may start it, and is left off by the alias pass; a running one is verified and aliased; a terminated record counts as absent.

• An encrypted value (ENC[age:...]) anywhere in the entry is decrypted by the core before this plugin sees it; the plugin has no behaviour of its own for encryption.

What it tests and verifies

At load (pydantic validators and finalize).

• AwsCredentials: unknown keys refused (extra="forbid"), so profile_nme is a validation error naming credentials.profile_nme.
• RuntimeNetworkingModel.__post_init__: network non-empty; at least one subnet; exactly one default subnet. AwsCloudNetworkingModel: the name non-empty after stripping. RuntimeBuilderModel: default_machine_type present. CloudBuilderModel: region present.
• update_networking() (at finalize, after the base): discovers the account (four describe calls) and refuses a default network with no default VPC, a VPC the account does not have, a declared subnets[].subnet_id that is not among the subnets the account lists for that VPC (stage 63; skipped when the account lists none for it), a security group in addl_security_groups the account does not have, and more than five groups in addl_security_groups. Missing networking is a warning, not an error. Every verdict is a ValueError with the message logged at ERROR first; the load fails and run exits 1 (validate exits 1).
• security_group_ids under networking: is refused as an unknown key by the shared model config, like any other undeclared key (stage 63; until 2026-09-25 it was a field that was validated and counted but never emitted).
• The AWS CLI version: validate and every run look up the checker for each declared executable; for the entry named aws-cli it is this package's, and a missing binary, an unreadable version or one below the floor fails validation by name.

At validate. Nothing of this plugin's own beyond the load and the tool check. The base's N9 dead-end rule reads session_mechanism(): a base image with no admin public key and no session mechanism on a runtime is a validation error naming the runtime.

At generation. session_mechanism() raises ValueError on a value other than ssm. amazon_ebs_source() asserts that the source image has query assets (an AssertionError naming the image and runtime; it cannot fire for a resolved or deferred PSI). ena_support and sriov_support are written into the source only when declared, as the declared boolean; there is no further check on them (stage 63). Resolution's query_provider_image raises when the region is unset or the AMI record carries neither ImageOwnerAlias nor OwnerId, and returns None on no match, which resolve.py turns into a failed run. For an entry that pins image_id, query_provider_image_by_id returns None when AWS does not know the id, and resolve.py stops the run naming the id and the runtime (stage 63).

At apply. Nothing in this package runs at apply: packer and tofu do, with what this package emitted. Packer's own in-bake verification carries the two SSM assertions from session_verify_commands (the agent binary present, the unit enabled); a failed assertion fails the bake.

After apply (post-finalize hooks and deferred commands).

• After a bake the packer plugin calls retag_image() for every recorded build; the verdict is the AMI's tags (checked by the next state query).
• verify instance <name> (the ephemeral sequence, just cloud-verify, the post-bake tests) calls verify_instance(): three checks named startup scripts (the marker within timeout, polled every 15 s over SSM), booted image (the running instance's ImageId against the expected build, or against lineage when nothing is pinned yet) and, when the instance declares storages, data disks mounted (findmnt targets under /mnt/ at least the declared count). The base appends declared tests from tests.post_bake, run over run_session_command. The verdict lands in meta-state/verifications.yaml with the last 20 lines of output as evidence; a failed verdict stops the runner and verify assert fails the run.
• Before a detach, unmount storage runs the unmount script over run_session_command; the verdict is the receipt under unmount-receipts/<instance>__<storage>.json in the root's workspace, which gate-plan --require-unmounted reads.
• After an applies-on instance run the alias pass asks query_instance_identity and query_instance_power_state, and writes AltNames over run_session_command to running machines only; the verdict is a log line per instance and reality.instances in the state report. The generation ledger (meta-state/instance-state.yaml) records the provider's instance id from the same identity query.
• verify login skips a machine whose power state is not running.

In the state query (state query, just cloud-preflight, the start of every run): query_images on this runtime feeds missing, foreign and changed; query_instance_boot_image compares each pinned instance's booted AMI with its pin (changed when they differ, a note when a pending replacement explains it), and a standing ephemeral is reported; query_instance_power_state turns a stopped machine into a note and a stopped ephemeral into a STANDING line. A hook that raises makes an unavailable: line, never a failure. The verdicts land in generated/state-report.json (run-local, never committed) and on the console; --strict exits 1 on anything but stale.

tests/test_v2_bootstrap_aws.py holds the bootstrap's AWS section: the entry point and the tree-derived gate, the starter's placeholders and an unaskable account refused by name under --quiet, the forks following a stubbed account, the subject forms following the ids, the import blocks, the backend (the declared bucket when it stands, local when the bootstrap makes it, and the second interview that binds it), the module's statements held to the guide's Sid for Sid, and a real tofu validate of a root carrying the GitHub and AWS sections together.

When it fails

Failures that have happened, oldest first, then the ones the code raises that have not. Dates are from docs/history/LEDGER.md, docs/OPERATIONS.md and the tests.

Seen live

• 2026-08-31, stage 1: packer's data "amazon-ami" found no credentials. No valid credential sources found from the datasource although the operator's shell was logged in: packer does not read a profile it is not told about. The source now carries profile = "<profile_name>" on both the lookup and the source. If it recurs, the runtime's credentials.profile_name is unset or the profile has no live session.
• 2026-08-31, stage 1: Timeout waiting for SSH on every bake. The VPC has no internet gateway, so packer's SSH communicator could never reach the build instance. session_mechanism: ssm with a profile makes packer connect through Session Manager; the same message today means the Session Manager plugin is missing on the baking machine, the build instance never registered with SSM (no route to the SSM endpoints from the default subnet, the user_data bootstrap failed, or the profile lacks the SSM policy), or the bake ran past ssh_timeout = "15m". Look at the packer log of the bake and at SSM Fleet Manager for the instance.
• 2026-08-31, stage 1: a Debian chain baked as ec2-user never authenticated (again Timeout waiting for SSH, over the SSM tunnel). Only the vendor user's authorized_keys receives packer's key, so the bake user now follows the chain root's family (admin for debian). A runtime ssh_username or an entry ssh_username that names the wrong user brings it back.
• 2026-08-31, stage 1: two root-slot volumes on the deb-11 chain. A hard-coded /dev/sda1 device on a Debian root (whose root is /dev/xvda) baked an AMI carrying two volumes in the root slot; instances from it never booted. The mapping now uses the source AMI's real RootDeviceName, else the family default. A deferred source still falls back by family, so a chain whose root family is unknown to _chain_root_family gets /dev/sda1.
• 2026-08-31 to 2026-09-01, stage 1: t2.micro out of memory at bake (dnf update killed) and on instances (the SSM agent killed, finding 25). The OS builder entries bake on t3.medium; an instance declares its own machine_type. The symptom is a bake that dies mid-provisioner or an instance that stops answering SSM after boot.
• Stage 1: curl: (23) re-downloading the SSM agent on Debian during the agent provisioner, because the user_data bootstrap had already left a root-owned /tmp/ssm.deb. The provisioner now skips the install when the agent is present and downloads to its own file name.
• 2026-09-01 to 2026-09-02, stage 1, findings 27 to 31: sft ssh hung on the advertised private address. Instances wore only the VPC default group; the instance root now creates csis-instances with SSH ingress from ssh_ingress_security_group_ids only (opening the shared CIDRs is forbidden), and the gateway relay also needs the target in the gateway's own group (addl_security_groups). An instance that cannot be reached by the relay while SSM works points at those two lists.
• 2026-09-02 and 2026-09-03: orphan AMIs from failed and unrequested bakes. Eleven AMIs and twelve snapshots from stage 1, and two more from stale runner scripts, appeared as foreign in the state query (they carry our tags, no record has them). The operator deregistered them by hand; state import adopts such an image, dispose image removes a recorded one. A foreign line is never hard drift, but every line is actionable.
• 2026-09-08: a follow image kept its generation-time tags. A child baked in the same run as its parent was tagged with a series-... parent and a placeholder fingerprint; the state query reported changed. The packer plugin now calls retag_image after every bake, and lineage relabel --runtime <rt> re-tags any build whose tags still disagree with its record.
• 2026-09-08: the AWS SSO session lapsed mid-run. A tofu plan failed with No valid credential sources found ... backend s3 ... the SSO session has expired after cloud-preflight had passed at the start; the same lapse ends any load with AWS Error: ... from get_vpc_map_and_default_vpc_id, on GCE work too. Environmental, not a defect: aws sso login --profile noaa, then just full-test-legs or the legs that were running. preflight and the strict state query flag a session expiring within config.preflight.expected_run_minutes; an sso-session profile reports no fixed expiry.
• 2026-09-09: the first live AWS disposal (dispose image <two EL8 AMIs> --no-dry-run --commit) deregistered both and deleted their snapshots; describe-images on both ids answered an empty list afterwards. Not a failure, but the proof that a disposal's success is read from the cloud, not from the log line.
• 2026-09-10, ledger 72: InvocationDoesNotExist right after send_command. The unmount had already run on the instance when get_command_invocation raised, so no receipt was written and the gate refused the detach. The poll now tolerates that code until its deadline (test_v2_aws_parity.py); any other SSM error still raises.
• 2026-09-12, stage 17: credentials.profile_nme was silently dropped. With credentials a plain mapping the typo left the profile unset and boto3 fell back to AWS_PROFILE or nothing. The typed AwsCredentials makes it a validation error naming credentials.profile_nme.
• 2026-09-21, stage 57: a stopped machine was reported as "the provider could not answer". The boot-image probe filters on running, so a machine the operator had switched off looked like an unreachable cloud and run_session_command said no running instance for both. The power-state hook separates them: the state report says STOPPED (switched off; not drift) in a note, and a session command on a stopped machine fails with cannot run a command on '<name>': it is STOPPED (switched off; not drift).
• 2026-09-22: the strict query refused the launch that finishes an upgrade. After upgrade instance moved the pin, the booted AMI was behind it by design, and cloud-launch's preflight called that changed. With the pending-replacement marker present it is now a note; without the marker it is drift again.
• 2026-09-22, stage 58: the first live replacement wrote no aliases. The alias pass ran before the new machine had enrolled; a machine launched in the same run is now waited for, and one that never enrolls gets no alias that run (the log says so).

Raised by the code, not yet seen

Load (ValueError, logged at ERROR, run/validate exit 1):

No VPC ID specified in networking configuration for AWS cloud builder <name>, and no default VPC found in AWS account. ...
VPC ID <vpc> specified in networking configuration for AWS cloud builder <name> not found in AWS account.
Subnet <id> (<name>) in networking configuration for AWS cloud builder <runtime> is not in VPC <vpc>.
Security group ID <sg> specified in networking configuration for AWS cloud builder <name> not found in AWS account.
Total number of security groups specified in networking configuration for AWS cloud builder <name> is <n>, which may exceed limits for certain instance types. ...
AWS Error: <botocore message>
Networking configuration name cannot be empty for <model>.

The subnet line (stage 63) means a declared subnets[].subnet_id is not one of the subnets the account lists for the declared VPC: a typo, a subnet from another VPC or account, or a network: default that resolved to a VPC other than the one the subnets belong to. Fix the subnet id or the network in the tree. Before 2026-09-25 this passed the load and surfaced at the first bake or apply as a packer or tofu error about the subnet. The security-group count line counts only addl_security_groups since stage 63.

Declaring networking.security_group_ids fails the load as an unknown key (stage 63), with the shared model config's extra-key error naming security_group_ids. Until 2026-09-25 the field was accepted, checked and counted but never emitted; move its ids into addl_security_groups, which is the list that reaches the instance and storage roots.

AWS Error: wraps every BotoCoreError/ClientError of the discovery: an unknown profile (The config profile (<p>) could not be found), an expired token, a role without the four describe permissions (UnauthorizedOperation), no network. The message names the cause; the fix is on the credential side, never in the tree. The warning No networking configuration provided for AWS cloud builder <name> is not fatal at load but the image generation that follows raises on the missing VPC.

Resolution:

Failed to create EC2 client: <e>                      (AMIQueryError)
Failed to query AMI with filters: <e>                 (AMIQueryError: a describe_images refusal, e.g. an invalid filter name)
Region must be specified in builder config to resolve image identifier, but got None
Could not get owner alias or owner id for image <ami> in region <region>, got <dict>
No resolved Image identifiers for OS <os builder> in predefined_resolve   (base: the query matched nothing)
Failed to query AMI by ID: <e>                        (AMIQueryError: a DescribeImages refusal other than NotFound/Malformed, for a pinned image_id)
OS builder <name>: image_id '<id>' on runtime <rt> is not known to the provider   (base, stage 63: the pinned AMI is not in this region, is deregistered, or is malformed)

The No resolved Image identifiers line is the usual symptom of a vendor query that is too narrow, an owner id that is wrong, or a vendor image that went away; the request that was sent is logged at DEBUG (Querying for AMI with filters: ...) and the count found at INFO.

Generation:

AWS runtime <name>: unknown session mechanism '<value>' (supported: ssm)
Provider-specific image for image <image> on runtime <rt> must have query assets ...   (AssertionError)

Instance operations (RuntimeError unless noted; the caller records or reports it):

AWS runtime <name>: cannot run a command on '<instance>': it is STOPPED (switched off; not drift)
AWS runtime <name>: cannot run a command on '<instance>': it is absent (no such machine)
AWS runtime <name>: cannot run a command on '<instance>': it is power state unavailable (no runtime could answer)
AWS runtime <name>: no instance named '<instance>' to start
AWS runtime <name>: no instance named '<instance>' to stop
ssm command <id> still <status> after <timeout>s          (returned as exit 1 with this text, not raised)

An SSM send_command on a running instance whose agent is not registered raises botocore's InvalidInstanceId; a machine that exists but cannot be reached shows as TargetNotConnected in the verification evidence. The EC2 waiters raise WaiterError when a start or stop exceeds timeout; running_for_task then logs the machine as started but unreachable and skips the work, and a failed stop is logged at ERROR with the instruction to stop it by hand. An unrecognised State.Name is logged (reports unrecognised state '<raw>'; treating it as unknown) and answered as unknown.

Verification (verify_instance; the check's detail in meta-state/verifications.yaml):

no completion marker within <timeout>s
booted <ami>, expected <build>
<n> mount(s) under /mnt, <m> declared

Disposal and tags: AMI <id> not found; nothing to deregister is a warning and returns False (the record is still dropped by the caller); any other describe_images, deregister_image or delete_snapshot refusal raises and stops dispose image with nothing recorded. A create_tags refusal in retag_image is caught by lineage relabel (could not retag <id>) and the state query keeps showing changed; in the packer plugin's post-bake retag it propagates.

The state query and the read-only probes never fail a run on this plugin's account: query_images raising becomes state query images/<rt> unavailable: <e> and an unavailable: line; query_instance_boot_image, query_instance_power_state and query_instance_identity swallow every exception into a DEBUG line and answer None (no claim).

Release: the runner script runs aws ec2 create-tags ... --profile <p> under set -euo pipefail; a missing aws binary, a profile without a session or a role without ec2:CreateTags fails the release lifecycle's apply (failed in run-summary.json) after meta-state already recorded the release. Re-run run release once the cause is fixed; the command is idempotent.

empty --runtime <aws runtime> prints empty: AwsCloudBuilder cannot list its inventory and exits 2; the runtime cannot be proved empty this way.

Tool check (validate and every run, exit 1):

aws-cli: binary 'aws' not found (declared in cfg/executables.yml; an absolute path, or a name on PATH)
aws-cli: could not read its version (`aws --version`): <e>
aws-cli: AwsCLIVersionChecker could not parse a version from `aws --version`

and the base's version-floor message naming aws-cli and the requirement when the installed version is below it.

Related

• gcloud-runtime-plugin: the other runtime type, with the same hook surface on Compute Engine.
• default-os-plugin: the OS builders whose runtime entries this plugin resolves to AMIs.
• packer-plugin: the image builder that asks this plugin for the amazon-ebs source and the session provisioners.
• tf-ebs-instance-plugin: the instance and storage roots that read this model's networking, profile and session profile.
• dummy-plugin: the extension template; an unregistered copy of its group model sits at the bottom of this package's models module.
• Base classes: runtime.py, cloud_builder.py, credentials.py, provider_specific_image.py, builder_base_runtime.py, abstract_version_checker.py.
• Consumers of the hooks: state_query.py, retention.py, dispose.py, verify_instance.py, unmount.py, relabel.py, release.py, power_state.py, provider_aliases.py, runtime_facts.py.
• docs/OPERATIONS.md for the credential contract and the operator's cycles; docs/DESIGN.md for the design; docs/PLUGINS.md for the package index.
• The configuration reference -- every field of the YAML this plugin reads, with an example.