Amazon Ec2 Image Builder

Troubleshooting an Image Builder AMI build

(Applies to any Image Builder build, not only ones created through this skill; exit-code examples are Linux — Windows conventions are in other-image-types.md.)

The single most important fact: where the real error lives depends on which stage failed. The approach:

1. Read image.state.status and image.state.reason (command below) — which stage failed decides everything else.

2. Match state.reason against the failure map. When it only names a component, go straight to the CloudWatch logs section — the APIs won't have more.

3. No matching row: localize by stage and check the pipeline's build history (the paragraph after the map), and let list-workflow-step-executions name the failing step.

4. Root cause found: fix it, then retry-image for test/distribution-stage failures or a new build otherwise. Root cause unclear: retry with the build instance kept alive (Live debugging below) and inspect it directly.

aws imagebuilder get-image --image-build-version-arn ${build_arn} --region ${region} --query "image.{state:state,name:name}"

Failure map: stage → where the truth is

When state.reason matches none of these, localize by stage — launch instance → SSM agent connect → build components → test components → create image (snapshot/Sysprep) → distribute → integrate — and check history: list-image-pipeline-images on the image's sourcePipelineArn shows whether the pipeline ever succeeded. A first-build failure points at configuration; a regression on a scheduled build points at a new base-image version or a changed component.

CloudWatch logs (the build's full record)

Image Builder streams the build's full execution record to CloudWatch by default — the service's own workflow activity (launching the instance, polling, creating the output image) alongside every command's stdout/stderr from AWSTOE (the on-instance component runner):

aws logs tail "/aws/imagebuilder/${image_name}" --region ${region} --since 1h

The log group is named after the image name from the get-image output above (it matches the recipe name for pipeline builds; don't derive it from the build ARN — the ARN lowercases the name while the log group preserves its case). Within the group, each build writes to a stream named <image version>/<build version> (e.g. 1.0.0/1) — the same two segments that end the build ARN — so when a pipeline's group holds many builds, read the stream for the failing build version. Pipeline-level events are in a separate group, /aws/imagebuilder/pipeline/${pipeline_name}, with date-named streams (YYYY/MM/DD).

A build can fail at any workflow step — launching the instance, running a component (TOE document), creating the output image — and each shape reads differently in the stream, so start from the first error and work outwards rather than hunting for one marker:

• A failed command step (components using the ExecuteBash/ExecutePowerShell/ExecuteBinary action modules): searching ExitCode jumps to it. ExitCode 126 ("cannot execute") is usually an architecture mismatch — an x86 binary running on an arm64 instance or vice versa (exec format error) — or a file that isn't executable; for a mismatch, fix the component's download URLs or the parent image/instance types so all three agree. For other nonzero codes, read the surrounding stdout/stderr — it is the full output of the failing command.

• Everything else carries no exit code: components built on the other AWSTOE action modules (downloads, file operations, reboots, assertions, OS updates) each fail with their own error message, and the same goes for TOE document validation, bootstrap problems, and service-side steps failing (launch, image creation) — all surface as error text in the stream. The Common errors list below carries the recurring signatures (Unable to bootstrap TOE, GetComponent ... dial tcp, a step restarting mid-run), and list-workflow-step-executions names the failing step when the stream alone doesn't make it obvious.

• The logs also reference the SSM RunCommand invocations that drove each stage (command id: <uuid>) — aws ssm get-command-invocation --command-id ${command_id} --instance-id ${build_instance_id} returns that command's status and output, a useful second view when the log stream cuts off.

If S3 logging was configured on the infrastructure configuration, the bucket holds the detailed on-instance AWSTOE logs — more granular than the CloudWatch stream. Both destinations carry everything build commands print — a secret echoed by a step lands in the logs, which is why secrets are fetched at run time from AWS Secrets Manager or SSM Parameter Store, never embedded in components (creating-images.md step 4).

Live debugging on the build instance

By default Image Builder terminates the build instance on failure, so logs are all you get. To keep the instance running for SSH/SSM inspection, update the infrastructure configuration before retrying. update-infrastructure-configuration is a full replacement: read the current settings back with aws imagebuilder get-infrastructure-configuration --infrastructure-configuration-arn ${infra_arn} and re-pass every set field (instance profile, instance types, metadata options, logging, subnet/security groups) or they are dropped. If the configuration doesn't use S3 logging, drop the --logging line below; add --subnet-id/--security-group-ids if the configuration sets them:

aws imagebuilder update-infrastructure-configuration \
  --infrastructure-configuration-arn ${infra_arn} \
  --instance-profile-name ${name}-role \
  --instance-types ${type_a} ${type_b} \
  --instance-metadata-options httpTokens=required \
  --logging "s3Logs={s3BucketName=${log_bucket},s3KeyPrefix=imagebuilder}" \
  --no-terminate-instance-on-failure --region ${region}

Instance charges apply until you terminate it manually, and the kept instance still vends the build role's credentials via IMDS — terminate it as soon as inspection is done, then revert with --terminate-instance-on-failure.

Common errors

• Build fails at launch because the parent AMI can't be found or isn't available — and re-runs keep failing the same way: the recipe is likely pinned to a hardcoded AMI ID that has since been deregistered or unshared. Repoint the recipe's parent image at an ssm: parameter or an Image Builder image ARN with x.x.x (creating-images.md step 2).

• SSM unreachable (InvalidInstanceId ... not in a valid state / step timeout): the instance has no path to Systems Manager. Causes: account has no default VPC (builds need --subnet-id/--security-group-ids), private subnet without VPC endpoints (needs ssm, ssmmessages, ec2messages, and imagebuilder, plus S3; add logs for CloudWatch logging), or the role is missing AmazonSSMManagedInstanceCore. A component-download failure in the logs (GetComponent ... dial tcp ... i/o timeout) usually points at the same connectivity gap — check the imagebuilder VPC endpoint in particular.

• Unable to bootstrap TOE: two common causes to check — the instance can't fetch the AWSTOE bootstrap from the regional ec2imagebuilder-toe-<region>-prod S3 bucket (S3 endpoint policy), or a hardened (CIS/STIG) base image mounts /tmp noexec; setting the recipe's workingDirectory somewhere executable is the usual fix for the latter.

• ssm:ListCommandInvocations returned terminal state Failed at ApplyBuildComponents: often a component rebooted the instance outside the supported reboot protocol (Windows updates are the usual suspect) — check the logs for a step that restarted mid-run, and use the exit-code protocol (194/3010) or the Reboot action module instead (other-image-types.md has the Windows specifics).

• The aws-cli-version-2-linux component fails during its install steps: the base image may already ship the AWS CLI, and the preinstalled copy collides with the component's install. Check whether the base includes it (which aws on the base image, or its documentation) — if so, remove the component; it's redundant there.

• inspector-test-linux/-windows component fails with an Inspector CreateResourceGroup AccessDenied: that component targets the retired Inspector Classic — remove it and use the build's native scanning instead (--image-scanning-configuration imageScanningEnabled=true, creating-images.md step 7b).

• Pipeline is ENABLED but scheduled dependency-update runs never happen: compare dateLastRun/dateNextRun from get-image-pipeline against the cron, then check the recipe's references are update-detectable — a pinned AMI ID or full component build version never registers an update, and an ssm: parent image only counts if Image Builder can read the parameter at schedule time (creating-images.md step 7b has the trigger requirements).

• You are not authorized to use the provided image on create-image-recipe: usually the CALLING role, not the build role — it needs ec2:DescribeImages to validate the parent image.

• Scheduled pipeline silently stopped building: pipelines auto-disable after consecutive failed scheduled builds (5 if unset; the pipeline's autoDisablePolicy sets the threshold — manual runs don't count toward it, and any success or pipeline update resets the counter). Check get-image-pipeline for status: DISABLED, fix the underlying failure, then re-enable with update-image-pipeline (a full replacement — re-pass every field).

• InsufficientInstanceCapacity: the infrastructure configuration accepts a list of instance types — pass 2–3 so Image Builder can pick an available one.

• InvalidParameterValueException — "The provided instance profile does not exist" on create-infrastructure-configuration right after creating the role: IAM propagation. Wait 10–15 seconds and retry.

• ResourceAlreadyExistsException on create-component: the identical document already exists — reuse it (the existing ARN is in the error). Do not delete components to resolve this.

• InvalidParameterValueException: almost always a malformed identifier. Use the exact ARN a create call returned; check the parent-image form (AMI ID, ssm: parameter, Image Builder ARN, or Marketplace ID — an arn:aws:ec2:...:image/... ARN is rejected).

• Image FAILED after build and tests succeeded, ssm:PutParameter AccessDenied in state.reason: the SSM-parameter distribution permissions gotcha — the service-linked role can only write under /imagebuilder/. Fixes in distribution-options.md (rename the parameter under /imagebuilder/, or use an execution role); the output AMI already exists, so publish the parameter manually or start a new build (retry-image doesn't cover integration-stage failures).

• Container build FAILED after components succeeded, ECR AccessDenied/denied in logs: the build role is missing EC2InstanceProfileForImageBuilderECRContainerBuilds, or the target ECR repository doesn't exist in that Region (see other-image-types.md).

• Build stage succeeds but the TEST stage fails at LaunchInstance or times out: something the build changed broke the output AMI's connectivity — commonly networking changes or hardening that blocks the SSM agent on the fresh test instance. Compare what the components changed against the SSM requirements above.

• macOS build never launches an instance: no capacity on the Mac dedicated host, missing Dedicated Host placement (tenancy host plus a host target) in the infrastructure configuration, or host family vs image architecture mismatch — the launch error appears in image.state.reason.

• Windows build looks hung: update-windows and Sysprep legitimately run a long time — check the CloudWatch log group is still progressing before declaring it stuck. A step restarting repeatedly in the logs is an unguarded exit-3010 reboot loop (see other-image-types.md).

• Stuck or unwanted in-flight build: aws imagebuilder cancel-image-creation --image-build-version-arn ${build_arn}. For test- or distribution-stage failures fixed in place (flaky tests, permissions, target roles), retry-image retries from the failed stage without rebuilding the image.

• CLI ParamValidation errors on --filters: quote filter values containing spaces: --filters "name=name,values=Amazon Linux 2023 x86".

A note on baked-AMI oddities

If an instance launched from the output AMI "behaves oddly" (SSM agent missing, SSH host keys regenerated, machine-id reset): that's Image Builder's post-build cleanup, which sanitizes the image before snapshotting. The SSM agent uninstall is controlled by systemsManagerAgent.uninstallAfterBuild in the recipe.