AWS Fault Injection Service

AWS FIS Experiment Workflow

Follow this procedure to build, run, monitor, and analyze a FIS experiment. Consult fis-api-reference.md before writing any command and fis-actions-reference.md to choose actions/targets.

Contents

• 0. Plan first (do not skip)

• 1. Create (or reuse) the experiment IAM role

• 2. Author the experiment template

• 3. Add stop conditions (blast-radius guardrail)

• 4. Enable experiment logging (strongly recommended)

• 5. (Optional) Configure an experiment report

• 6. (Optional) Experiment options & multi-account

• 7. Preview targets before running (recommended)

• 8. Start and monitor the experiment

• 9. Stop manually if needed

• 10. Analyze results

• 11. Iterate the template

• CloudFormation option

0. Plan first (do not skip)

1. Define steady state — the business/technical metrics that mean the app is healthy (latency, error rate, successful transactions). These become your stop-condition alarm(s).

2. Form a hypothesis — "when the fault is injected, the system stays within steady state because of a specific control."

3. Decide blast radius — narrowest useful target (COUNT(1) / low PERCENT), pre-production first. Only widen after it passes.

4. Confirm this is authorized — production fault injection needs change-management approval.

1. Create (or reuse) the experiment IAM role

The template needs a roleArn FIS assumes to perform actions. Create it with a trust policy for fis.amazonaws.com and confused-deputy conditions, plus least-privilege permissions for the target service APIs (and SSM / logs / report S3+CloudWatch if used). See fis-security.md for the exact trust policy and permission examples.

2. Author the experiment template

An experiment template combines: description, roleArn, actions, targets, stopConditions, optional logConfiguration, optional experimentReportConfiguration, optional experimentOptions, and tags.

Option A — from a scenario (recommended when one fits)

Scenarios are console-only. In the FIS console: Scenario library → pick a scenario → Create template with scenario → fill missing parameters, choose the service-access role, add stop conditions and logging → Create experiment template. To automate later, export the created template's JSON. (See scenario list in fis-actions-reference.md.)

Option B — hand-authored template (CLI/JSON)

Build a JSON document and pass it to create-experiment-template. Minimal shape:

{
  "description": "Reboot one prod EC2 instance; stop if error rate alarms",
  "roleArn": "arn:aws:iam::123456789012:role/FISExperimentRole",
  "actions": {
    "RebootInstance": {
      "actionId": "aws:ec2:reboot-instances",
      "targets": { "Instances": "oneProdInstance" }
    }
  },
  "targets": {
    "oneProdInstance": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": { "env": "prod" },
      "selectionMode": "COUNT(1)"
    }
  },
  "stopConditions": [
    { "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:HighErrorRate" }
  ],
  "tags": { "Name": "reboot-one-prod-instance", "team": "checkout" }
}

Create it:

aws fis create-experiment-template --cli-input-json file://template.json

stopConditions is required in the template. Use [{"source":"none"}] only when you genuinely have no alarm — strongly discouraged for anything beyond a sandbox. Prefer a real aws:cloudwatch:alarm on your steady-state metric.

Sequencing actions

Actions run in parallel by default. To sequence, set startAfter to the names of prerequisite actions. Use an aws:fis:wait action to insert a controlled pause (e.g., hold a fault for N minutes before a follow-on action or before letting steady state recover).

3. Add stop conditions (blast-radius guardrail)

A stop condition stops the experiment when a CloudWatch alarm enters ALARM. Base the alarm on the steady-state metric from step 0.

"stopConditions": [
  { "source": "aws:cloudwatch:alarm",
    "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:P99LatencyHigh" }
]

You can specify multiple alarms (subject to the account quota). Alarm/dashboard creation itself belongs to the AWS Observability skill.

4. Enable experiment logging (strongly recommended)

Capture per-experiment records (experiment-start, target-resolution-*, action-start/end/ error, experiment-end) to CloudWatch Logs and/or S3. logSchemaVersion must be 2.

FIS does not require logging, but without it there is no experiment-level audit trail beyond CloudTrail, which makes post-incident troubleshooting very difficult. Enable it for anything beyond a throwaway sandbox.

"logConfiguration": {
  "logSchemaVersion": 2,
  "cloudWatchLogsConfiguration": {
    "logGroupArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/fis/experiments:*"
  },
  "s3Configuration": { "bucketName": "my-fis-logs", "prefix": "experiments" }
}

Grant FIS the vended-logs delivery permissions for each destination. First run to a new destination is delayed ~15 s to configure delivery. Logging is disabled by default; enable via create-experiment-template or add later with update-experiment-template. Use logs (not the report) to troubleshoot failed experiments.

Encrypt both destinations. Experiment logs reveal resilience posture and architectural weaknesses. Encrypt the CloudWatch Logs log group with a KMS key (kmsKeyId at creation, or aws logs associate-kms-key), and enable SSE-KMS on the S3 bucket with a bucket policy that denies non-TLS requests (aws:SecureTransport). See fis-security.md — Data handling.

5. (Optional) Configure an experiment report

Generate a PDF summarizing the experiment plus a CloudWatch dashboard snapshot, delivered to S3.

"experimentReportConfiguration": {
  "outputs": { "s3Configuration": { "bucketName": "my-fis-reports", "prefix": "checkout" } },
  "dataSources": {
    "cloudWatchDashboards": [
      { "dashboardIdentifier": "arn:aws:cloudwatch::123456789012:dashboard/CheckoutHealth" }
    ]
  },
  "preExperimentDuration": "PT20M",
  "postExperimentDuration": "PT20M"
}

• Reports are NOT generated for cancelled runs or target-preview (actionsMode: skip-all) runs. preExperimentDuration and postExperimentDuration are ISO 8601 and default to 20 min each; both are capped, as is the number of dashboards per report. Check the current values in the FIS quotas documentation rather than assuming a fixed limit.

• Encrypt the report bucket. Reports are PDFs summarizing resilience posture — enable SSE-KMS and enforce TLS with a bucket policy condition on aws:SecureTransport, the same way as the log destinations above.

• Report role permissions: cloudwatch:GetDashboard, cloudwatch:GetMetricWidgetImage, s3:GetObject, s3:PutObject (+ kms:GenerateDataKey, kms:Decrypt for CMK-encrypted buckets). Use the report only for successfully completed runs — troubleshoot failures with logs.

6. (Optional) Experiment options & multi-account

experimentOptions on the template controls:

• accountTargeting: single-account (default) or multi-account.

• emptyTargetResolutionMode: fail (default) or skip (don't fail when a target resolves to zero resources).

• actionsMode: run-all (normal) or skip-all (target preview — resolves targets without running actions).

Multi-account experiments run from an orchestrator account against resources in target accounts (same Region). Set accountTargeting: multi-account, then add a target account configuration per account (account ID + IAM role FIS assumes there + optional description):

aws fis create-target-account-configuration \
  --experiment-template-id EXTxxxx --account-id 222233334444 \
  --role-arn arn:aws:iam::222233334444:role/FISTargetAccountRole \
  --description "checkout target account"

Target accounts are notified via their AWS Health dashboards. Prefer consistent resource tags across accounts and target AZs by AZ ID. Per-account action quotas apply.

7. Preview targets before running (recommended)

Generate a target preview (run with actionsMode: skip-all, or use the console/get-experiment resolved-targets view) to confirm the exact resources that will be affected. This is the cheapest way to catch an over-broad tag/filter before a real fault.

8. Start and monitor the experiment

# Start
aws fis start-experiment --experiment-template-id EXTxxxx \
  --tags Name=game-day-checkout

# Watch state
aws fis get-experiment --id EXPxxxx --query 'experiment.state'

# List which resources FIS actually selected
aws fis list-experiment-resolved-targets --id EXPxxxx

States: pending → initiating → running → completed | stopping → stopped | failed. Watch your steady-state dashboard live. A stopped/failed experiment cannot be resumed.

9. Stop manually if needed

aws fis stop-experiment --id EXPxxxx

Use this if you observe unacceptable impact the stop-condition alarm didn't catch. Afterward, consider tightening the alarm so the guardrail catches it automatically next time.

10. Analyze results

1. Compare steady-state metrics before/during/after; confirm the hypothesis and measure actual recovery time (RTO) vs. objective.

2. Read experiment logs for the action/target timeline and any action-error reasons.

3. If enabled, retrieve the S3 experiment report as evidence of the test.

4. If validating a Resilience Hub finding, only now mark it resolved (see aws-resilience-lifecycle).

5. Iterate: widen blast radius, add multi-fault scenarios, or promote from pre-prod to prod under change management. If the experiment surfaced a defect, fix it and re-run to confirm.

11. Iterate the template

aws fis update-experiment-template --id EXTxxxx --cli-input-json file://template-v2.json
aws fis delete-experiment-template --id EXTxxxx   # when retiring

CloudFormation option

Model the template as AWS::FIS::ExperimentTemplate (and AWS::FIS::TargetAccountConfiguration for multi-account) so experiments are version-controlled and reviewed like other infrastructure. The property shape mirrors the CLI/JSON above.