Debugging Mwaa Workflow

MWAA Failure Catalog

Category-keyed context checks and remediation. Categorize in this priority order: infra, then drift, then code-data. Apply the first that matches the evidence.

Category: infra (check first)

DAG not appearing in the UI

Not a single cause. When you explain the causes — or how to check a DAG that is not appearing — cover all five below, each paired with its specific check, rather than stopping at the first likely one. The Airflow REST API via invoke-rest-api is the fastest check for most (see provisioned-diagnostics.md): a thorough answer names both scan-interval config keys in step 2 and the GET /importErrors and GET /dags/{dag_id} checks (the latter reading last_parsed_time, fileloc, and is_active/is_stale). Do not conclude the code is broken until 1 and 2 are ruled out.

1. Import / parse error — the file is present but fails to parse, so it never registers. Check GET /importErrors (Airflow REST API) for the file's traceback, or the DAGProcessing CloudWatch log group. Fix the code; it reappears on the next parse.

2. Scheduler scan interval not yet elapsed — a valid new file only lists after the DAG-directory scan runs. That cadence is scheduler.dag_dir_list_interval (Airflow 2.x) / dag_processor.refresh_interval (Airflow 3.x). The S3-to-container sync is only ~30s; the listing delay is this scan interval. Wait one interval before assuming a defect.

3. dag_id collision — if another file already defines the same dag_id, Airflow keeps one and silently drops the duplicate, so the new file "does not appear." Confirm the dag_id is unique across the whole dags/ prefix.

4. Parsed but not yet registered/active — after the first successful parse the DAG registers. GET /dags/{dag_id} and read last_parsed_time and fileloc (present on both Airflow 2.x and 3.x) to confirm it parsed and points at the file you uploaded. Check is_active on Airflow 2.x; on Airflow 3.x that field was replaced by is_stale (inverted — a stale DAG's file no longer parses). The has_import_errors flag on the same object points straight at a parse failure.

5. S3-sync delay — the object must land under the environment's DagS3Path. Confirm the object exists at that key; MWAA syncs it to the containers within ~30s.

Disambiguating AccessDenied / EntityNotFound

A denial on a resource is one of four failures; the fix differs for each. Read the resource ARN from the exception. Determine the account and region the resource is expected in — this is NOT always the environment's own account. Cross-account access is common: the execution role may access resources in another account via resource policies (S3, SNS, KMS, etc.) or via sts:AssumeRole into a role in the target account (configured as an Airflow AWS connection with IAM role type in Provisioned). Check existence from the account/role that should have access:

1. Resource exists, role lacks the action (true IAM gap) — the classic case. Add the missing action to the execution role (same-account) or to the assumed role (cross-account). Do not broaden to wildcard. HUMAN-GATED.

2. Resource does not exist — the identifier is a placeholder, a typo, or in a different region. AccessDenied (not NoSuchBucket) is common here because IAM denies before existence is revealed. Fix the identifier in the DAG/workflow (ARTIFACT fix via authoring-mwaa-workflow), not the role.

3. Resource is cross-account, resource-policy path — the ARN's account differs from the execution role's and access is via a resource policy on the target (S3 bucket policy, SNS topic policy, KMS key policy). Needs both: the resource policy granting the execution role's principal, and the execution role having the action. A role change alone or a resource policy alone will not resolve it. HUMAN-GATED.

4. Resource is cross-account, assume-role path — the task assumes a role in the target account (Provisioned: Airflow AWS connection with IAM role type; Serverless: not applicable — no connections). The denial is on the assumed role, not the execution role. Check: (a) execution role has sts:AssumeRole on the target role ARN, (b) the target role's trust policy allows the execution role as principal, (c) the target role has the needed action on the resource. HUMAN-GATED.

Do not reflexively broaden IAM: adding permissions for a resource that does not exist (case 2) hides the real bug and leaves a standing over-grant.

Extracting the required IAM policy from an AccessDeniedException

AWS error messages for access denials follow a consistent format that contains everything needed to construct the missing IAM statement:

User: arn:aws:sts::<account>:assumed-role/<role>/...
is not authorized to perform: <action>
on resource: <resource-arn>
because no identity-based policy allows the <action> action

Extract these three fields and construct the minimum policy:

{
  "Effect": "Allow",
  "Action": ["<action>"],
  "Resource": "<resource-arn>"
}

For the resource ARN, determine the appropriate scope:

• If the ARN ends in a specific resource ID, keep it exact for least privilege.

• If the ARN contains a log group or prefix pattern (e.g., /aws/sagemaker/TrainingJobs:*), use the pattern with a wildcard suffix to cover log streams or child resources.

• If the service requires multiple related actions together (e.g., DescribeLogStreams often needs GetLogEvents), check AWS documentation for the minimum action set.

After identifying the needed statement, verify which role requires it:

• The MWAA execution role (for actions the DAG tasks perform directly).

• A service role (e.g., Glue role, SageMaker role) if the denied principal is not the MWAA role.

• A cross-account assumed role (case 4 above).

Present the exact policy statement and target role to the user as a HUMAN-GATED remediation.

Category: drift (check when code did not change)

A DAG that passed last week and now fails on import, with no code change, is almost always dependency drift. MWAA re-resolves dependencies on environment update.

Three artifact paths can carry versioned objects (get-environment returns both the S3 path and the *S3ObjectVersion in use when bucket versioning is enabled). Always compare the specific object version the environment is using, not the latest object at that path — a newer upload not yet picked up by the environment is irrelevant to the current failure.

• requirements.txt (RequirementsS3Path / RequirementsS3ObjectVersion): compare the pinned version in use against the last known good. Unpinned packages resolve to new releases on environment update.

• plugins.zip (PluginsS3Path / PluginsS3ObjectVersion): may bundle .whl files providing Python packages outside of requirements.txt. A changed plugins.zip can introduce or remove packages, native shared libraries, or operator plugins. Diff contents (unzip -l) between the current version and the prior one.

• Startup script (StartupScriptS3Path / StartupScriptS3ObjectVersion): installs system libraries (yum/apt) that Python packages depend on (e.g., libpq-devel for psycopg2, unixODBC-devel for pyodbc). A changed or removed startup script can break packages that compiled against those libraries at install time.

• Check whether the environment version was bumped (get-environment AirflowVersion) — a version change re-resolves the entire provider stack.

Remediation to recommend: pin exact versions in requirements.txt and use the MWAA constraints file for the target Airflow version; pin .whl contents in plugins.zip; pin system library versions in the startup script; use S3 object versioning and reference explicit *S3ObjectVersion values in environment config to prevent unintended drift on upload.

Category: code-data (check last)

• Recent DAG changes: diff the DAG file in S3 against the prior version.

• Data volume spike: an operator that worked on small input may time out or OOM on large input — inspect input size for the failing run's data interval.

• Upstream health: a sensor timeout usually means the upstream producer is late, not that the DAG is broken.

• Remediation to recommend: fix the code or data; adjust timeouts/retries per the authoring-mwaa-workflow rules.

Category: serverless custom-code failures (Python/Bash)

MWAA Serverless PythonOperator/BashOperator tasks add failure modes the Provisioned rows above do not cover. Serverless only.