AWS Lambda

Troubleshooting Guide

Symptom-Based Diagnosis

High Latency

Possible causes (check in order):

1. Cold start — check if latency is only on first invocations after idle

2. Under-provisioned memory — more memory = more CPU

3. Slow external calls — database, HTTP APIs, other AWS services

4. Large deployment package — increases cold start time

Diagnosis steps:

• Use get_metrics to check duration (average vs p99) and memory utilization

• Enable X-Ray tracing to identify which segment is slow

• Check if function is in a VPC (adds ENI setup time on cold start)

Resolution:

• For cold starts: initialize SDK clients outside handler, reduce package size, consider provisioned concurrency

• For slow external calls: use connection reuse, add VPC endpoints, increase timeout

• For CPU-bound work: increase memory allocation

Function Errors

Possible causes:

1. Unhandled exceptions in application code

2. Timeout exceeded

3. Out of memory (OOM)

4. Permission denied on AWS API calls

Diagnosis steps:

• Use sam_logs to retrieve recent CloudWatch logs for the function

• Look for Task timed out, Runtime.ExitError, or AccessDeniedException messages

• Check the error rate trend with get_metrics

Resolution by error type:

Async Invocations and Throttling

Common misconception: Async Lambda invocations (InvocationType: Event) are subject to throttling like sync invocations.

Reality: Async invocations are always accepted — Lambda's Event Invoke Frontend queues the request without checking concurrency limits. The invocation call itself never returns a throttle error. Throttling is only checked later when the internal poller attempts to run the function synchronously. If throttled at that point, the event is returned to the internal queue and retried for up to 6 hours.

Practical implications:

• You do not need SNS or EventBridge as a throttle-protection buffer in front of async Lambda invocations — direct Lambda-to-Lambda async calls are safe

• If you need guaranteed delivery with a DLQ, configure one on the function directly

• Async invocations with reserved concurrency set to 0 will still be accepted but will fail during processing — they will retry and eventually go to the DLQ or on-failure destination

Throttling and Concurrency Limits

Symptoms: TooManyRequestsException, 429 errors from API Gateway, Throttles metric rising

Key concurrency facts:

• Default account limit: 1,000 concurrent executions per region (shared across all functions)

• Concurrency ≠ requests per second: Concurrency = avg_RPS × avg_duration_seconds

• Burst limit: Lambda can scale by 1,000 new concurrent executions per 10 seconds (on-demand)

Diagnosis:

1. Check ConcurrentExecutions and Throttles metrics in CloudWatch with get_metrics

2. Check if a single function is consuming all available concurrency

3. Run aws lambda get-account-settings to see your account limit vs reserved allocations

Resolution:

• Set reserved concurrency on high-priority functions to guarantee capacity

• Set reserved concurrency on lower-priority functions to cap their usage

• Request a concurrency quota increase via AWS Service Quotas if the account limit is the bottleneck

• Use SQS as a buffer in front of Lambda to absorb traffic spikes without throttling

Deployment Failures

Common errors and solutions:

API Gateway Issues

CORS Errors

Symptoms: Browser blocking requests, Access-Control-Allow-Origin errors

Checklist:

• Verify CORS is configured on the API Gateway (AllowOrigin, AllowMethods, AllowHeaders)

• Check that OPTIONS method returns correct headers

• Ensure AllowOrigin matches the frontend domain (not * in production)

• Verify Lambda response includes CORS headers if using proxy integration

5xx Errors

Symptoms: API returning 500/502/503 errors

Diagnosis:

• 502 Bad Gateway: Lambda returned invalid response format. Check that response includes statusCode and body.

• 503 Service Unavailable: Lambda throttled. Check concurrency limits.

• 500 Internal Server Error: Check Lambda logs for unhandled exceptions.

Event Source Mapping Issues

When to Use Which Tool

DynamoDB Streams — High Iterator Age

Symptoms: IteratorAge metric increasing in CloudWatch

Diagnosis steps:

1. Check ParallelizationFactor — default is 1, maximum is 10

2. Check function duration — slow processing causes backlog

3. Check for poison records causing repeated retries

4. Check concurrency — throttling prevents scaling

Resolution:

• Increase ParallelizationFactor and BatchSize

• Enable BisectBatchOnFunctionError to isolate bad records

• Set MaximumRetryAttempts to limit retries on persistent failures

• Use esm_optimize for specific tuning recommendations

Kinesis — Shard Throttling

Symptoms: ReadProvisionedThroughputExceeded errors

Resolution:

• Check if multiple consumers share the same shard (each shard supports 2 MB/s reads)

• Use enhanced fan-out for multiple consumers

• Consider switching to ON_DEMAND stream mode for automatic scaling

• Increase shard count for PROVISIONED mode

SQS — Messages Going to DLQ

Symptoms: Messages accumulating in dead-letter queue

Diagnosis:

• Check that VisibilityTimeout on the queue is >= Lambda function timeout

• Check for partial batch failures: enable ReportBatchItemFailures in FunctionResponseTypes

• Check maxReceiveCount in the redrive policy (too low causes premature DLQ routing)

Kafka/MSK — Connection Failures

Symptoms: ESM stays in Creating or Failed state

Use esm_kafka_troubleshoot with the error message. Common causes:

• Lambda not in same VPC as MSK cluster

• Security group missing inbound rule on ports 9092/9094

• IAM authentication not configured correctly

• SASL/SCRAM secret not in the correct format

VPC and Networking

Lambda Cannot Reach AWS Services

Symptoms: Timeouts when calling DynamoDB, S3, SQS from VPC-attached Lambda

Cause: Lambda in VPC private subnets cannot reach AWS service endpoints without a path.

Resolution options (choose one):

• Add VPC gateway endpoints for DynamoDB and S3 (free, recommended)

• Add VPC interface endpoints for other services (per-hour + per-GB cost)

• Add NAT Gateway in public subnet (higher cost, required for internet access)

• Use IPv6 + Egress-Only Internet Gateway for internet access — Lambda now supports IPv6, so if your VPC has IPv6 CIDR blocks and an Egress-Only Internet Gateway, Lambda functions can reach the internet without a NAT Gateway, eliminating NAT Gateway costs entirely

ENI Exhaustion

Symptoms: Lambda functions fail to start, ENILimitReached errors

Resolution:

• Use multiple subnets across AZs (each /24 subnet provides ~250 IPs)

• Set reserved concurrency to cap the maximum ENI usage

• Lambda uses Hyperplane ENIs which are shared, but high concurrency can still exhaust IPs

Lambda SnapStart Issues

SnapStart issues are specific to Java 11+, Python 3.12+, and .NET 8+ functions with SnapStart: ApplyOn: PublishedVersions.

Stale unique values

Symptom: All invocations share the same UUID, timestamp, or random value

Cause: Value was generated during initialization (before snapshot), so all restored environments use the same value.

Fix: Move any call to uuid.uuid4(), random, time.time(), or similar into the handler function body, not at module level.

Stale database connection after restore

Symptom: Database errors on the first call after a period of inactivity

Cause: The connection object was captured in the snapshot but the actual TCP connection is no longer valid after resume.

Fix: Validate the connection before use, or use a lambda_runtime_api_prepare_to_invoke hook to re-establish it after restoration.

SnapStart not activating

Symptom: Cold starts still slow despite SnapStart enabled

Check:

• SnapStart only applies to published versions and aliases pointing to them, not $LATEST

• The function is not using provisioned concurrency, EFS, or ephemeral storage > 512 MB

• The runtime is Java 11+, Python 3.12+, or .NET 8+

Debugging Workflow

When a function is failing and the cause is unclear, follow this sequence:

1. Check logs: Use sam_logs to get recent log output

2. Check metrics: Use get_metrics to identify error rate, duration, and throttle trends

3. Check configuration: Verify timeout, memory, VPC, and IAM settings in the SAM template

4. Test locally: Use sam_local_invoke with the failing event payload to reproduce

5. Test deployed function: Use sam remote invoke with the failing event to test directly in AWS — bypasses local environment differences

6. Trace calls: Enable X-Ray tracing to identify which downstream call is failing

7. Check dependencies: Verify external services (databases, APIs) are reachable and healthy