Managing Amazon Kinesis Data Streams
AWSOfficialPublished by AWS
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides operational guidance for configuring and managing Amazon Kinesis Data Streams (KDS) delivery channels. It focuses on serverless, zero-ops delivery of stream data directly to Apache Iceberg tables on S3 Tables or general-purpose S3 buckets. The skill covers channel lifecycle management, IAM configuration, monitoring via CloudWatch, and troubleshooting common delivery issues, ensuring efficient data movement without the need for additional consumers or Flink applications.
What it does
- Configures KDS channels for direct delivery to S3 or Apache Iceberg tables.
- Provides IAM policy templates for secure service execution.
- Explains monitoring strategies using CloudWatch metrics and delivery logs.
- Outlines troubleshooting steps for failed channels and data freshness issues.
- Defines hard limits and prerequisites for KDS delivery configurations.
When to use it
- When you need to deliver KDS data to S3 or Iceberg tables without managing consumers.
- When building a data lake or lakehouse architecture on AWS.
- When you require a serverless, append-only delivery mechanism for streaming data.
- When troubleshooting delivery failures or monitoring data freshness in KDS channels.
Advantages
- Eliminates the need for managing Kinesis consumers or Flink applications.
- Channels consume no shard capacity or enhanced fan-out slots.
- Fully managed, serverless delivery architecture.
- Supports direct integration with Apache Iceberg for queryable streaming data.
Usage examples
- “How do I set up a channel to deliver Kinesis stream data to an S3 bucket?”
- The agent provides the necessary IAM trust and permission policies, along with the configuration steps for the general-purpose S3 delivery channel.
- “My Kinesis delivery channel is ACTIVE but no data is appearing in S3. How do I debug this?”
- The agent guides you to check CloudWatch metrics like FailedRecordCount, verify IAM permissions, and enable delivery logging to identify potential access or schema errors.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/aws/managing-amazon-kinesis-data-streams/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip managing-amazon-kinesis-data-streams.zip -d .agents/skills
Documentation by AWS
Amazon Kinesis Data Streams
Overview
Domain expertise for Amazon Kinesis Data Streams. This skill covers streaming tables and Amazon S3 delivery, which provide serverless, fully managed delivery from a KDS stream directly to Apache Iceberg tables on S3 Tables or to general-purpose S3 buckets — no consumers, no Firehose, no Flink required for append-only delivery.
Delivery is configured as a channel on a stream via CreateChannel. Channels consume no shard capacity and no enhanced fan-out slots, so they do not compete with existing consumers.
The AWS MCP server is recommended for executing AWS operations. When the MCP server is not available, use the AWS CLI or shell commands instead.
Which Workflow Do You Need?
| Customer Intent | Reference |
|---|---|
| Deliver stream data to Apache Iceberg tables on S3 Tables — setup, IAM, Glue schema, type mapping, partitioning, channel lifecycle | references/streaming-tables.md |
| Deliver stream data to a general-purpose S3 bucket as objects — setup, IAM, compression, storage class, output key templates, channel lifecycle | references/general-purpose-delivery-to-s3.md |
| Build a lakehouse / data lake from Kinesis; make streaming data queryable in Athena | references/streaming-tables.md |
| Zero-ops, serverless alternative to Firehose or Kinesis consumers for KDS → S3 delivery | references/general-purpose-delivery-to-s3.md |
| CloudWatch metrics and alarms, delivery logging, CloudTrail, encryption, channel states, quotas, naming rules, what can be updated | references/monitoring-security-and-limits.md |
| Channel is ACTIVE but no data arrives; records landing in the DLQ; rising data freshness | Start with references/monitoring-security-and-limits.md, then the destination-specific troubleshooting table |
| Deliver to both S3 and S3 Tables from one stream | Supported — create one channel per destination |
Prerequisites
Both destinations require an ON_DEMAND stream — provisioned-mode streams are not supported. Each channel also needs an IAM service execution role trusted by kinesis.amazonaws.com, and a destination in the same Region as the stream.
Hard Limits to Check First
These rule out the feature entirely, so confirm them before designing a solution:
Append-only — no CDC, upserts, or deletes
No schema evolution — a schema change means deleting and recreating the channel
No backfill — only records produced after the channel is
ACTIVEare deliveredNo transformations — records are delivered as-is
New table per channel — cannot deliver into an existing Iceberg table
Data freshness is 300–900 seconds — not sub-minute
Same Region only — cross-account is supported, cross-Region is not
If any of these block the customer, recommend Managed Service for Apache Flink instead.
Additional Resources
Files
- ATTRIBUTION.mdAdded by AutoGineer961 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer67 B
- SKILL.md4.4 KiB
- references/general-purpose-delivery-to-s3.md13.7 KiB
- references/monitoring-security-and-limits.md7.5 KiB
- references/streaming-tables.md19.4 KiB
references/general-purpose-delivery-to-s3.md
General purpose delivery to Amazon S3
Delivers records from a Kinesis stream to a general-purpose S3 bucket as objects, with configurable compression, storage class, and output key layout. Serverless, no consumer management, 5-minute minimum data freshness. Records are delivered in their original source format — no transformation is applied.
Records are batched into objects: multiple stream records land in a single S3 object, keyed by the output key template.
See monitoring-security-and-limits.md for metrics, alarms, channel states, encryption, quotas, and update constraints.
Constraints
ON_DEMAND streams only — provisioned-mode streams are not supported
Data freshness is 300–900 seconds. No minimum throughput requirement for the 5-minute window
No backfill — only records produced after the channel is
ACTIVEare deliveredNo transformations, filtering, or CDC
Cross-Region delivery is not supported; the bucket must be in the stream's Region
One S3 channel and one S3 Tables channel per stream at launch — adjustable, and expected to rise over time. Channel reads consume no additional shard capacity
If the customer needs transformations, filtering, or CDC, recommend Managed Service for Apache Flink instead.
Prerequisites
KDS stream in ON_DEMAND mode
General-purpose S3 delivery bucket in the same Region as the stream
General-purpose S3 bucket for the DLQ — optional here; if omitted, errors are written to the delivery bucket under an error prefix. Configure one explicitly so failures do not land among delivered records
IAM service execution role (see IAM Setup below)
Customer-managed KMS key if using SSE-KMS (recommended)
Record Formats
RecordFormat | Description |
|---|---|
BYTE_ARRAY | Raw bytes written as-is — no schema |
STRING | UTF-8 string records written as-is |
JSON | Plain JSON records |
GSR_JSON | GSR-serialized JSON — schema ID embedded per record; requires GSRSchemaARN |
Only GSR_JSON requires a schema registry. When the customer asks for "raw" delivery, byte-for-byte copies, or does not mention a specific serialization, use BYTE_ARRAY — it writes each record's bytes unmodified regardless of source encoding. Use JSON or STRING only when the customer explicitly describes their records as JSON or plain text.
Destination Options
| Option | Values | Notes |
|---|---|---|
Compression | NONE, GZIP, ZSTD | Required. GZIP balances ratio and speed; ZSTD gives a higher ratio with faster decompression |
StorageClass | STANDARD, INTELLIGENT_TIERING, GLACIER_IR | Defaults to STANDARD |
OutputKeyTemplate | Template string | Optional — a default is applied if omitted |
DataFreshnessInSeconds | 300–900 | Defaults to 300 |
IAM Setup
Trust Policy
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "kinesis.amazonaws.com"},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {"aws:SourceAccount": "ACCOUNT_ID"},
"ArnLike": {"aws:SourceArn": "arn:aws:kinesis:REGION:ACCOUNT_ID:channel/*"}
}
}]
}Permission Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DeliveryBucketList",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::DELIVERY_BUCKET",
"arn:aws:s3:::DELIVERY_BUCKET/*"
]
},
{
"Sid": "DeliveryBucketWrite",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:UploadPart",
"s3:CompleteMultipartUpload",
"s3:CreateMultipartUpload",
"s3:ListMultipartUploads",
"s3:ListMultipartUploadParts"
],
"Resource": "arn:aws:s3:::DELIVERY_BUCKET/PREFIX*"
},
{
"Sid": "DLQBucketAccess",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:UploadPart",
"s3:CompleteMultipartUpload",
"s3:CreateMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::DLQ_BUCKET",
"arn:aws:s3:::DLQ_BUCKET/*"
]
},
{
"Sid": "KMSAccess",
"Effect": "Allow",
"Action": ["kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"],
"Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/KEY_ID"
},
{
"Sid": "KMSAccessDLQ",
"Effect": "Allow",
"Action": ["kms:Decrypt", "kms:GenerateDataKey"],
"Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/DLQ_KEY_ID",
"Condition": {
"StringEquals": {"kms:ViaService": "s3.REGION.amazonaws.com"},
"StringLike": {"kms:EncryptionContext:aws:s3:arn": "arn:aws:s3:::DLQ_BUCKET/*"}
}
},
{
"Sid": "CloudWatchLogsAccess",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/kinesis/CHANNEL_NAME:*"
}
]
}Statement notes:
KMSAccesscovers the delivery bucket. Grant it unconditionally — do not add akms:ViaService/EncryptionContextcondition.CreateChannelvalidates the role's KMS access with a direct KMS call, not one proxied through S3, so aViaService: s3.REGION.amazonaws.comcondition passes at runtime delivery but fails channel creation. Includekms:DescribeKeyalongsidekms:Decrypt/kms:GenerateDataKey.KMSAccessDLQis required when the DLQ bucket uses SSE-KMS. Omit if the DLQ uses SSE-S3 (AES-256).The
aws/kinesismanaged key cannot be used for destination encryption — use a customer managed KMS key with its full ARN.CloudWatchLogsAccessis required whenever delivery logging is enabled. Without it, delivery errors are invisible.
Cross-account delivery
Supported within the same Region. Attach a bucket policy on the destination bucket granting the source account's execution role s3:PutObject (plus the multipart actions above) on the delivery prefix.
Create the Channel
1. Create supporting resources
aws s3 mb s3://my-kds-delivery-$ACCOUNT --region $REGION
aws s3 mb s3://my-kds-dlq-$ACCOUNT --region $REGION
# CW log group — create before the channel
aws logs create-log-group \
--log-group-name "/aws/kinesis/my-s3-channel" --region $REGION
aws logs put-retention-policy \
--log-group-name "/aws/kinesis/my-s3-channel" \
--retention-in-days 30 --region $REGIONConfigure SSE-KMS on both buckets.
2. Create the channel
Pass --service-execution-role-arn, --stream-configuration-list, --s3-destination-configuration, and --logging-configuration as separate top-level parameters. If StorageConfiguration.OutputKeyTemplate is set, write that JSON blob to a file and reference it with file:// — the !{...} template syntax gets mangled by shell quoting (even single quotes) when passed inline. Other JSON blobs without !{...} can be passed inline or via file interchangeably.
cat > s3-dest.json << 'EOF'
{
"DataFreshnessInSeconds": 300,
"DeadLetterQueueS3Configuration": {
"BucketARN": "arn:aws:s3:::my-kds-dlq-ACCOUNT_ID",
"ExpectedBucketOwner": "ACCOUNT_ID",
"ErrorOutputPrefix": "errors/my-s3-channel/"
},
"StorageConfiguration": {
"BucketARN": "arn:aws:s3:::my-kds-delivery-ACCOUNT_ID",
"ExpectedBucketOwner": "ACCOUNT_ID",
"StorageClass": "STANDARD",
"CompressionType": "GZIP",
"OutputKeyTemplate": "data/!{channel-name}/!{yyyy}/!{MM}/!{dd}/!{HH}/!{channel-id}-!{mm}!{extension}"
}
}
EOF
aws kinesis create-channel \
--region $REGION \
--channel-name my-s3-channel \
--service-execution-role-arn $SER_ARN \
--stream-configuration-list '[{"StreamARN": "'$STREAM_ARN'", "RecordConfiguration": {"RecordFormatType": "BYTE_ARRAY"}}]' \
--s3-destination-configuration file://s3-dest.json \
--logging-configuration '{"CloudWatchLogs": {"Enabled": true, "LogGroupName": "/aws/kinesis/my-s3-channel", "LogStreamName": "DestinationDelivery"}}'ExpectedBucketOwner (12-digit account ID) is required on both DeadLetterQueueS3Configuration and StorageConfiguration. CompressionType (NONE/GZIP/ZSTD) is required inside StorageConfiguration. OutputKeyTemplate is optional there — see the Output Key Template section below for the supported variables.
For STRING, JSON, or GSR_JSON, set RecordFormatType accordingly in stream-config.json. For GSR_JSON, also add "GSRSchemaARN": "arn:aws:glue:...".
3. Wait for ACTIVE
aws kinesis describe-channel \
--region $REGION \
--channel-arn $CHANNEL_ARN \
--query 'ChannelDescription.ChannelStatus'4. Produce test records
aws kinesis put-record \
--region $REGION \
--stream-arn $STREAM_ARN \
--data "test-record" --partition-key "key1" \
--cli-binary-format raw-in-base64-outOutput Key Template
Default when OutputKeyTemplate is omitted:
kinesis-channel/!{channel-name}/!{channel-id}/!{yyyy}/!{MM}/!{dd}/!{HH}/!{channel-name}-!{channel-id}-!{yyyy}-!{MM}-!{dd}-!{HH}-!{mm}!{extension}Variables
| Variable | Description | Example |
|---|---|---|
!{channel-name} | Channel name | my-channel |
!{channel-id} | Channel identifier | abc123def456 |
!{stream-name} | Source Kinesis Data Streams stream name | my-stream |
!{yyyy} | Four-digit year (UTC) | 2026 |
!{yy} | Two-digit year (UTC) | 26 |
!{MM} | Month, zero-padded (UTC) | 07 |
!{dd} | Day of month, zero-padded (UTC) | 20 |
!{HH} | Hour, 24-hour zero-padded (UTC) | 14 |
!{mm} | Minute, zero-padded (UTC) | 30 |
!{extension} | File extension implied by Compression | .gz, .zst |
Use !{channel-id} combined with time variables down to !{mm} to keep object keys unique across delivery cycles — the default template above is a working pattern for this.
Rules
Maximum 1024 characters
Must contain at least one variable
Must not start or end with
/Must not contain path traversal (
..)All variable references must be balanced (matching
!{and})Allowed characters: alphanumerics, hyphens, underscores, periods, forward slashes, and variable references
Examples
| Template | Valid | Reason |
|---|---|---|
data/!{channel-name}/!{yyyy}/!{MM}/!{dd}/!{HH}/!{channel-id}-!{mm}!{extension} | Yes | Has variables, unique per channel/minute |
!{channel-name}/!{yyyy}-!{MM}-!{dd}/!{HH}!{mm}-!{channel-id}!{extension} | Yes | Valid characters, date-based prefix |
!{channel-name}/!{channel-id}/!{yyyy}!{MM}!{dd}!{HH}!{mm} | Yes | Minimal valid template |
/data/!{channel-name}/!{yyyy} | No | Starts with / |
data/!{channel-name}/!{yyyy}/ | No | Ends with / |
data/../!{channel-name}/!{yyyy} | No | Path traversal |
static-prefix-only | No | No variables |
data/!{channel-name}/!{yyyy | No | Unbalanced reference |
Match the template to downstream query patterns — use date-based prefixes when consumers filter by time range.
Channel Management
All lifecycle calls except create-channel and list-channels identify the channel by --channel-arn alone — not stream ARN + channel name.
# Describe
aws kinesis describe-channel --region $REGION --channel-arn $CHANNEL_ARN
# List channels on a stream
aws kinesis list-channels --region $REGION --stream-filter "{\"StreamARN\": \"$STREAM_ARN\"}"
# Update — only DataFreshnessInSeconds and logging are mutable
aws kinesis update-channel --region $REGION \
--channel-arn $CHANNEL_ARN \
--s3-destination-configuration '{"DataFreshnessInSeconds": 600}'
# Delete — already-delivered objects are NOT removed
aws kinesis delete-channel --region $REGION \
--channel-arn $CHANNEL_ARNDestination bucket, output key template, record format, service role, and source stream are immutable. Delete all channels on a stream before deleting the stream.
Verify Delivery
# Delivered objects — wait DataFreshnessInSeconds after producing
aws s3 ls s3://my-kds-delivery-$ACCOUNT/data/ --recursive
# Failed records
aws s3 ls s3://my-kds-dlq-$ACCOUNT/errors/ --recursive
# Delivery errors
aws logs filter-log-events \
--log-group-name "/aws/kinesis/my-s3-channel" --region $REGIONTroubleshooting
| Symptom | Likely Cause | Action |
|---|---|---|
Channel stuck in CREATING, then FAILED | Invalid role ARN, insufficient permissions, or bucket missing / in another Region | Read ChannelStatusReason from describe-channel. FAILED is not recoverable — fix, delete, recreate |
| No objects in the delivery prefix | Records produced before ACTIVE (no backfill), or silent delivery failure | Confirm records were sent after ACTIVE; check logs and DLQ |
| Objects in DLQ | Record format does not match the configured RecordFormat, KMS failure, or template rendering error | Inspect DLQ keys and bodies plus CloudWatch Logs |
| No CloudWatch logs written | Service role missing logs:CreateLogStream / logs:PutLogEvents | Add CloudWatch Logs permissions |
AccessDenied in logs | Role policy or bucket policy changed after creation, trust policy broken, or KMS key policy denies access | Review the role policy, bucket policy, trust policy, and KMS key policy |
Rising DataChannel.DataFreshness | Low throughput against a low freshness setting | Raise DataFreshnessInSeconds so more data batches per cycle |
| Channel suspended | Bucket owner mismatch or destination unavailable | Cannot be resumed — create a new channel |
| Channel not available for the stream | Stream is in provisioned mode | Switch the stream to On-Demand |
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS
- Publisher
- AWS (Official)
- Source
- aws/agent-toolkit-for-aws · skills/specialized-skills/analytics-skills/managing-amazon-kinesis-data-streams · commit 0a218f8
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by AWS.
Related skills
Modal
K-Dense· Community
A serverless cloud platform for deploying and scaling Python applications, AI models, and GPU-accelerated workloads.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreApache-2.0Low riskGitHub stars of the source repository56.4k
Cloud Infrastructure Network Engineer
Seth Hobson· Community
An expert network engineer persona specializing in cloud architecture, security, and performance optimization.
Cloud and DevOpsClaude CodeMITGitHub stars of the source repository42.3k
Deployment Validation Cloud Architect
Seth Hobson· Community
Expert cloud architect for multi-cloud infrastructure, cost optimization, and resilient system design.
Cloud and DevOpsClaude CodeMITGitHub stars of the source repository42.3k
Airflow Dag Patterns
Seth Hobson· Community
Best practices and design patterns for building and maintaining production-ready Apache Airflow DAGs.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Cost Optimization
Seth Hobson· Community
A comprehensive framework for reducing cloud infrastructure costs across AWS, Azure, GCP, and OCI.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Hybrid Cloud Networking
Seth Hobson· Community
A guide to configuring secure, high-performance network connectivity between on-premises infrastructure and major cloud providers.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download managing-amazon-kinesis-data-streams.zip