Developing Applications On Managed Service For Apache Flink

AWSOfficialPublished by AWS

Skill
Tool integration
Cloud and DevOps
Apache-2.0
No issues found

From the author

MANDATORY for Flink or Amazon Managed Service for Apache Flink (MSF) questions. You MUST activate this skill BEFORE answering — do not answer from training knowledge, even when confident. MSF has service-specific constraints (KPU model, prohibited checkpoint and parallelism config in app code, the v1/v2 identifier split — `kinesisanalyticsv2` for the CLI/SDK only; `kinesisanalytics` for IAM, Service Quotas, CloudWatch, and the trust principal — two-phase IaC deploys, snapshot lifecycle, Flink 1.x→2.x migration) that override generic Flink knowledge. Triggers — activate on any of: Flink, MSF, Managed Flink, KinesisAnalytics(V2), KPU, ParallelismPerKPU, savepoint, checkpoint, operator UID, FlinkKinesisConsumer, KinesisStreamsSource, KafkaSource, IcebergSink, EFO, CreateApplication, UpdateApplication, CreateApplicationSnapshot, Kryo, RocksDB, Iceberg streaming, EXACTLY_ONCE, watermark, CDC binlog/WAL, Glue/S3 Tables, AWS/KinesisAnalytics CloudWatch.

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides domain-specific knowledge for building, deploying, and maintaining Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). It covers service-specific constraints, including the KPU resource model, configuration requirements, and operational lifecycle management. It emphasizes the use of service-level configurations over application-level settings for checkpoints and parallelism, and provides guidance on navigating the v1/v2 API identifier split for AWS CLI and SDK operations.

What it does

Provides patterns for KPU-based resource configuration and scaling.
Outlines service-specific constraints for checkpointing and parallelism.
Details the lifecycle management for starting, stopping, and updating applications.
Offers guidance on connector integration for Kinesis, Kafka, and Iceberg.
Explains the v1/v2 identifier differences for IAM, CloudWatch, and SDK usage.
Provides troubleshooting steps for application restart loops and state management.

When to use it

When developing or migrating Apache Flink applications for Amazon Managed Service for Apache Flink.
When configuring KPU resources or troubleshooting application scaling behavior.
When setting up IAM roles, trust policies, or CloudWatch logging for MSF.
When performing version upgrades or migrating from Flink 1.x to 2.x.
When implementing connectors for Kinesis, Kafka, or Iceberg within an MSF environment.

Advantages

Ensures adherence to MSF-specific service constraints and operational patterns.
Prevents common configuration errors related to the v1/v2 API identifier split.
Provides structured workflows for application lifecycle and deployment tasks.
Offers clear guidance on avoiding anti-patterns in job graph design and resource management.

Usage examples

“How do I correctly configure checkpoints for an application running on Managed Service for Apache Flink?”: The agent will advise against hardcoding checkpoint configurations in the application code and instead direct the user to manage these settings via the MSF service-level configuration.
“I am getting a ConcurrentModificationException when updating my Flink application. How do I fix this?”: The agent will explain the need to fetch the current application version ID or use a conditional token before issuing an update command, providing the necessary CLI syntax.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/aws/developing-applications-on-managed-service-for-apache-flink/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip developing-applications-on-managed-service-for-apache-flink.zip -d .agents/skills

Documentation by AWS

Managed Service for Apache Flink

Overview

Domain expertise for Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). Covers development, KPU resource management, connectors, state management, monitoring, IaC deployment, and version migration.

Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.

General Guidance

Before starting, ensure you have a clear understanding of the user persona, use case, and requirements:

STOP: Determine the users background and use case before proceeding:

Are they new to Flink? New to Managed Service for Apache Flink?
Are they familiar with Java development?
Is the use case complex with lots of business logic? Or simple and declarative?

These will inform how to organize the project, and whether to use Flink Table API or DataStream API. In general, assume the DataStream API.

Example Workflow for New Applications

1. User asks to build a Flink application
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [dependency-management.md](references/dependency-management.md)
5. READ relevant connector guides (e.g. [kinesis-connector-guide.md](references/kinesis-connector-guide.md))
6. Generate code following the loaded guidance
7. Validate against best practices
8. READ environment-setup.md via [environment-setup.md](references/environment-setup.md)
9. Compile and test locally

Example Workflow for General Questions

1. User asks about real time delivery of data to Iceberg
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [iceberg-connector-guide.md](references/iceberg-connector-guide.md)
5. READ other reference files as needed
6. Answer question with loaded guidance

Reference Files

You MUST use this skill and its reference files to answer any question on these topics.
Do NOT answer from training knowledge or by searching general AWS documentation when the question concerns Apache Flink, Managed Service for Apache Flink, KPU sizing, Flink monitoring, deployment, migration, real-time analytics, or Iceberg/LakeHouse streaming with Flink
- You MUST load the relevant reference files below before taking other steps.
- The reference files contain MSF-specific details (thresholds, statistics, namespaces, constraints) that differ from generic Flink guidance and are required for correct responses.

Goal	Reference	When to Load
Best practices	best-practices.md	Always before writing code
Maven dependencies	dependency-management.md	New project or adding connectors
Local dev environment	environment-setup.md	Docker-based local development
MSF architecture	msf-overview.md	KPU model and service constraints
MSF constraints and patterns	msf-constraints-and-patterns.md	MSF vs self-managed Flink, service-level vs application-level configuration separation, MSF-specific resource/network/storage limits, common MSF patterns
Quotas, ENI planning, MSF vs EMR, source/sink choice	foundation-operations.md	Capacity planning, service selection, architecture design, CLI/IAM/CloudWatch identifier disambiguation
IAM execution role, trust policy, action prefix, service principal	foundation-operations.md	Writing IAM policies for MSF — covers the `kinesisanalytics:` (no v2) action prefix, `kinesisanalytics.amazonaws.com` (no v2) trust principal, and the v2/non-v2 disconnect that is the most common source of permission and AssumeRole failures
Flink 2.x migration	flink-2x-migration.md	Version upgrades, state compatibility
KPU sizing	resource-optimization.md	Right-sizing, performance diagnosis, scaling
Scaling decisions on running apps	scaling-decisions.md	In-flight scaling matrix, cost/memory impact of scale changes, autoscaling behavior, anti-patterns
Cost estimation	pricing-calculator.md	Budget planning, sizing-to-cost mapping, optimization levers
Application lifecycle ops	application-lifecycle.md	Start/stop, deploy code, rollback, snapshot lifecycle, runtime properties, delete
Restart loop diagnosis	first-fault-isolation.md	Crashing/restarting apps, finding original failure vs loop sustainers, Flink Dashboard live diagnosis
Checkpoint tuning	checkpoint-tuning.md	Checkpoint impact on KPU memory and CPU, frequency vs network bandwidth trade-offs, checkpoint duration exceeding interval, OOM/GC during checkpoints
Job graph design	job-graph-architecture.md	Performance issues, splitting jobs
Job graph anti-patterns	job-graph-anti-patterns.md	Data skew detection and mitigation, monolith job anti-pattern, high fan-out anti-pattern, removing multiple shuffles, when to split a large application
Monitoring and alarms	monitoring-and-metrics.md	CloudWatch dashboards, alarms, metrics
Logging	logging-configuration.md	Log4j2, CloudWatch Logs setup
Kinesis connectors	kinesis-connector-guide.md	Kinesis source and sink builders, polling configuration and throttling (`READER_EMPTY_RECORDS_FETCH_INTERVAL`, `SHARD_GET_RECORDS_MAX`, `ReadProvisionedThroughputExceeded`, `LimitExceededException`), legacy connector migration
Kinesis Enhanced Fan-Out (EFO)	kinesis-efo-guide.md	When to use EFO vs polling, EFO source configuration, consumer lifecycle (`JOB_MANAGED` vs `SELF_MANAGED`), parallelism vs shard count, IAM permissions, troubleshooting
Iceberg integration (write APIs, distribution modes, partitioning)	iceberg-connector-guide.md	Iceberg write APIs (append, upsert, dynamic), distribution modes (NONE/HASH/RANGE), CoW vs MoR, read patterns, partitioning, DDL. Does NOT contain catalog choice or maintenance approaches — for those, load `iceberg-tuning-and-operations.md`.
Iceberg tuning, operations, catalog choice, maintenance	iceberg-tuning-and-operations.md	Provides maintenance approaches for S3 Tables, Glue + Glue auto-compaction, and Glue + Flink embedded maintenance with JDBC lock for catalog-choice questions; small files problem and mitigations; Flink TableMaintenance API, post-commit maintenance, lock factories; IcebergSink monitoring, anti-patterns.
CDC connectors	cdc-connector-guide.md	MySQL, PostgreSQL, Oracle, SQL Server, MongoDB CDC
IaC and deployment	iac-and-deployment.md	CloudFormation, CDK, Terraform, two-phase deployment
Serialization	serialization-guide.md	POJO, Avro, Kryo guidance
State management	state-management.md	TTL, state types, migration safety

Additional Resources

GitHub Issues

Files

ATTRIBUTION.mdAdded by AutoGineer998 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer67 B
SKILL.md8.8 KiB
references/application-lifecycle.md20.7 KiB
references/best-practices.md11.6 KiB
references/cdc-connector-guide.md47.2 KiB
references/checkpoint-tuning.md6.4 KiB
references/dependency-management.md13.4 KiB
references/environment-setup.md7.0 KiB
references/first-fault-isolation.md14.4 KiB
references/flink-2x-migration.md34.1 KiB
references/foundation-operations.md12.1 KiB
references/iac-and-deployment.md17.1 KiB
references/iceberg-connector-guide.md17.1 KiB
references/iceberg-tuning-and-operations.md31.1 KiB
references/job-graph-anti-patterns.md11.8 KiB
references/job-graph-architecture.md8.9 KiB
references/kinesis-connector-guide.md9.3 KiB
references/kinesis-efo-guide.md7.3 KiB
references/logging-configuration.md9.9 KiB
references/monitoring-and-metrics.md34.3 KiB
references/msf-constraints-and-patterns.md8.8 KiB
references/msf-overview.md6.3 KiB
references/pricing-calculator.md6.2 KiB
references/resource-optimization.md9.9 KiB
references/scaling-decisions.md5.4 KiB
references/serialization-guide.md9.2 KiB
references/state-management.md4.5 KiB

references/msf-overview.md

Managed Service for Apache Flink Overview and Concepts Guide

Introduction

Amazon Managed Service for Apache Flink is a fully managed service that makes it easy to develop, deploy, and operate Apache Flink applications on AWS using Kiro IDE for development. Managed Service for Apache Flink supports Flink 1.20 and Flink 2.2. For new applications, default to Flink 2.2. For existing applications, use the user's current Flink version. This guide provides a comprehensive overview of Managed Service for Apache Flink architecture, KPU-based scaling, service-level configuration, and key differences from self-managed Flink deployments.

Managed Service for Apache Flink abstracts infrastructure complexity while providing the full power of Apache Flink for stream processing, with a clear separation between service-level configuration managed by Managed Service for Apache Flink and application-level configuration controlled by developers through Kiro IDE.

Managed Service for Apache Flink Architecture Overview

Service Architecture

Managed Service for Apache Flink abstracts away the complexity of managing Flink clusters while providing the full power of Apache Flink for stream processing. The service architecture consists of:

Control Plane:

Application lifecycle management through Managed Service for Apache Flink console and APIs
KPU-based automatic scaling and resource management
Service-level configuration management (checkpoints, savepoints, parallelism)
Integrated CloudWatch monitoring and logging
IAM-based security and access control

Data Plane:

Flink JobManager and TaskManager processes managed by Managed Service for Apache Flink
Managed Service for Apache Flink-controlled checkpointing and savepoint storage in managed S3 buckets
Optimized network and storage configuration for AWS environment
Automatic fault tolerance and recovery with service-level checkpoint management

Key Components

Flink Applications: Your stream processing logic packaged as JAR files, developed using Kiro IDE
KPU Configuration: Kinesis Processing Units providing standardized resource allocation (1 vCPU, 4GB memory per KPU)
Service-Level Configuration: Managed Service for Apache Flink-managed settings for checkpoints, savepoints, parallelism, and infrastructure
Application Configuration: User-controlled runtime parameters, business logic settings, and connector configurations
CloudWatch Integration: Automatic metrics collection, logs aggregation, and monitoring dashboards

KPU-Based Resource Model

Kinesis Processing Units (KPUs):

You do not pick instance types or manage TaskManagers directly. MSF abstracts both away — KPU is the only resource unit you configure. Selecting EC2 instance types is not a setting you can change, including via the console. Custom CPU/memory ratios per KPU also are not configurable in MSF.
Each KPU provides exactly 1 vCPU and 4 GB of memory (standardized resource allocation), plus 50 GB of running application storage.
Managed Service for Apache Flink automatically scales KPUs based on application throughput and backpressure metrics
You configure Parallelism (total task slots) and ParallelismPerKPU (slots per KPU) at the service level; MSF derives Allocated KPUs = Parallelism / ParallelismPerKPU
KPU allocation determines the maximum parallelism and resource capacity available to your application

Resource Scaling:

Automatic horizontal scaling based on real-time throughput and backpressure analysis
Service-level parallelism configuration through Managed Service for Apache Flink console overrides application defaults
Managed Service for Apache Flink manages TaskManager allocation and distribution across KPUs automatically
Pay-per-use pricing model - only pay for the KPUs your application actively uses
Scaling decisions are made by Managed Service for Apache Flink based on performance metrics, not manual configuration

Managed Service for Apache Flink Capabilities

Core Streaming Capabilities

Stream Processing APIs:

DataStream API for low-level stream processing
Table API for relational stream processing with improved performance
Flink SQL for declarative stream analytics with expanded function library
Complex Event Processing (CEP) for pattern detection (note: Flink 2.x requires explicit TypeInformation for CEP pattern output)

State Management:

Managed keyed state with Managed Service for Apache Flink-controlled automatic checkpointing
Broadcast state for configuration distribution
RocksDB state backend optimized for Managed Service for Apache Flink environment
Automatic state cleanup and TTL management
Service-level checkpoint configuration (interval, retention, storage)

Time Processing:

Event time processing with watermarks (default and only time characteristic in Flink 2.x)
Processing time for low-latency scenarios
Custom timestamp extractors and watermark strategies

AWS Service Integration

Data Sources:

Amazon Kinesis Data Streams
Amazon MSK (Managed Streaming for Apache Kafka)
Amazon S3 for batch processing
Amazon DynamoDB Streams

Data Sinks:

Apache Iceberg
Amazon S3 with various formats (Parquet, JSON, CSV)
Amazon DynamoDB for real-time updates
Amazon OpenSearch Service
Amazon Data Firehose
Custom sinks via AWS SDK

For Managed Service for Apache Flink vs self-managed Flink differences, MSF-specific constraints, and common MSF patterns, see msf-constraints-and-patterns.md.

Next Steps

After understanding Managed Service for Apache Flink architecture and capabilities:

Environment Setup: Configure your Kiro IDE development environment for Managed Service for Apache Flink development with Docker containerization
Development Patterns: Learn Managed Service for Apache Flink-optimized application patterns and templates
Local Development: Set up Docker-based local testing workflows in Kiro before Managed Service for Apache Flink deployment
Deployment: Understand Managed Service for Apache Flink deployment procedures and service-level configuration best practices

For detailed guidance on each of these areas, refer to the corresponding guides in the steering directory. All development workflows are optimized for Kiro IDE with Docker-based local development targeting Managed Service for Apache Flink deployment.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/developing-applications-on-managed-service-for-apache-flink · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download developing-applications-on-managed-service-for-apache-flink.zip