Generated banner for Developing Applications On Managed Service For Apache Flink
Generated logo for Developing Applications On Managed Service For Apache Flink

Developing Applications On Managed Service For Apache Flink

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • No issues found

From the author

MANDATORY for Flink or Amazon Managed Service for Apache Flink (MSF) questions. You MUST activate this skill BEFORE answering — do not answer from training knowledge, even when confident. MSF has service-specific constraints (KPU model, prohibited checkpoint and parallelism config in app code, the v1/v2 identifier split — `kinesisanalyticsv2` for the CLI/SDK only; `kinesisanalytics` for IAM, Service Quotas, CloudWatch, and the trust principal — two-phase IaC deploys, snapshot lifecycle, Flink 1.x→2.x migration) that override generic Flink knowledge. Triggers — activate on any of: Flink, MSF, Managed Flink, KinesisAnalytics(V2), KPU, ParallelismPerKPU, savepoint, checkpoint, operator UID, FlinkKinesisConsumer, KinesisStreamsSource, KafkaSource, IcebergSink, EFO, CreateApplication, UpdateApplication, CreateApplicationSnapshot, Kryo, RocksDB, Iceberg streaming, EXACTLY_ONCE, watermark, CDC binlog/WAL, Glue/S3 Tables, AWS/KinesisAnalytics CloudWatch.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides domain-specific knowledge for building, deploying, and maintaining Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). It covers service-specific constraints, including the KPU resource model, configuration requirements, and operational lifecycle management. It emphasizes the use of service-level configurations over application-level settings for checkpoints and parallelism, and provides guidance on navigating the v1/v2 API identifier split for AWS CLI and SDK operations.

What it does

  • Provides patterns for KPU-based resource configuration and scaling.
  • Outlines service-specific constraints for checkpointing and parallelism.
  • Details the lifecycle management for starting, stopping, and updating applications.
  • Offers guidance on connector integration for Kinesis, Kafka, and Iceberg.
  • Explains the v1/v2 identifier differences for IAM, CloudWatch, and SDK usage.
  • Provides troubleshooting steps for application restart loops and state management.

When to use it

  • When developing or migrating Apache Flink applications for Amazon Managed Service for Apache Flink.
  • When configuring KPU resources or troubleshooting application scaling behavior.
  • When setting up IAM roles, trust policies, or CloudWatch logging for MSF.
  • When performing version upgrades or migrating from Flink 1.x to 2.x.
  • When implementing connectors for Kinesis, Kafka, or Iceberg within an MSF environment.

Advantages

  • Ensures adherence to MSF-specific service constraints and operational patterns.
  • Prevents common configuration errors related to the v1/v2 API identifier split.
  • Provides structured workflows for application lifecycle and deployment tasks.
  • Offers clear guidance on avoiding anti-patterns in job graph design and resource management.

Usage examples

“How do I correctly configure checkpoints for an application running on Managed Service for Apache Flink?”
The agent will advise against hardcoding checkpoint configurations in the application code and instead direct the user to manage these settings via the MSF service-level configuration.
“I am getting a ConcurrentModificationException when updating my Flink application. How do I fix this?”
The agent will explain the need to fetch the current application version ID or use a conditional token before issuing an update command, providing the necessary CLI syntax.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/aws/developing-applications-on-managed-service-for-apache-flink/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip developing-applications-on-managed-service-for-apache-flink.zip -d .agents/skills

Documentation by AWS

Overview

Domain expertise for Apache Flink applications on Amazon Managed Service for Apache Flink (MSF). Covers development, KPU resource management, connectors, state management, monitoring, IaC deployment, and version migration.

Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.

General Guidance

Before starting, ensure you have a clear understanding of the user persona, use case, and requirements:

STOP: Determine the users background and use case before proceeding:

  • Are they new to Flink? New to Managed Service for Apache Flink?

  • Are they familiar with Java development?

  • Is the use case complex with lots of business logic? Or simple and declarative?

These will inform how to organize the project, and whether to use Flink Table API or DataStream API. In general, assume the DataStream API.

Example Workflow for New Applications
1. User asks to build a Flink application
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [dependency-management.md](references/dependency-management.md)
5. READ relevant connector guides (e.g. [kinesis-connector-guide.md](references/kinesis-connector-guide.md))
6. Generate code following the loaded guidance
7. Validate against best practices
8. READ environment-setup.md via [environment-setup.md](references/environment-setup.md)
9. Compile and test locally
Example Workflow for General Questions
1. User asks about real time delivery of data to Iceberg
2. Confirm user's goals and use case
3. READ [best-practices.md](references/best-practices.md)
4. READ [iceberg-connector-guide.md](references/iceberg-connector-guide.md)
5. READ other reference files as needed
6. Answer question with loaded guidance

Reference Files

  • You MUST use this skill and its reference files to answer any question on these topics.

  • Do NOT answer from training knowledge or by searching general AWS documentation when the question concerns Apache Flink, Managed Service for Apache Flink, KPU sizing, Flink monitoring, deployment, migration, real-time analytics, or Iceberg/LakeHouse streaming with Flink

    • You MUST load the relevant reference files below before taking other steps.

    • The reference files contain MSF-specific details (thresholds, statistics, namespaces, constraints) that differ from generic Flink guidance and are required for correct responses.

GoalReferenceWhen to Load
Best practicesbest-practices.mdAlways before writing code
Maven dependenciesdependency-management.mdNew project or adding connectors
Local dev environmentenvironment-setup.mdDocker-based local development
MSF architecturemsf-overview.mdKPU model and service constraints
MSF constraints and patternsmsf-constraints-and-patterns.mdMSF vs self-managed Flink, service-level vs application-level configuration separation, MSF-specific resource/network/storage limits, common MSF patterns
Quotas, ENI planning, MSF vs EMR, source/sink choicefoundation-operations.mdCapacity planning, service selection, architecture design, CLI/IAM/CloudWatch identifier disambiguation
IAM execution role, trust policy, action prefix, service principalfoundation-operations.mdWriting IAM policies for MSF — covers the kinesisanalytics: (no v2) action prefix, kinesisanalytics.amazonaws.com (no v2) trust principal, and the v2/non-v2 disconnect that is the most common source of permission and AssumeRole failures
Flink 2.x migrationflink-2x-migration.mdVersion upgrades, state compatibility
KPU sizingresource-optimization.mdRight-sizing, performance diagnosis, scaling
Scaling decisions on running appsscaling-decisions.mdIn-flight scaling matrix, cost/memory impact of scale changes, autoscaling behavior, anti-patterns
Cost estimationpricing-calculator.mdBudget planning, sizing-to-cost mapping, optimization levers
Application lifecycle opsapplication-lifecycle.mdStart/stop, deploy code, rollback, snapshot lifecycle, runtime properties, delete
Restart loop diagnosisfirst-fault-isolation.mdCrashing/restarting apps, finding original failure vs loop sustainers, Flink Dashboard live diagnosis
Checkpoint tuningcheckpoint-tuning.mdCheckpoint impact on KPU memory and CPU, frequency vs network bandwidth trade-offs, checkpoint duration exceeding interval, OOM/GC during checkpoints
Job graph designjob-graph-architecture.mdPerformance issues, splitting jobs
Job graph anti-patternsjob-graph-anti-patterns.mdData skew detection and mitigation, monolith job anti-pattern, high fan-out anti-pattern, removing multiple shuffles, when to split a large application
Monitoring and alarmsmonitoring-and-metrics.mdCloudWatch dashboards, alarms, metrics
Logginglogging-configuration.mdLog4j2, CloudWatch Logs setup
Kinesis connectorskinesis-connector-guide.mdKinesis source and sink builders, polling configuration and throttling (READER_EMPTY_RECORDS_FETCH_INTERVAL, SHARD_GET_RECORDS_MAX, ReadProvisionedThroughputExceeded, LimitExceededException), legacy connector migration
Kinesis Enhanced Fan-Out (EFO)kinesis-efo-guide.mdWhen to use EFO vs polling, EFO source configuration, consumer lifecycle (JOB_MANAGED vs SELF_MANAGED), parallelism vs shard count, IAM permissions, troubleshooting
Iceberg integration (write APIs, distribution modes, partitioning)iceberg-connector-guide.mdIceberg write APIs (append, upsert, dynamic), distribution modes (NONE/HASH/RANGE), CoW vs MoR, read patterns, partitioning, DDL. Does NOT contain catalog choice or maintenance approaches — for those, load iceberg-tuning-and-operations.md.
Iceberg tuning, operations, catalog choice, maintenanceiceberg-tuning-and-operations.mdProvides maintenance approaches for S3 Tables, Glue + Glue auto-compaction, and Glue + Flink embedded maintenance with JDBC lock for catalog-choice questions; small files problem and mitigations; Flink TableMaintenance API, post-commit maintenance, lock factories; IcebergSink monitoring, anti-patterns.
CDC connectorscdc-connector-guide.mdMySQL, PostgreSQL, Oracle, SQL Server, MongoDB CDC
IaC and deploymentiac-and-deployment.mdCloudFormation, CDK, Terraform, two-phase deployment
Serializationserialization-guide.mdPOJO, Avro, Kryo guidance
State managementstate-management.mdTTL, state types, migration safety

Additional Resources

Files

references/pricing-calculator.md

MSF Pricing Calculator

Overview

Estimate MSF application monthly cost for sizing decisions, budgeting, and optimization analysis. Prices, tables, and worked examples below are us-east-1 and show the billing model; they are not current rates.

Get Current Rates

Before quoting a dollar figure, fetch the rates for the application's region from the Price List API (the pricing endpoint is called in us-east-1 for every region):

aws pricing get-products --region us-east-1 --service-code AmazonKinesisAnalytics \
  --filters Type=TERM_MATCH,Field=regionCode,Value=<region> \
  --query 'PriceList[]' --output json

Use these usage types (prefixed with a region code, e.g. USE1-, EUW1-):

Usage type suffixComponent
KPU-Hour-JavaKPU-hour for Apache Flink applications, including the orchestration KPU (not KPU-Hour, which is SQL applications)
RunningApplicationStorageRunning application storage, per GB-month
DurableApplicationBackupsSnapshots, per GB-month

Apply the live rates to the formula below. If the API is unavailable, use the MSF pricing page, and say which source and region the estimate uses.

Pricing Components

ComponentPrice (us-east-1)Notes
KPU-hour$0.11 / KPU-hourPer allocated KPU per running hour
Orchestration KPU$0.11 / hour1 additional KPU per running app, always billed
Running application storage$0.10 / GB-month50 GB is allocated and billed per application KPU (not the orchestration KPU), regardless of state size: ~$5 per application KPU per month at 24/7
Durable application backups (snapshots)$0.023 / GB-monthBilled for total snapshot footprint

There is no separate charge for application code in S3 (your S3 bucket cost only) and no per-API-call charge for Flink REST API or kinesisanalyticsv2 operations.

Formula

KPU_count   = Parallelism / ParallelismPerKPU
Total_KPUs  = KPU_count + 1                                    # orchestration

KPU_cost     = Total_KPUs × $0.11 × hours_per_month              # Total_KPUs for the KPU charge only
storage_cost = KPU_count × 50 × $0.10 × hours_per_month / 730
backup_cost  = total_snapshot_GB × $0.023

monthly_total = KPU_cost + storage_cost + backup_cost

24/7 = 730 hours/month.

To check an estimate against a real bill: in Cost Explorer (service Amazon Kinesis Analytics), RunningApplicationStorage GB-Mo × 730 / 50 equals application KPU-hours, and total KPU-Hour minus that figure equals orchestration KPU-hours.

Sizing → Cost Reference (24/7, KPU + running storage)

Input RateParallelismPPKKPUs (+orch)Monthly
< 5 MB/s212+1=3$251
5–20 MB/s414+1=5$422
20–50 MB/s824+1=5$422
50–100 MB/s1628+1=9$763
100–200 MB/s32216+1=17$1,445
200–500 MB/s64416+1=17$1,445
> 500 MB/s128+432+1=33+$2,810+

Monthly = KPU charge for all KPUs + $5 running storage per application KPU.

These assume stateless-to-moderate stateful workloads. Stateful workloads (keyed windows with large state) typically require PPK=1, doubling KPU count and cost vs. the stateless equivalent.

Worked Examples

Small / development (Parallelism=2, PPK=1, 24/7, negligible state, no snapshots):

  • KPU count: 2 / 1 = 2 application KPUs, plus 1 orchestration KPU = 3 billed KPUs

  • KPU cost: 3 × $0.11 × 730 = $240.90

  • Running application storage: 2 × 50 × $0.10 = $10.00

  • Snapshot cost: $0 (none retained)

  • Total: ~$251/month

The orchestration KPU is the most common sizing mistake at this scale: it raises the bill from $171 to $251, a 47% increase, and is 32% of the total. Always include it.

Medium production (8 parallel, PPK=1, 24/7, 5 GB state):

  • KPUs: 8 + 1 = 9 → 9 × $0.11 × 730 = $722.70

  • Storage: 8 × 50 × $0.10 = $40.00

  • Snapshots: 5 GB × $0.023 = $0.12

  • Total: ~$763/month

Enterprise with autoscaling (avg 16 KPU, peak 32 KPU 20% of time, 60 GB state):

  • Peak: 33 × $0.11 × 146 = $530

  • Off-peak: 17 × $0.11 × 584 = $1,092

  • Storage: (32 × 146 + 16 × 584) × 50 / 730 × $0.10 = $96.00

  • Snapshots: 60 GB × $0.023 = $1.38

  • Total: ~$1,719/month

Cost Optimization Levers

Ranked by typical impact:

  1. Right-size PPK for workload: Stateless apps at PPK=2 cost half of PPK=1 at the same parallelism. Rule of thumb: stateless → PPK=2; stateful → PPK=1.

  2. Stop non-prod when idle: Dev/test apps run 200h/month vs 730h is a 73% reduction. Snapshot before stop, restore on start.

  3. Enable autoscaling for variable load: 30–50% average reduction for spiky workloads with predictable off-peaks.

  4. Match Parallelism to source partitions: Idle subtasks waste KPUs. Parallelism > shard/partition count creates them.

  5. Snapshot retention strategy: Without retention, a streaming app's snapshot footprint grows monotonically.

  6. Always remember the orchestration KPU: At 1–2 application KPU sizing it's ~32–48% of the bill — easy to under-estimate.

Common Mistakes

MistakeImpact
Forgetting orchestration KPUUnder-estimates by 1 KPU = $80/month at 24/7
Treating running storage as freeUnder-estimates by $5 per application KPU per month (~6% of KPU cost)
Confusing Parallelism with KPU countWrong with PPK > 1; KPUs = Parallelism / PPK
Quoting the us-east-1 figures above as current or for another regionRates vary by region and change over time; fetch them as in Get Current Rates
Estimating peak as averageAutoscaling jobs have asymmetric peak/off-peak — calculate separately
Ignoring CloudWatch metric cardinality costOPERATOR/PARALLELISM metric levels add custom-metric cost (not MSF cost). See monitoring-and-metrics.md

References

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/developing-applications-on-managed-service-for-apache-flink · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download developing-applications-on-managed-service-for-apache-flink.zip