Setting Up Cloudwatch Observability

Instrument an Application with ADOT (CloudWatch Application Observability)

Instrument an application that is not yet instrumented so it emits OpenTelemetry traces and metrics, using the AWS Distro for OpenTelemetry (ADOT) auto-instrumentation SDKs. This is plain OTel instrumentation — the ADOT SDK is wired into the workload and exports over OTLP, to the SDK's default endpoint unless the Scope table below calls for a collector or an existing destination.

Before you start: confirm a Space exists in the target Region. list-spaces is account-global (the --region flag only selects the endpoint; the response lists every Space in the account, each with its own region), so filter to the Region rather than trusting a non-empty list: aws cloudwatchomni list-spaces --region <region> --query "items[?region=='<region>']". If that filtered list is empty, stop and run ../spaces-and-domains.md first — instrumentation started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see.

Never modify application source code (.py, .js, .ts, .java, .cs). Only edit infrastructure-as-code, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests. Make the minimum change needed and preserve existing configuration. Present changes for the user to review; do not run terraform apply, cdk deploy, kubectl apply, helm install/helm upgrade, kubectl annotate, kubectl delete, or aws ecs update-service automatically. That applies to the collector step too: some of it is imperative commands rather than a diff, and those still belong to the user.

Scope

The job is always instrumentation — getting the ADOT SDK loaded into the running workload and emitting telemetry. Whether it also includes a destination (a collector the SDK exports to) depends on what the user said and what already exists:

What counts as an existing destination: a collector container, sidecar, Deployment, or DaemonSet (cloudwatch-agent, aws-otel-collector, otelcol-contrib, otel-collector); a CloudWatch Agent installed in UserData or a launch template with an opentelemetry receiver in its config; or OTEL_EXPORTER_OTLP_ENDPOINT already set on the workload. A CloudWatch Agent configured only for Application Signals (port 4316) is not one — it has no OTLP receiver for this path. When you add a collector because none was found, say which files you checked, and that the collector step can be dropped if one already runs outside them (a cluster-wide collector on EKS, an agent baked into an AMI).

Pointing the SDK at the destination. The SDK falls back to its spec defaults (http://localhost:4318 for OTLP/HTTP, http://localhost:4317 for OTLP/gRPC). Which of the two applies depends on the SDK: the Java agent defaults to OTLP/gRPC on 4317, while the Python, Node.js and .NET distros default to OTLP/HTTP on 4318. A collector on the same host or in the same task that listens on that port needs no endpoint setting; set OTEL_EXPORTER_OTLP_ENDPOINT (and OTEL_EXPORTER_OTLP_PROTOCOL if it differs) only to reach one elsewhere, and only to an address the files show. Never invent an endpoint.

The collector exports straight to CloudWatch's own per-signal OTLP endpoints, with upstream otelcol-contrib as the backup. ../data-forwarding-and-centralization.md is not an alternative here — it accepts no telemetry over OTLP, it forwards what is already in CloudWatch log groups into a Dataset.

Lambda is outside this check: the ADOT layer exports to the function's own X-Ray receiver (see "IAM follows the destination" below), so a Lambda change never adds a collector.

Do NOT add these

These belong to CloudWatch Application Signals and must not appear in an Omni instrumentation change. If the user explicitly wants Application Signals instead, stop and switch to the sibling CloudWatch path in this same skill: references/cloudwatch/application-signals-onboarding.md, then the matching references/cloudwatch/appsignals-guides/<platform>-<language>.md. That is a folder switch, not a skill switch — do not hand the request to aws-observability, which holds no onboarding procedure.

• OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT

• OTEL_AWS_SERVICE_EVENTS_* (git/deployment metadata, PACKAGES_INCLUDE)

• OTEL_AWS_OTLP_LOGS_ENDPOINT, OTEL_AWS_OTLP_METRICS_ENDPOINT

• OTEL_AWS_DYNAMIC_INSTRUMENTATION_*

• The amazon-cloudwatch-observability EKS add-on, and the instrumentation.opentelemetry.io/inject-* annotation that depends on it

• Any endpoint on the CloudWatch Agent's port 4316 (a CloudWatch-specific port, not an OTel default)

One exception, on Lambda only: set OTEL_AWS_APPLICATION_SIGNALS_ENABLED=false explicitly. This is required, not stylistic — the layer's otel-instrument wrapper defaults the variable to true when it is unset, so omitting it silently ENABLES Application Signals. See the Lambda guides.

A benign ServiceEvents WARN on ADOT Java 2.30.0. Even with none of these variables set, the Java agent logs OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED=true but OTEL_AWS_SERVICE_EVENTS_PACKAGES_INCLUDE ... around four times during startup, interleaved with [SERVICE_EVENTS] INFO lines such as ServiceEventsInstrumentation initialized at startup that read as though it were enabled, and then ServiceEventsSpanProcessor not registered (serviceevents disabled). It defaults the flag internally and self-disables; nothing leaks. Expect the warning and leave it alone — do not "fix" it by setting OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED=false, which is one of the variables above.

IAM follows the destination, not the instrumentation

On EC2, ECS, and EKS, instrumentation itself needs no IAM changes — loading the SDK into a workload grants nothing, and the SDK is left pointing at its default local endpoint. Do not add CloudWatchAgentServerPolicy, AWSXRayDaemonWriteAccess, or application-signals:* permissions to those workloads.

That is not the same as saying the workload will never need permissions — it means instrumentation does not. Permissions follow whatever ends up talking to AWS:

• A destination the user already has — the permission belongs with it. Do not add a policy speculatively.

• A collector deployed via collector.md — the collector is the thing calling AWS, so it does get IAM: CloudWatchAgentServerPolicy on the instance profile (EC2), the task role (ECS), or an IRSA role (EKS). The application still gets nothing.

Lambda is different, and does get a policy here. The ADOT layer exports to the execution environment's X-Ray receiver under active tracing, so the destination is already known and lives inside the function's own configuration. The Lambda guides therefore grant the execution role xray:PutTraceSegments and xray:PutTelemetryRecords — via AWSXRayDaemonWriteAccess, or the inline grant CDK adds when tracing is enabled. CloudWatchLambdaApplicationSignalsExecutionRolePolicy is still out of scope, since Application Signals is disabled.

Step 1: Determine platform and language

Detect from the IaC and application code, and confirm with the user if ambiguous.

• EKS: k8s Deployment manifests (kind: Deployment), Helm charts, kubectl in scripts, Terraform aws_eks_*

• ECS: ECS task definitions, containerDefinitions, Terraform aws_ecs_*

• EC2: EC2 instances, userdata scripts, launch templates, Terraform aws_instance

• Language: requirements.txt/pyproject.toml/*.py → Python; package.json/*.ts/*.js → Node.js; pom.xml/build.gradle/*.java → Java; *.csproj/*.sln/*.cs → .NET

If the workload is an AI agent built on a framework (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI), stop — use ../omni-agents-instrumentation/omni-agents-instrumentation.md instead. That path instruments at the framework layer.

Step 2: Read the matching guide

When the Scope table calls for a collector, continue to collector.md — it is language-agnostic and covers EC2, ECS, and EKS. Lambda is excluded: there is nowhere to run a collector in it, and its layer already exports to the execution environment's X-Ray receiver.

How the SDK gets in

The mechanism differs by platform, but on ECS and EKS it is the same idea — an init container copies the SDK into a shared volume at startup, so the application image never changes:

For instrumentation alone, no cluster-level components are installed on EKS — no add-on and no operator; injection is written directly into the Deployment manifest. (When the Scope table calls for a collector, collector.md deploys the CloudWatch Agent by Helm, which necessarily brings its operator. That is expected there, and is not the amazon-cloudwatch-observability add-on.)

Lambda is the odd one out: the execution environment supplies an X-Ray receiver that the layer exports to, so unlike the other three platforms its export path is already wired. That is why the Lambda guides look different and why OTEL_AWS_APPLICATION_SIGNALS_ENABLED=false is set explicitly there.

Verifying without a receiver

When the change is instrumentation only and no destination exists yet, the SDK exports to its default local endpoint (localhost:4318, or localhost:4317 for Java) where nothing is listening. OTLP connection errors in the logs after instrumenting are expected and are not a failure — they mean the SDK loaded and is trying to export. Tell the user the errors will clear once a receiver exists.

Do not rely on "the SDK announces itself" as the check — it only works for Java. What a working deployment actually logs, by language:

For Python and .NET, verify by other means: the init container exiting 0, the loader variables present in the running process (printenv, or tr '\0' '\n' < /proc/1/environ — use PID 1 in a single-process container, since slim images often lack pgrep), and ultimately spans at the destination.

For .NET the strongest check is the SDK's own Managed log above. If you also check paths, expand the variable inside the container — kubectl exec <pod> -- sh -c 'test -f "$CORECLR_PROFILER_PATH"'; unwrapped, it expands in your local shell where it is unset and always reports failure. Note DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE legitimately dangle in the container image.

If a collector was deployed in the same change, these errors should be absent — and on the CloudWatch Agent, all of them. The agent builds traces, metrics and logs pipelines from its opentelemetry section, so a correctly wired app logs no OTLP errors at all — a GET probe of /v1/logs returns 405 (method not allowed on a path that IS registered), not 404. Treat any OTLP error on the agent path as a real problem.

On the otelcol-contrib backup only: the hand-written Option A config in collector.md defines only traces and metrics pipelines, while the SDKs default OTEL_LOGS_EXPORTER=otlp — so the receiver returns 404 on /v1/logs and the app logs that permanently. That is expected there and is not a connectivity failure; traces flow fine alongside it. Either add a logs pipeline or set OTEL_LOGS_EXPORTER=none. Do not set OTEL_LOGS_EXPORTER=none on the CloudWatch Agent path — it would discard a signal the agent delivers for free.

Step 3: Set the service identity

Every guide sets these two, and nothing else about export:

Derive OTEL_SERVICE_NAME from the existing workload name in the IaC (deployment name, ECS service name, or the application name) rather than inventing one. If it cannot be determined, ask.

Custom metrics: what makes a metric queryable in Omni

Auto-instrumentation produces the standard request metrics on its own. A custom metric — a latency histogram, a queue depth, a business counter — is a code change the customer makes in the application, and this reference never edits source; so when the question is "how do I get my own metric to show up in Omni", hand them the rule and the instrument to use rather than a file diff.

Omni serves metrics from a single OpenTelemetry-native, PromQL-queryable surface. A metric is on that surface when it arrives as an OTLP metric and carries the workload's OTel resource attributes — service.name above all, which Omni exposes as the label "@resource.service.name" and uses to tie the metric to the service. Two paths put a metric there, and they are the same two the rest of this file wires:

• OTel SDK in the application — create the instrument through the OTel Metrics API (Meter → Histogram for latency, Counter for counts, Gauge for levels) and record to it. The ADOT SDK the guides above load already runs a MeterProvider exporting over OTLP, so a custom instrument rides the same exporter, endpoint, and resource as the auto-instrumented telemetry — no extra configuration in the app.

• An OTel Collector the application exports OTLP to — the SDK sends to the collector, which exports to CloudWatch's OTLP metrics endpoint (https://monitoring.<region>.amazonaws.com/v1/metrics, collector.md).

Either way the exporter must reach a destination that delivers into the account's Dataset in a Region with a Space; an SDK left at its localhost default records the metric and delivers it nowhere.

On Lambda, neither path is wired. The layer's export goes to the execution environment's X-Ray receiver, which takes traces, not metrics, and there is nowhere in the function to run a collector. This reference has no verified configuration that sends a custom OTLP metric from a function to the OTLP metrics endpoint, so say that plainly: give the rule above, and do not present the layer as already exporting the metric or invent exporter variables for it.

What does not work, and why customers try it: the CloudWatch PutMetricData API and Embedded Metric Format (EMF) log lines are the familiar CloudWatch ways to publish a custom metric — and they produce a CloudWatch metric in a CloudWatch namespace, which is a different surface. CloudWatch metrics are not queryable in Omni PromQL. The one bridge between the surfaces, OTel enrichment, projects AWS-vended service metrics (EC2, Lambda, RDS, ...) onto the OTel surface and does not apply to a custom application metric, so EMF or PutMetricData on its own never makes a custom metric visible in Omni. (The IAM action the OTLP metrics endpoint authorizes against is also named cloudwatch:PutMetricData; that is the permission, not the API — a metric sent through the OTLP endpoint stays OTel-native.)

Constraints:

• You MUST NOT present PutMetricData or EMF as a way to get a custom metric into Omni. If the customer already publishes that way, the answer is to emit the metric through the OTel SDK (or a collector) as well — not to look for a setting that makes the CloudWatch metric appear.

• When the customer already publishes a custom metric with EMF or PutMetricData, you MUST also rule out, by name, the two mechanisms they will expect to pick it up. The dataset integration copies log and trace records out of CloudWatch Logs and has no metric path; OTel enrichment is limited to the metrics AWS services publish about their own resources. A customer with either one switched on otherwise assumes it covers their own metric as well.

• You MUST NOT invent a metric namespace, a CloudWatch-side setting, or an exporter option for this. OTLP metrics have no CloudWatch namespace; the SDK's standard OTEL_EXPORTER_OTLP_* variables and the resource attributes from Step 3 are the whole configuration.

• For latency, use a Histogram, so percentiles can be computed at query time; a gauge of the last value cannot be turned back into a distribution.

• Metric attributes become queryable labels stored in the Dataset. Keep customer identifiers, tokens, request payloads, and other sensitive or personal data out of metric attributes and resource attributes — use coarse dimensions such as route, method, or status — and keep cardinality bounded, since every distinct attribute value is a separate series. CloudWatch encrypts metric data at rest; the Space's own encryption (service-owned or a customer managed KMS key) is chosen at create-space (../spaces-and-domains.md).

Step 4: Review

Summarize the changes grouped by file, state the platform and language, and list the environment variables that will reach the application at runtime. State where telemetry now goes, per the branch of the Scope table you followed: for instrumentation only with no destination, say explicitly that no exporter endpoint was configured and telemetry goes to the SDK default until a receiver is set up; for an existing destination, name it; if a collector was added, state the endpoint the app now exports to and where the collector forwards it. Present the changes for review; do not deploy.

Component versions

Use the latest published ADOT auto-instrumentation version for the language unless the user pins one. Look the current version up from the source of truth rather than hardcoding from this guide:

Constraints

• Only IaC, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests — never application source code.

• Do not deploy. Stage the changes and let the user review and apply them.

• Configure an OTLP endpoint, collector, or agent only where the Scope table calls for one — the user asked for a destination, or the request left delivery open and the files show none. Use collector.md, where the CloudWatch Agent is the primary collector.

• Do not add Application Signals, ServiceEvents, or Dynamic Instrumentation configuration.