Temporal Ops
OpenAIOfficialCreated by TemporalDistributed by OpenAI
- Skill
- Tool integration
- Cloud and DevOps
- MIT
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured approach to managing and troubleshooting Temporal environments. It covers two primary modes: Operations, for administrative tasks like namespace management, API key rotation, and batch workflow actions; and Diagnosis, for investigating issues like stuck workflows, connectivity errors, and performance bottlenecks. It enforces strict operational discipline, including explicit confirmation for destructive commands and a bottom-up diagnostic methodology to ensure root causes are identified accurately.
What it does
- Performs administrative operations on Temporal Cloud via tcld and self-hosted clusters via temporal operator.
- Executes batch operations on workflows using query-based filtering with safety-first confirmation steps.
- Diagnoses system issues using a layered, bottom-up approach from network connectivity to workflow logic.
- Manages security configurations including API keys, mTLS certificates, and SAML/SCIM integrations.
- Provides guidance on interpreting audit logs and cloud capacity metrics.
- Routes complex tasks to specialized skills for development, worker tuning, or infrastructure configuration.
When to use it
- When you need to perform administrative tasks like creating namespaces or rotating credentials.
- When a Temporal environment is experiencing errors such as connection timeouts or authentication failures.
- When you need to investigate or recover from stuck, non-deterministic, or unhealthy workflows.
- When you need to perform bulk operations on workflows and require a safe, verified execution process.
- When you need to audit or monitor the health and capacity of your Temporal Cloud account.
Advantages
- Enforces safe operational practices by requiring explicit approval for destructive commands.
- Provides a systematic diagnostic ladder to avoid speculative fixes.
- Distinguishes clearly between Cloud and self-hosted administrative procedures.
- Integrates safety checks like counting affected workflows before executing batch mutations.
- Standardizes the interpretation of common gRPC and Temporal-specific error codes.
Usage examples
- “How do I rotate my mTLS certificates for my Temporal Cloud namespace?”
- The agent will guide you through the tcld commands for generating new certificates and updating the namespace configuration, ensuring you follow the correct rotation sequence.
- “I am seeing 'context deadline exceeded' errors in my workflow. How do I debug this?”
- The agent will initiate a bottom-up diagnosis, starting by verifying network connectivity and gRPC health before investigating specific workflow task execution logs.
- “Terminate all workflows of type 'OrderProcessing' that are currently stuck.”
- The agent will first run a count to identify the number of affected workflows, present this number to you for confirmation, and then execute the batch termination only after your approval.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/openai/temporal-ops/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip temporal-ops.zip -d .agents/skills
Documentation by Temporal
Skill: temporal-ops
Overview
This skill operates and diagnoses Temporal environments. It has two modes:
Operations: the user wants to do something — create a namespace, rotate a key, check capacity, find unhealthy workflows, cancel a batch, set up export. The skill executes the right commands and interprets the output.
Diagnosis: the user arrives with a symptom — a stuck workflow, a cert error, a connection timeout, a non-determinism panic. The skill routes the investigation through a layered, bottom-up diagnosis until a root cause is identified with a confidence score.
It does not teach how to write workflows or activities (use skill-temporal-developer for that), and it does not reproduce exhaustive CLI flag tables — run temporal <cmd> --help for those, and see cli-conventions.md for cross-command CLI conventions. The boundary is: if the user needs to administer or troubleshoot a running Temporal environment, this skill applies.
Out of scope
Writing workflows, activities, or SDK code →
skill-temporal-developer.Exhaustive CLI flags / command reference → run
temporal <cmd> --help; cross-command CLI conventions → cli-conventions.md.Worker performance tuning, sizing, capacity planning →
skill-temporal-workertuning.Helm, Kubernetes, database admin, monitoring stack config for self-hosted — beyond the CLI surface.
If the conversation drifts into one of these areas, hand off to the relevant sibling skill rather than improvising.
Philosophy
Operator discipline
When the user wants to perform an operational task:
Identify the intent and backend. Is this a Cloud operation (
tcld) or a self-hosted operation (temporal operator)? Data-plane operations (temporal workflow,temporal batch, etc.) work on both. If the backend is ambiguous, ask before proceeding — do not assume Cloud or self-hosted and do not output environment-specific commands until you know.Execute commands and interpret output. Run the documented command, read the result, and report what it means — or act on it if the user asked for an action. Read-only commands (
get,list,describe,count,show) run freely. Anything listed under Destructive operations is proposed to the user first.Verify the result. After a mutating operation, confirm the new state matches the user's intent.
Destructive operations
An operation belongs to this tier if it is irreversible (tcld namespace delete), revokes access for a live identity (tcld apikey delete), moves
production traffic (tcld namespace failover), or fans out to every match
(any --query form). Apply the test to the operation in front of you — this is a
rule, not a list, and a command's absence from any list in this skill does not
place it outside the tier.
For anything in the tier: gather the evidence and propose. Do not run it on your own initiative, and do not run one to find out what it would do. The reference file for each command states its specific blast radius; read that before proposing, not after.
Blast radius as a number, not a description. For any
--queryform, runtemporal workflow count --query '<query>'with the byte-identical query first and carry the result into the proposal. A filter with no narrowing predicate beyondExecutionStatus="Running"matches every open Execution in the Namespace.Name the target. State the exact command, the target, and the Namespace it resolves to. Connection settings can come from
TEMPORAL_*env vars or a config-file profile, so the target is frequently not visible in the command text. If the backend or Namespace was inferred from context rather than stated by the user, say so — a destructive command aimed at the wrong Namespace is the most common way this goes wrong.Ask explicitly, then run it so it completes. Put the command, the target, and — for any
--queryform — the count from step 1 to the user as a direct question, and wait for an answer. Once they approve, run it with--yeson the--querybatch forms; that flag is what lets an approved batch finish, since the interactive prompt needs a terminal and without one the command reportsuser denied confirmationand does nothing.--yesbelongs in a command the user approved, never in a retry of one that failed its prompt. Do not substitute a loop over single-targetworkflow terminate --workflow-id. Approval covers one command against one target; it does not carry to the next command, a widened query, or a second Namespace.Verify, and know the abort path. Re-run the corresponding
get,describe, orcount. A batch job drains asynchronously:temporal batch describe --job-id <id>shows how far it has gotten andtemporal batch terminate --job-id <id>stops it before it reaches the rest of its matches.
When a reversible sibling reaches the same goal, propose it alongside: cancel
lets Workflow cleanup code run where terminate does not; apikey disable is
reversible where delete is not; accepted-client-ca add appends where set
replaces.
Assume nothing in the environment will stop a destructive command on your behalf. Credential scope, command denylists, and confirmation prompts may or may not be configured, and their possible presence is not a reason to skip any step above — you are the safeguard the user is relying on.
Diagnostic discipline
When the user arrives with a failure or anomaly:
Bottom-up diagnosis. Verify the lower layer before blaming the upper one. The layers, from bottom to top:
DNS / network path
TCP / port reachability
TLS handshake
Authentication (API key or mTLS client cert)
gRPC health and Temporal frontend reachability
Temporal namespace, task queues, workers
Workflow code (determinism, signals, timers, child workflows)
The full ladder lives in diagnostic-ladder.md.
Always verify the next layer up rather than prescribing a speculative fix. If TLS works, prove auth works before blaming the workflow. If pollers are present, prove the workflow's last event before blaming the worker.
Attach a confidence score (1-10) to every proposed diagnosis:
9-10: symptoms, operation, and confirming signals line up cleanly.
6-8: evidence is good but at least one alternative remains plausible.
1-5: the issue is still ambiguous; the "fix" is the next discriminating check, not a root cause.
Name ambiguity explicitly. Errors like
context deadline exceededare not self-describing, and a single code such asRESOURCE_EXHAUSTEDcan mean more than one condition (account-limit throttling vs. per-Workflow lock contention). Surface that, gather more context, and scope the next step narrowly.
These are skill conventions, not docs-derived facts.
Intent routing
Operations
Find the row that matches the user's intent. The reference file contains the commands and procedures.
| Intent | Category | Reference |
|---|---|---|
| Create, get, list, delete a Cloud namespace | Cloud namespace admin | cloud-namespace-admin.md |
| Add/remove region, failover, HA config | Cloud namespace admin | cloud-namespace-admin.md |
| Set retention, tags, codec-server, connectivity rules | Cloud namespace admin | cloud-namespace-admin.md |
| Add or rename Cloud search attributes | Cloud namespace admin | cloud-namespace-admin.md |
| Check current APS / capacity mode | Cloud capacity | cloud-capacity.md |
| Switch On-Demand ↔ Provisioned, set TRUs | Cloud capacity | cloud-capacity.md |
| Understand APS / RPS / OPS limits and throttling | Cloud capacity | cloud-capacity.md |
| Create, disable, enable, delete an API key | Cloud IAM | cloud-iam.md |
| Invite, list, remove users; set roles/permissions | Cloud IAM | cloud-iam.md |
| Manage user groups and service accounts | Cloud IAM | cloud-iam.md |
| Generate mTLS certs, upload CA, set cert filters | Cloud certs | cloud-certs.md |
| Rotate mTLS certificates | Cloud certs | cloud-certs.md |
| Set up Workflow History Export (S3 / GCS) | Cloud namespace admin | cloud-namespace-admin.md |
| Set up PrivateLink / PSC, manage connectivity rules | Cloud connectivity | cloud-connectivity.md |
| Self-hosted cluster health, describe, namespace CRUD | Self-hosted admin | self-hosted-admin.md |
| Self-hosted search attributes, Nexus endpoints | Self-hosted admin | self-hosted-admin.md |
| Check or manage a Cloud Nexus Endpoint's caller-Namespace allowlist; the 1,000-caller default | Cloud namespace admin | cloud-namespace-admin.md#tcld-nexus-endpoint-allowed-namespace |
| Find stuck/hung/unhealthy workflows via list queries | Workflow health | workflow-health.md |
| Task queue poller status, workflow counts | Workflow health | workflow-health.md |
| Cancel, terminate, or reset workflows | Workflow recovery | workflow-stuck.md#recovery-commands |
Bulk / batch operations on workflows (--query) | CLI conventions | cli-conventions.md |
| Schedule CRUD, time-spec, and operations | CLI conventions | cli-conventions.md |
| Complete or fail an activity externally | CLI conventions | cli-conventions.md |
| Cloud Ops API access, rate limits, Go SDK | Cloud Ops API | cloud-ops-api.md |
| View billing, generate billing report, cost attribution | Cloud billing | cloud-billing.md |
| Audit Logs: view, query via API, configure sink (AWS/GCP) | Cloud audit logs | cloud-audit-logs.md |
| Terraform provider: Namespace/User/SA/API Key/Nexus CRUD | Cloud Terraform | cloud-terraform.md |
| Expiry alerts (cert, API key, credit), status page | Cloud notifications | cloud-notifications.md |
| SAML SSO, SCIM provisioning, IdP integration | Cloud SAML/SCIM | cloud-saml-scim.md |
| Migrate self-hosted to Cloud (automated or manual), migrate between Cloud regions | Cloud migration | cloud-migration.md |
| End-to-end ops playbook (setup, rotation, audit, billing, Terraform) | Ops recipes | ops/recipes.md |
Diagnosis
Find the row that matches the user's symptom. Start the investigation at the first check, then read the linked reference.
| Symptom | Category | First check | Reference |
|---|---|---|---|
connection refused, cannot reach frontend | Connectivity | nc -zvw10 <host> 7233 | connectivity.md#connection-refused |
no such host, DNS resolution fails | Connectivity | dig +short <host> or nslookup <host> | connectivity.md#dns |
tls: handshake failure, server rejects handshake | Certificates | openssl s_client -connect <host>:7233 -servername <host> </dev/null | certificates.md#handshake-failure |
x509: certificate has expired or not yet valid | Certificates | openssl x509 -enddate -noout -in cert.pem | certificates.md#expired-or-not-yet-valid |
x509: certificate signed by unknown authority | Certificates | openssl verify -CAfile ca.pem client.pem | certificates.md#unknown-authority |
tcld session / auth fails, Cloud role unclear | Authentication | tcld account get | authentication.md#cloud-role-and-permission-model |
UNAUTHENTICATED, API key rejected | Authentication | env | grep -i TEMPORAL_API_KEY, then tcld apikey get --id <apikey_id> | authentication.md#things-to-check-when-unauthenticated-is-returned-with-an-api-key |
namespace not found / wrong namespace string with an API key | Authentication | Confirm Regional Endpoint form <region>.<cloud_provider>.api.temporal.io:7233 | authentication.md#address-form-for-api-key-connections |
RESOURCE_EXHAUSTED gRPC status | Rate limits | Identify which limit fired: throttle metrics on Cloud v1, the resource_exhausted_cause label on v0 / self-hosted | rate-limits.md#identifying-which-limit-was-hit |
| Task queue shows no pollers | Worker health | temporal task-queue describe --task-queue <q> | worker-health.md#what-no-pollers-looks-like |
| Workflow stuck on a pending activity / timer / child / signal | Workflow stuck | temporal workflow describe --workflow-id <id> | workflow-stuck.md#the-primary-inspection-command-temporal-workflow-describe |
NondeterminismError, repeating WorkflowTaskFailed | Non-determinism | Identify the last WorkflowTaskFailed cause in the Event History | non-determinism.md#the-wft-failure-signature-of-non-determinism |
| Replay fails locally but prod workflow was running | Non-determinism | Fetch the history and run the SDK replayer in one test | replay.md#step-2--run-the-sdk-replayer-all-supported-sdks |
| HA failover did not route traffic to failover region | HA failover | tcld namespace get --namespace <ns>.<acct> vs. DNS CNAME | ha-failover.md#start-here-establish-ground-truth |
| Serverless Worker (AWS Lambda) stopped processing work after a Namespace failover | HA failover | Confirm a FailoverNamespace audit event, then compare the new active region against the Lambda ARN on the Worker Deployment Version | ha-failover.md#symptom-serverless-workers-kept-running-in-the-old-region-after-failover |
context deadline exceeded (unknown layer) | Runtime errors | Identify which operation and SDK emitted it | runtime-errors.md#deadline-exceeded |
Workflow is busy / ResourceExhausted on signal/update/query to one Workflow (BusyWorkflow) | Runtime errors | Rule out account-limit throttling, then split temporal_cloud_v1_resource_exhausted_error_count by operation | runtime-errors.md#workflow-lock-contention-busyworkflow |
PAYLOADS_TOO_LARGE, exceeds size limit, payload/gRPC blob size error | Blob size limits | Check whether the issue is payload (2 MB) or gRPC message (4 MB) | blob-size-limits.md |
| Workflow stuck in invisible retry loop (gRPC message too large) | Blob size limits | Check Worker logs for ResourceExhausted, reduce batch size | blob-size-limits.md |
| High schedule-to-start latency, task slot depletion, slow Workflow Tasks | Performance bottlenecks | Check temporal_workflow_task_schedule_to_start_latency P95 | performance-bottlenecks.md |
| High replay latency, cache evictions, deadlock detected | Performance bottlenecks | Check workflow_task_replay_latency and sticky cache metrics | performance-bottlenecks.md |
| Schedule did not fire, missed catchup window | Missed Schedule Actions | Alert on temporal_cloud_v1_schedule_missed_catchup_window_count | schedule-missed.md |
If a symptom does not map to a row, start at diagnostic-ladder.md and work up from whichever layer was last known healthy.
The process
Operations path
Step 1: Identify intent and backend
Determine what the user wants to do and whether it targets:
Temporal Cloud → use
tcldcommands (requirestcld login)Self-hosted cluster → use
temporal operatorcommandsData plane (either backend) → use
temporal workflow,temporal batch,temporal schedule, etc.
If the backend is unambiguous from context — .tmprl.cloud address, tcld command, Cloud namespace format ns.account → Cloud; Kubernetes/Helm, docker-compose, self-hosted config files → self-hosted — proceed without asking. Otherwise, stop and ask: "Are you on Temporal Cloud or self-hosted?" before outputting any environment-specific commands. Do not default to either environment. Once known, save the answer to memory so you don't ask again in future conversations.
Step 2: Execute and interpret
Look up the intent in the Operations table above. Read the linked reference file for the exact commands, flags, and expected output. Run the command and interpret the result for the user.
Step 3: Verify
After a mutating operation (create, update, delete, rotate), confirm the new state:
Re-run the corresponding
getordescribecommandConfirm the output matches the user's intent
Report the result
Diagnosis path
Step 1: Identify the symptom
Ask the user for the exact, copy-pasted error text. Do not accept paraphrases — the exact string often encodes the layer (e.g., x509: prefix means TLS/cert layer, RESOURCE_EXHAUSTED: prefix means gRPC rate limit, NondeterminismError means workflow replay layer). Note that RESOURCE_EXHAUSTED alone does not tell you which condition fired — account-limit throttling and per-Workflow lock contention (Workflow is busy, i.e. BusyWorkflow) share the code. Split the resource-exhausted metric by its label (operation on the Cloud v1 family, resource_exhausted_cause on v0 and self-hosted) rather than parsing the free-text message.
Confirm three things before continuing:
What command was run, or what SDK call produced the error?
What environment produced it (local dev server, self-hosted cluster, Temporal Cloud)? If clear from context (addresses, commands, namespace format), don't ask — but if uncertain, ask now before proceeding with any diagnosis. Save the answer to memory for future conversations.
What changed recently (new deploy, new certs, new namespace, new region)?
Step 2: Gather context
The context the investigation needs depends on the category. At minimum:
For any Cloud auth / connectivity issue: auth method (API key vs mTLS), exact address, exact namespace, SDK + version. The endpoint family differs by auth method — see connectivity.md#endpoint-formats. For private connectivity (PrivateLink / PSC), TLS server name overrides also vary by auth method — see cloud-connectivity.md.
For a stuck workflow: namespace, workflow ID, run ID, and the output of
temporal workflow describe --workflow-id <id>(pending-operation state lives here, not in the Event History alone). Event History viatemporal workflow showis the companion view.For a worker health issue: worker logs (registration errors, auth errors, panics), the output of
temporal task-queue describe --task-queue <q>, andtemporal worker describe --task-queue <q>for per-worker details.For a non-determinism error: the worker log line containing the error, the workflow type name, and access to the history JSON for replay.
Step 3: Validate pasted SDK config (if any)
If the user has pasted SDK connection code — even just the address/namespace/auth fields — review it against sdk-snippet-review.md before descending the ladder. Wrong endpoint family, short namespace, or mismatched auth method will make every network-layer probe below look broken when nothing lower actually is.
Skip this step when the user has an established, previously-working config and the symptom is new — the snippet is not the culprit, the environment changed. Otherwise treat snippet validation as Layer 0.
Step 4: Descend the ladder
Use diagnostic-ladder.md to pick the right starting layer. As a rule of thumb:
Auth / connectivity / cert symptom → start at layer 1 (DNS) and walk up.
Worker / task-queue symptom → start at layer 6 (namespace + pollers).
Stuck workflow / determinism symptom → start at layer 7 (workflow code), but confirm layer 6 (worker is actually polling) first.
Each layer has a command that proves it healthy and a failure signature that tells you whether the problem lives at that layer or higher.
Step 5: Fix and verify
Prescribe the fix scoped to the root cause. Then verify by re-running the layer's healthy-check command and, if possible, the original user operation. Attach the confidence score to the diagnosis.
If the layer above the fix is still failing, return to step 4 and continue walking upward — the first broken layer is rarely the only one.
Prerequisites
Temporal CLI (
temporal) — required for data-plane operations and self-hosted admin. Install:brew install temporalor see Temporal CLI docs.tcld — required for Cloud operations. Install:
brew install temporal-cloud-clior see tcld docs. Authenticate withtcld loginbefore use.
Reference files
Operations
cloud-namespace-admin.md — Cloud namespace lifecycle via
tcld: create, get, list, delete, failover, add-region, retention, tags, codec-server, HA config, connectivity rules, search attributes, accepted-client-ca, certificate filters, export, and thetcld nexus endpoint allowed-namespacecaller allowlist (1,000-caller Access Policy ceiling).cloud-capacity.md — Capacity modes (On-Demand / Provisioned), APS/RPS/OPS definitions, TRUs,
tcld namespace capacity update, default limits, throttling, APS management best practices.cloud-iam.md — API key lifecycle (
tcld apikey), users (tcld user), user groups (tcld user-group), service accounts, account operations (tcld account), roles, namespace permissions.cloud-certs.md — mTLS cert management: generating certs with
tcld generate-certificates, uploading CAs, certificate filters, cert rotation, switching mTLS ↔ API keys.cloud-connectivity.md — Private connectivity (AWS PrivateLink / GCP PSC), connectivity rules: setup, rule parameters, tcld commands, attaching rules to namespaces.
cloud-migration.md — Migration paths: automated self-hosted→Cloud (S2S proxy,
tcld migrationcommands, 5 phases), manual self-hosted→Cloud (client changes, workflow strategies), within-Cloud region-to-region (HA add-region/failover).cloud-ops-api.md — Cloud Ops API: HTTP and gRPC endpoints (
saas-api.tmprl.cloud), Go SDK, protobuf compilation, rate limits (160 RPS account, 40 user, 80 SA, 10 concurrent async), API version header, use cases.cloud-billing.md — Cloud billing: Billing Center (invoices, credits, plans, cost by namespace), Usage Dashboards, Billing API: async CSV report generation, FOCUS-friendly format, 27-column report schema, date range constraints.
cloud-audit-logs.md — Cloud Audit Logs: supported control plane events (Account, API Keys, Connectivity Rules, Namespace, Export, Nexus, Service Accounts, User, User Groups), JSON format, API access (30-day retention), AWS Kinesis and GCP Pub/Sub sink configuration.
cloud-terraform.md — Terraform provider: setup (
TEMPORAL_CLOUD_API_KEY),temporalcloud_namespace/temporalcloud_user/temporalcloud_service_account/temporalcloud_apikey/temporalcloud_nexus_endpointCRUD, import, data sources (regions, namespaces), limitations (API keys not importable, cannot manage Account Owner).cloud-notifications.md — Cloud notifications: certificate expiry (15/10/5 days), API key expiry (30/20/10 days), credit consumption/expiry alerts, plan changes, failover events, recipient roles,
[email protected]sender.cloud-saml-scim.md — SAML SSO (Entra ID, Okta): entity identifier (
urn:auth0:prod-tmprl:ACCOUNT_ID-saml), callback URL (login.tmprl.cloud), IdP configuration steps, support ticket workflow. SCIM: supported vendors, prerequisites (SAML first), 10-minute sync window, group-to-role mapping.self-hosted-admin.md — Self-hosted control plane via
temporal operator: cluster health/describe, namespace CRUD, search-attribute create/list/remove, Nexus endpoint CRUD.workflow-health.md — Data-plane health queries:
temporal workflow listwith List Filters,temporal workflow describe/show/count,temporal task-queue describefor poller status.cli-conventions.md — Cross-command
temporalCLI conventions: connection/identity (TEMPORAL_*env vars ↔--address/--namespace/--api-key,--identity), output/formatting (--output,--time-format, payload shorthand), the--query⇒ batch-job bridge (withtemporal batch describe/list/terminate), and schedule time-spec forms. Ends with an operation→command index that routes each data-plane operation to its owner file. Delegates exhaustive flags totemporal <cmd> --help.ops/recipes.md — End-to-end ops playbooks: set up new namespace, check APS, switch capacity mode, find hung workflows, rotate API key, audit access, rotate mTLS certs, check self-hosted health, view billing / generate billing report, configure audit log sink, provision resources with Terraform, set up SAML SSO.
Diagnosis
sdk-snippet-review.md — Layer-0 config check for pasted SDK connection snippets: endpoint form per auth method, namespace format, auth / TLS expectations,
TEMPORAL_*env vars, common misconfigurations. Run before the diagnostic ladder.diagnostic-ladder.md — the seven-layer bottom-up model, with one canonical command per layer and cross-links into the topical leaves.
connectivity.md — DNS, TCP, endpoint families (Namespace Endpoint for mTLS vs. Regional Endpoint for API keys), firewall/proxy shapes, PrivateLink/PSC, quick diagnostic scripts.
certificates.md — x509 and TLS alert strings, expiry / unknown-authority / hostname-mismatch / key-mismatch diagnosis, Cloud accepted-client-CA set via
tcld namespace accepted-client-ca, Cloud mTLS certificate requirements, rotation and expiry notifications, openssl recipes.authentication.md —
UNAUTHENTICATEDvsPERMISSION_DENIED, API-key lifecycle (tcld apikeycommands, env var propagation, required Regional Endpoint form), mTLS after TLS (certificate filters, identity-to-role mapping), Cloud account-level roles and namespace-level permissions.workflow-stuck.md — Workflow Execution Status values,
temporal workflow describeas the primary inspection command, Event History viatemporal workflow show, pending activities / child workflows / signals / Nexus operations / Workflow Tasks, WorkflowTaskFailed retry loops, recovery commands (signal, terminate, cancel, reset, pause/unpause).non-determinism.md — determinism definition, WFT-failure signature, ND-inducing code patterns, per-SDK error shapes, identifying ND from Event History, local replay reproduction, remediation via Worker Versioning / patching / reset.
worker-health.md — no-pollers runbook via
temporal task-queue describe, reachability and versioning, worker-level describe, schedule-to-start latency, worker task slots, sticky execution and sticky cache, worker heartbeating, Cloud namespace-level poller limits, worker log signatures.rate-limits.md — what
RESOURCE_EXHAUSTEDmeans (and does not), Cloud APS / RPS / OPS under On-Demand and Provisioned capacity modes, self-hostedfrontend.rps/frontend.namespaceRPSdynamic config, identifying which limit fired via the throttle metrics (Cloud v1) or theresource_exhausted_causelabel (v0 / self-hosted), and separating account-limit throttling from single-resource exhaustion.ha-failover.md — Cloud HA routing via the Namespace Endpoint CNAME, verifying the active region (control-plane
tcld namespace getvs. DNS view), clients that did not follow the failover, PrivateLink after failover, failover-not-executing, handover-window errors, platform limits, RPO/RTO semantics, and Serverless Workers (AWS Lambda) not following a failover because compute-provider configuration is region-scoped.runtime-errors.md — deadline-exceeded disambiguated by operation and by where the call was made, Workflow lock contention (BusyWorkflow) separated from account-limit throttling and confirmed via the
operationbreakdown, routing forno pollers/INVALID_ARGUMENT/ unspecifiedUNAVAILABLE.replay.md — fetching Event History with the SDK client (CLI export as fallback), running the SDK replayer in every supported SDK (Go, Python, TypeScript, Java, .NET, Ruby, PHP),
TEMPORAL_DEBUGand the deadlock detector, interpreting divergent and successful replays, and the TypeScript-only VS Code extension.blob-size-limits.md — Payload size limit (2 MB) and gRPC message size limit (4 MB): error messages, per-SDK behavior (Python 1.23.0+ vs. others), claim check pattern, External Storage (Pre-release), batch-size reduction.
performance-bottlenecks.md — Latency and throughput diagnosis via SDK metrics: schedule-to-start latency, workflow task execution latency, replay latency, activity execution latency, task slot depletion, network request metrics, sticky cache metrics.
schedule-missed.md — Missed Schedule Actions: alerting via
temporal_cloud_v1_schedule_missed_catchup_window_count/schedule_missed_catchup_window, investigation viatemporal schedule list+temporal schedule describe, DescribeSchedule fields (missedCatchupWindow,overlapSkipped,bufferDropped), default catchup window (one year), root causes, overlap policies (6 values), backfill remediation.recipes.md — four end-to-end triage walkthroughs: stuck workflow at 3am, cert expired with workers offline, task-queue backlog mystery, non-determinism caught in prod.
Feedback
Feedback prompt
When you have first loaded this skill, output this message to the user:
"Thank you for trying out the public preview of the Temporal Ops skill! We would love to hear your feedback - positive or negative - over in the Community Slack, in the #topic-ai channel."
Do not output this message multiple times in the same conversation.
Reporting Issues in This Skill
If you (the AI) find this skill's explanations are unclear, misleading, or missing important information, draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-ops/issues/new. Do not file the issue autonomously.
Files
- ATTRIBUTION.mdAdded by AutoGineer888 B
- LICENSELicence file added by AutoGineer1.1 KiB
- SKILL.md33.7 KiB
- agents/openai.yaml177 B
- references/ops/cli-conventions.md10.0 KiB
- references/ops/cloud-audit-logs.md7.7 KiB
- references/ops/cloud-billing.md9.7 KiB
- references/ops/cloud-capacity.md10.1 KiB
- references/ops/cloud-certs.md11.3 KiB
- references/ops/cloud-connectivity.md9.0 KiB
- references/ops/cloud-iam.md15.4 KiB
- references/ops/cloud-migration.md11.0 KiB
- references/ops/cloud-namespace-admin.md20.5 KiB
- references/ops/cloud-notifications.md2.3 KiB
- references/ops/cloud-ops-api.md4.8 KiB
- references/ops/cloud-saml-scim.md8.6 KiB
- references/ops/cloud-terraform.md8.2 KiB
- references/ops/recipes.md19.1 KiB
- references/ops/self-hosted-admin.md13.9 KiB
- references/ops/workflow-health.md12.8 KiB
- references/triage/authentication.md19.3 KiB
- references/triage/blob-size-limits.md6.3 KiB
- references/triage/certificates.md32.9 KiB
- references/triage/connectivity.md14.0 KiB
- references/triage/diagnostic-ladder.md18.5 KiB
- references/triage/ha-failover.md17.8 KiB
- references/triage/non-determinism.md20.7 KiB
- references/triage/performance-bottlenecks.md8.9 KiB
- references/triage/rate-limits.md16.9 KiB
- references/triage/recipes.md21.1 KiB
- references/triage/replay.md13.5 KiB
- references/triage/runtime-errors.md16.6 KiB
- references/triage/schedule-missed.md5.8 KiB
- references/triage/sdk-snippet-review.md11.9 KiB
- references/triage/worker-health.md22.4 KiB
- references/triage/workflow-stuck.md30.7 KiB
references/triage/certificates.md
Certificates
Diagnose TLS and x509 failures against Temporal Cloud or a self-hosted frontend. This file covers layer 3 of the diagnostic ladder.
Prerequisite: before reading this file, rule out layers 1 and 2 (DNS, TCP). TLS errors can masquerade as connectivity errors when a middlebox drops packets mid-handshake, so confirm TCP reachability first via connectivity.md. The reverse also happens: when a TLS handshake completes but the cert is invalid, the Go client emits cert-shaped errors that are firmly in layer 3.
Ports referenced in this file: SDK and CLI gRPC traffic to a Namespace (Cloud or self-hosted frontend) uses 7233, for both mTLS and API-key Namespaces. The web UI uses 443. All examples below assume 7233 unless stated.
Flexible Auth (mTLS + API key on the same Namespace): When a Namespace has both auth methods enabled concurrently, a TLS-layer failure on the mTLS leg can be masked by a successful API-key path (and vice versa). Before triaging, confirm which auth method the failing client is using. If the same client is configured for both, disable one at a time to isolate which leg is broken. The recipes in this file apply to the mTLS leg only; API-key auth failures live in authentication.md.
Out of scope here:
DNS / TCP / endpoint / firewall → connectivity.md (layers 1-2)
gRPC
UNAUTHENTICATEDafter a successful TLS handshake → authentication.md (layer 4)API-key auth semantics (API-key connections still ride TLS, so TLS-level issues in this file apply — but the API-key authorization check itself is not a TLS issue) → authentication.md
gRPC
RESOURCE_EXHAUSTED→ rate-limits.md
Table of Contents
Error-string origin cheat sheet
Handshake failure
Expired or not-yet-valid
Unknown authority
Hostname mismatch
Server name override
Key does not match cert
Accepted client CA set (mTLS Cloud)
Certificate filters
Certificate requirements (Cloud mTLS)
Rotation and expiry notifications
Private key handling
openssl recipes
TLS / cert error reference
Error-string origin cheat sheet
Strings in this file come from the Go standard library (client side) or the TLS peer (alert descriptions). Every quoted string below is tagged with its source so you can tell a Go-local complaint from a peer-originated alert.
x509: ...— emitted by Go'scrypto/x509when the client rejects a peer cert locally. Source:src/crypto/x509/verify.go.tls: ...— emitted by Go'scrypto/tlsduring the handshake (typically a local protocol-level error). Source:src/crypto/tls/.remote error: tls: <alert description>— a TLS alert received from the peer, formatted by Go. The alert description words (handshake failure,bad certificate,unknown certificate authority,expired certificate,internal error, etc.) come fromsrc/crypto/tls/alert.go; theremote error: tls:prefix is added by Go when it surfaces the peer's alert.Failed reaching server: last connection error— surfaced by the Temporal Cloud connection path; the troubleshooting guide identifies an expired TLS certificate as a common root cause.
If you have an error string that doesn't start with x509:, tls:, or remote error: tls:, it is probably not a TLS-layer error — re-check the layer above (connectivity.md) or below (authentication.md).
Handshake failure
Symptom shape:
Client side:
tls: handshake failureor a gRPCUNAVAILABLEwhose wrapped cause starts withtls:orremote error: tls:.Peer-alerted:
remote error: tls: handshake failure(the wordhandshake failureis the TLS alert description emitted by the other side).
What it means: the two sides did not agree on protocol version, cipher, or client authentication. On its own, handshake failure is not specific — it is the generic TLS alert when the peer cannot continue. Use openssl s_client (see openssl recipes) to get a more specific line such as certificate has expired, unknown certificate authority, or a hostname-mismatch error.
First check: reproduce the handshake from the same host that saw the failure:
openssl s_client -connect <host>:7233 -servername <host> </dev/null
# -connect host:port: who to connect to
# -servername name: set the SNI server nameFor mTLS, add -cert and -key:
openssl s_client -connect <host>:7233 \
-servername <host> \
-cert client.pem -key client.key \
-showcerts </dev/null
# -cert file: client cert, PEM assumed
# -key file: client private key
# -showcerts: show all server certs(The same openssl s_client -connect <endpoint> -showcerts -cert ... -key ... form is used by the Temporal troubleshooting guide.)
Interpret by looking at the last few lines of output:
| Output line | Interpretation | Where to go |
|---|---|---|
Verify return code: 0 (ok) and certificate info printed | TLS succeeded | Not a TLS problem. Move to authentication.md. |
Verify return code: 10 (certificate has expired) or x509: certificate has expired or is not yet valid on the client | Cert past validity on server or client chain | Expired or not-yet-valid |
Verify return code: 19 (self signed certificate in certificate chain) or Verify return code: 20/21 | Client doesn't trust the server chain | Unknown authority |
tlsv1 alert unknown ca / remote error: tls: unknown certificate authority | Server rejected the client CA | Accepted client CA set |
tlsv1 alert bad certificate / remote error: tls: bad certificate | Server rejected the client cert (cert-filter mismatch, malformed cert, or wrong cert presented) | Accepted client CA set and certificate filters in docs/cloud/certificates.mdx |
| Handshake opens TCP but closes with no TLS alert bytes | Middlebox dropping / TLS-inspecting proxy | Back off to connectivity.md → firewall and proxy |
Expired or not-yet-valid
Symptom shapes:
Go client:
x509: certificate has expired or is not yet valid:followed by detailPeer alert:
remote error: tls: expired certificateTemporal Cloud:
Failed reaching server: last connection error— the troubleshooting guide names an expired TLS certificate as a common root cause.Workers that were fine yesterday stopped connecting overnight with no deploy.
What to check first — exact expiry times of each cert in play:
# Local cert file
openssl x509 -enddate -noout -in client.pem
# -enddate: print notAfter field
# -noout: no encoded output
# -in file: input file
# Both notBefore and notAfter
openssl x509 -dates -noout -in client.pem
# -dates: Both Before and After dates
# Server cert fetched from a live endpoint
openssl s_client -connect <host>:7233 -servername <host> </dev/null 2>/dev/null \
| openssl x509 -enddate -nooutFor the accepted-client-ca set on a Cloud Namespace, the troubleshooting guide lists expiry via tcld namespace accepted-client-ca list with jq:
tcld namespace accepted-client-ca list \
--namespace <namespace_id>.<account_id> \
| jq -r '.[0].notAfter'
# tcld namespace accepted-client-ca list:
# --namespace (-n) modifier:
# Recipe exactly as written in:Clock skew — the "not yet valid" variant: the same Go error string covers both "expired" and "not yet valid" (x509: certificate has expired or is not yet valid: ). If date -u on the client disagrees with a time authority by more than the cert's overlap window (common in containers with no NTP, on appliances with a dead RTC battery, or right after a host boot), a cert that is in fact valid will still fail verification. Verify with date -u before regenerating anything.
Classification:
End-entity (leaf) cert expired, CA still valid: regenerate the leaf against the existing CA. See openssl recipes → issue a new leaf with tcld.
CA cert expired (so all leaves under it fail): regenerate CA + leaf, upload the new CA alongside the existing one before removing the old one, then distribute new leaves and restart clients. See Rotation and expiry notifications. The Cloud docs describe this as a "rollover process" that "enables your Namespace to serve both CA certificates for a period of time until traffic to your old CA certificate ceases."
Temporal Cloud server-side cert expired: you don't manage the server side on Cloud. Open a support ticket per
docs/troubleshooting/last-connection-error.mdx.Self-hosted server cert expired: rotate the frontend's server TLS cert on your deployment. No tcld involvement.
:::caution
An expired root CA certificate invalidates all downstream certificates, per docs/cloud/get-started/certificates.mdx:31.
:::
Unknown authority
Symptom shapes:
Go client:
x509: certificate signed by unknown authorityGo client (verify chain):
x509: no valid chains builtorx509: failed to load system roots and no roots providedPeer alert:
remote error: tls: unknown certificate authorityPeer alert:
remote error: tls: bad certificate
Two directions this error travels — check which side is complaining before you act:
Client does not trust server (client-side
x509: ...). The client cannot validate the server certificate against any root it knows about. One thing to check is whether the client's trust store is populated — minimal container images often ship without system CAs — and whether the client is being pointed at a non-Temporal endpoint (TLS-inspecting proxy). The Go client does not ship its own root bundle; it relies on the host's trust store or an explicit--tls-ca-path/TEMPORAL_TLS_CA.Server does not trust client (peer-alert
remote error: tls: unknown certificate authority). For mTLS, the Namespace's accepted-client-ca set does not contain the CA that signed the client cert. Fix in Accepted client CA set.
Verify locally that the client cert chains to the CA you think it chains to:
openssl verify -CAfile ca.pem client.pem
# -CAfile file: Certificate Authority fileIf intermediates exist, supply them via -untrusted:
openssl verify -CAfile root-ca.pem -untrusted intermediate.pem client.pem
# -untrusted file: untrusted certificates fileHostname mismatch
Symptom shapes:
x509: certificate is valid for <SAN list>, not <requested host>x509: certificate is not valid for any names, but wanted to match <host>x509: cannot validate certificate for <host>Go client may also emit:
x509: certificate relies on legacy Common Name field, use SANs insteadwhen a server cert has no SAN and only a CN. Rare on modern clients: this string is from the deprecatedGODEBUG=x509ignoreCN=0path, which Go 1.17 removed. Go 1.17+ rejects the cert with a hostname-mismatch error instead. Treat the legacy-CN string as a Go 1.15–1.16 era signal; if you see it on a current build, the client is pinned to an older Go runtime.
What it means: the hostname the client asked for does not match any Subject Alternative Name (or DNSName) on the server certificate the peer presented.
First check: which hostname is the client asking for?
# Inspect server cert SANs as actually served:
openssl s_client -connect <host>:7233 -servername <host> </dev/null 2>/dev/null \
| openssl x509 -text -noout \
| grep -A1 "Subject Alternative Name"
# -text: print certificate in text formCommon cause on Temporal Cloud: connecting by VPC-endpoint DNS name (PrivateLink) or GCP PSC IP, or by Regional Endpoint, without overriding the TLS server name. The TLS peer serves the Namespace's certificate, whose SAN is the Namespace Endpoint hostname — the VPC-endpoint hostname is not in it. Fix in the next section.
Server name override
Symptom: TLS handshake fails with hostname-mismatch errors (see Hostname mismatch above) when connecting through a PrivateLink VPC endpoint, a GCP Private Service Connect IP, or a Cloud Regional Endpoint.
What the Cloud docs say:
When a client uses a PrivateLink / PSC endpoint instead of the Namespace Endpoint DNS name, the docs instruct you to "Set TLS configuration to override the TLS server name (e.g., my-namespace.my-account.tmprl.cloud)."
When a client uses a Regional Endpoint with mTLS, the docs say: "the Temporal Client must set the
server_nameproperty to<namespace endpoint value>in its request to the value of the Namespace endpoint. This tells the client to expect a different SNI header during the TLS handshake, since the request to the regional endpoint is redirected to the specific Namespace."Temporal recommends configuring private DNS instead, so the Namespace Endpoint hostname resolves to the VPC endpoint directly and no server-name override is needed.
How to override on each client — verified forms only:
temporal CLI flag (value: the Namespace Endpoint, e.g. <namespace>.<account>.tmprl.cloud):
temporal workflow list \
--address vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 \
--namespace <namespace>.<account> \
--tls-cert-path client.pem --tls-key-path client.key \
--tls-server-name <namespace>.<account>.tmprl.cloud
# --tls-server-name: Overrides the target TLS server name
# --tls-cert-path: Path to x509 certificate
# --tls-key-path: Path to private certificate keytemporal CLI env var equivalent:
export TEMPORAL_ADDRESS=vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233
export TEMPORAL_TLS_CERT=/path/to/cert.pem
export TEMPORAL_TLS_KEY=/path/to/cert.key
export TEMPORAL_TLS_SERVER_NAME=<namespace>.<account>.tmprl.cloud
# TEMPORAL_TLS_SERVER_NAME: Override for target TLS server name
# TEMPORAL_TLS_CERT: Path to x509 certificate
# TEMPORAL_TLS_KEY: Path to private certificate key
# TEMPORAL_ADDRESS: Host and port for the Temporal Frontend Service
temporal workflow list --namespace <namespace>.<account>The exact form above — TEMPORAL_ADDRESS=vpce-...:7233 paired with TEMPORAL_TLS_SERVER_NAME=my-namespace.my-account.tmprl.cloud — is written out in the Cloud connectivity guide.
grpcurl (useful as a SDK-independent probe) — exactly the form the Cloud docs give:
grpcurl \
-servername <namespace>.<account>.tmprl.cloud \
-cert path/to/cert.pem \
-key path/to/cert.key \
vpce-0123456789abcdef-abc.us-east-1.vpce.amazonaws.com:7233 \
temporal.api.workflowservice.v1.WorkflowService/GetSystemInfo
# Recipe as written in:For SDK clients: the SDK concept is the same — set the TLS ServerName (Go / Java / .NET / Python / TypeScript) to the Namespace Endpoint hostname. The exact property name varies by SDK; refer to each SDK's client-connection doc linked from docs/cloud/certificates#configure-clients-to-use-client-certificates and cross-check against skill-temporal-developer. This triage file deliberately does not spell SDK APIs out, to avoid drift.
Key does not match cert
Symptom shapes:
tls: failed to find any PEM data in certificate input(the file is empty, wrong format, or the wrong file)Go's
tls.LoadX509KeyPairsurfaces a keypair mismatch at client start. The precise string varies across Go versions, so don't pattern-match on it; the shape is an error at client dial saying the cert and key don't pair.
Verify by comparing modulus hashes of the cert and the private key:
# RSA
openssl x509 -modulus -noout -in client.pem | shasum -a 256
openssl rsa -modulus -noout -in client.key | shasum -a 256
# -modulus: print the RSA key modulus
# The two digests must be identical.If the modulus hashes disagree, the cert and key file are from different keypairs. Find the key that was generated alongside this cert — commonly in the same directory — or regenerate both together.
(For ECDSA keys, -modulus does not apply; compare public keys with openssl pkey -in client.key -pubout against openssl x509 -in client.pem -pubkey -noout.)
Accepted client CA set (mTLS Cloud)
On Temporal Cloud, an mTLS Namespace authenticates a client by validating the client cert against the CA set configured on the Namespace. The server-side error when the CA is not trusted is remote error: tls: unknown certificate authority. When a CA is trusted but the end-entity cert fails other checks (certificate filter mismatch, malformed cert), the server sends remote error: tls: bad certificate.
Certificate filters
Certificate filters are SAN/CN-based allow rules configured on the Namespace that further restrict which client certs are accepted, even when the signing CA is in the accepted set. A filter specifies allowed values for the leaf's Common Name, Subject Organization, Subject Organizational Unit, or SANs (DNS names, URIs, emails); a client cert that chains to a trusted CA but does not match any filter is rejected with remote error: tls: bad certificate.
Filters are the common cause of "the CA is trusted, the cert looks fine, but the handshake still fails." If you see remote error: tls: bad certificate and the local openssl verify -CAfile ... check passes, suspect a filter mismatch before regenerating certs.
# Inspect what filters (if any) are configured on the Namespace
tcld namespace certificate-filters export \
--namespace <namespace_id>.<account_id>
# Reference:
# Inspect the SANs / CN / OU on the client cert that's failing
openssl x509 -in client.pem -noout -subject -ext subjectAltNameIf the filter set is non-empty, the leaf cert's subject fields and SANs must match at least one filter entry. Either re-issue the leaf with subject/SAN values that match an existing filter, or update the filter set to allow the new cert.
List what the Namespace currently accepts:
tcld namespace accepted-client-ca list \
--namespace <namespace_id>.<account_id>
# Command:
# --namespace:Add a new CA to the accepted set:
add appends to the existing set without removing other CAs, so it is the safe verb for the upload-alongside step of a rollover.
tcld namespace accepted-client-ca add \
--namespace <namespace_id>.<account_id> \
--ca-certificate-file <path>
# Command:
# --ca-certificate-file:Remove a CA (by fingerprint, safer than by PEM):
tcld namespace accepted-client-ca remove \
--namespace <namespace_id>.<account_id> \
--ca-certificate-fingerprint <fingerprint>
# Command:
# --ca-certificate-fingerprint (--fp):Set the whole bundle at once (zero-downtime rollover):
The Cloud docs describe a concat-old-plus-new-then-set pattern for rolling over CA certs without dropping traffic:
# 1. Create a file with old + new CA PEM blocks concatenated.
# 2. Run:
tcld namespace accepted-client-ca set \
--namespace <namespace_id>.<account_id> \
--ca-certificate-file <path>
# Command and rollover procedure:
# Same procedure in:
# 3. Wait until all clients present leaves signed by the new CA (operator-confirmed; Cloud shows no drain signal).
# 4. Create a file with only the new CA and run the set command again.:::caution
set replaces the entire accepted-CA bundle with exactly what you pass, and it does not prompt for confirmation. Passing only the new CA silently drops every other trusted CA and locks out any client still presenting a leaf under them. Use add to append a CA during a rollover; use set only with a deliberately concatenated old+new bundle. Trust-changing commands (set, remove, rotation) are high-consequence: run them with operator confirmation of the blast radius, not autonomously.
:::
Verify locally before uploading that the client cert chains to the CA you're about to upload:
openssl verify -CAfile new-ca.pem client.pem
# Expected: client.pem: OKIf this fails locally, it will fail at the peer too. Fix before touching the Namespace.
Certificate requirements (Cloud mTLS)
The docs state hard requirements for any CA or leaf cert you upload to a Cloud Namespace. These are where subtle TLS rejections come from when a cert "looks fine" but the peer sends remote error: tls: bad certificate anyway. Quoting the requirements as listed at docs/cloud/get-started/certificates.mdx:62-96:
CA certificates:
X.509v3.
Each cert in a bundle is either a root or issued by another cert in the bundle.
Each cert includes
CA: true.Cannot be a well-known CA (DigiCert, Let's Encrypt, etc.) unless the user also specifies certificate filters.
Signing algorithm must be RSA or ECDSA and must include SHA-256 or stronger. SHA-1 and MD5 cannot be used.
Cannot be generated with a passphrase.
Bundle ≤ 16 CA certs, ≤ 32 KB before base64 encoding.
In a full end-entity → root chain, each certificate must have a unique Distinguished Name (DN comparison is case-insensitive).
End-entity (leaf) certificates:
X.509v3.
Basic constraints must include
CA: false.Key usage must include Digital Signature.
Signing algorithm: RSA or ECDSA with SHA-256 or stronger.
Certificate Revocation Lists: Temporal does not support or check CRLs (or OCSP); customers are expected to keep certificates up to date. Because there is no revocation list, plan for revocation by other means:
Short certificate lifetimes are the primary control: a short-lived leaf bounds how long a stolen key is usable. Set leaves to expire before their issuing CA.
Removing a CA from the accepted set (
tcld namespace accepted-client-ca remove) is the hard kill switch. This rejects all leaves signed by that CA, so confirm the blast radius before running it.Certificate filters scope acceptance to specific leaf identities (CN/OU/Subject Organization/SAN); tightening or removing a filter cuts off specific certs without rotating the CA.
There is no per-leaf revocation: to cut off a single leaf before it expires, you must rotate or remove its CA. Size leaf lifetimes accordingly.
Algorithm choice when generating with tcld: tcld gen ca defaults to ECDSA P-384; --rsa-algorithm (alias --rsa) switches to a 4096-bit RSA key pair.
The accepted set above (RSA or ECDSA, SHA-256+) is a floor, not a recommendation: Temporal does not mandate a specific key size or curve for customer CAs. Match key strength to certificate lifetime. RSA-2048 (~112-bit) is acceptable for short-lived certs today, but NIST (SP 800-131A Rev. 2) deprecates 112-bit strength after 2030, so a long-lived CA root that must stay trusted past then should use ECDSA (P-256/P-384) or RSA-3072 or larger (4096 preferred; the --rsa flag emits 4096). The tcld default, ECDSA P-384, is a good choice for most cases.
Duration caps when generating with tcld: tcld gen ca has a maximum duration of 1 year (-d 1y). You must set an end-entity cert to expire before its root CA.
Rotation and expiry notifications
Notifications. Temporal Cloud sends email notifications before CA expiry. The notifications doc lists "Certificate Expiring in 15 days" as an admin notification sent to Global Administrators, Namespace Administrators, and Account Owners. The certificates guide states: "Temporal Cloud begins sending notifications 15 days before expiration."
Rollover strategy. The Cloud docs prescribe a zero-downtime rollover pattern: add the new CA alongside the existing one (accepted-client-ca add), wait until every client has rolled to leaves signed by the new CA, then remove the old CA. Cloud exposes no signal for when traffic has fully shifted, so treat this as operator-driven: confirm your worker fleet is on new-CA leaves before removing the old CA. The same shape applies whether you use the UI or tcld.
Rotation command sequence (issue with tcld, upload with tcld): the generated *.key files are secrets; see Private key handling.
# Issue a new CA cert (if rotating CA). Default is ECDSA P-384.
tcld generate-certificates certificate-authority-certificate \
--organization <org> \
--validity-period 1y \
--ca-certificate-file new-ca.pem \
--ca-key-file new-ca.key
# Command (alias: tcld gen ca):
# --organization (--org):
# --validity-period (-d):
# --ca-certificate-file (--ca-cert):
# --ca-key-file (--ca-key):
# Issue a new end-entity (leaf) cert against a CA
tcld generate-certificates end-entity-certificate \
--organization <org> \
--validity-period 364d \
--ca-certificate-file new-ca.pem \
--ca-key-file new-ca.key \
--certificate-file client.pem \
--key-file client.key
# Command (alias: tcld gen leaf):
# --certificate-file (--cert):
# --key-file (--key):
# Upload concatenated old+new CA bundle to the Namespace, then (after drain) upload new-only bundle.
tcld namespace accepted-client-ca set \
--namespace <namespace_id>.<account_id> \
--ca-certificate-file <path-to-bundle>
# Procedure:
# --namespace:When the CA itself is already expired (the rollover window was missed), a new CA must be uploaded before any client can reconnect. This is the "cert expired at 3 a.m." shape; see recipes.md for the full step-by-step.
Private key handling
The recipes in this file generate private keys (new-ca.key, client.key). Treat them as secrets, and treat the CA private key as a root credential: it can mint client certificates the Namespace will accept, and there is no revocation backstop if it leaks (see Certificate Revocation Lists).
Never print, echo,
cat, or log private-key contents. To check whether a key matches a cert, use the modulus/fingerprint comparisons in Compare a keypair and Key does not match cert — those expose no secret material.The key is plaintext on disk. Cloud requires CA keys to be generated without a passphrase, so the file itself is the secret. Restrict permissions (
chmod 600), never commit it to source control (add*.keyto.gitignore), and keep the long-term CA key in a secrets manager or offline storage.Don't leave key material in working or shared directories. Where a key should live long-term depends on the environment, so flag leftover key files to the operator to place in durable secret storage rather than guessing or deleting them.
:::caution
Generating or rotating keys locally is safe, but anything that changes a Namespace's trust (accepted-client-ca set/remove, rotation) or deletes key material is high-consequence: it can lock out every worker on the Namespace or destroy an unrecoverable CA key. Run those steps with operator confirmation, not autonomously.
:::
openssl recipes
These are reference-card forms; each flag is cross-linked to the openssl usage output.
Inspect a local cert
# Full text
openssl x509 -in cert.pem -noout -text
# -in / -noout / -text
# Subject and issuer
openssl x509 -in cert.pem -noout -subject -issuer
# -subject / -issuer
# Validity dates
openssl x509 -in cert.pem -noout -dates
# -dates: both Before and After
# Fingerprint (for tcld --ca-certificate-fingerprint)
openssl x509 -in cert.pem -noout -fingerprint
# -fingerprintNote: LibreSSL (the openssl shipped with macOS as of LibreSSL 3.x) does not support the upstream OpenSSL -ext <extname> filter on openssl x509. To see SANs on LibreSSL, use openssl x509 -text -noout and read the X509v3 Subject Alternative Name section, or use grep as in Hostname mismatch.
Verify a chain
# Leaf + root
openssl verify -CAfile ca.pem client.pem
# Leaf + root + intermediate
openssl verify -CAfile root-ca.pem -untrusted intermediate.pem client.pem
# -CAfile / -untrustedTest a live endpoint
# Public-internet or mTLS Namespace, inspect the peer's cert chain
openssl s_client -connect <host>:7233 \
-servername <host> \
-showcerts </dev/null 2>/dev/null \
| openssl x509 -text -noout
# Full mTLS handshake (what the Temporal troubleshooting guide uses)
openssl s_client -connect <namespace>.<account>.tmprl.cloud:7233 \
-showcerts \
-cert client.pem -key client.key \
-tls1_2 </dev/null
# Recipe as written in the troubleshooting guide:
# -tls1_2: force TLSv1.2 (the troubleshooting guide uses this)Reading the output:
Server cert block(s) appear between
-----BEGIN CERTIFICATE-----markers underCertificate chain.The last line normally prints
Verify return code: 0 (ok)on success; a non-zero code is the OpenSSL verify error, e.g.10 (certificate has expired).If the server sent a TLS alert rather than certs, you will see
tlsv1 alert <description>; the alert description is one of the words listed insrc/crypto/tls/alert.go(handshake failure,bad certificate,unknown certificate authority,expired certificate, etc.).
Compare a keypair
openssl x509 -in cert.pem -modulus -noout | shasum -a 256
openssl rsa -in key.pem -modulus -noout | shasum -a 256
# Digests must match for the files to be from the same keypair.Self-signed cert workflow (self-hosted or temporary)
The troubleshooting guide suggests using temporal namespace describe with explicit TLS flags when working with self-signed certs:
temporal namespace describe \
--namespace <namespace_id>.<account_id> \
--address <namespace_grpc_endpoint> \
--tls-cert-path <path-to-mTLS-pem-file> \
--tls-key-path <path-to-mTLS-key-file>
# Recipe as written in:Issue a new leaf with tcld
tcld gen leaf \
--org <org> \
-d 364d \
--ca-cert ca.pem --ca-key ca.key \
--cert client.pem --key client.key
# tcld gen leaf = tcld generate-certificates end-entity-certificate
# Aliases listed in:
# Example as written in:TLS / cert error reference
Each row is tagged with the string's origin. When an error doesn't fit any row here, re-read Error-string origin cheat sheet to decide whether it is really a TLS-layer error at all.
| Error text (exact, as emitted) | Origin | One thing to check |
|---|---|---|
x509: certificate has expired or is not yet valid: <detail> | Go x509 | Expired or not-yet-valid; also check date -u for clock skew |
x509: certificate signed by unknown authority | Go x509 | Client doesn't trust peer's root. Unknown authority |
x509: certificate is valid for <SAN>, not <host> | Go x509 | Server name override |
x509: cannot validate certificate for <host> | Go x509 | Hostname mismatch |
x509: no valid chains built | Go x509 | Chain does not reach a trusted root; verify with openssl verify -CAfile ... |
x509: a root or intermediate certificate is not authorized to sign for this name: <detail> | Go x509 | Name-constraints extension rejects the leaf |
x509: certificate is not authorized to sign other certificates | Go x509 | Cert with CA: false is being used as an issuer |
x509: failed to load system roots and no roots provided | Go x509 | Minimal container missing ca-certificates; or TEMPORAL_TLS_CA / --tls-ca-path not set |
x509: certificate relies on legacy Common Name field, use SANs instead | Go x509 (Go 1.15–1.16 era) | Peer cert has no SAN, only CN; re-issue with SANs. Go 1.17+ emits hostname-mismatch instead; if you see this on current Go, the client is pinned to an older runtime |
tls: handshake failure | Go tls (local) | See Handshake failure and reproduce with openssl s_client |
remote error: tls: handshake failure | peer alert | Peer rejected the handshake; reproduce with openssl s_client for the alert description |
remote error: tls: bad certificate | peer alert | Accepted client CA set; also check certificate filters |
remote error: tls: unknown certificate authority | peer alert | Accepted client CA set |
remote error: tls: expired certificate | peer alert | Expired or not-yet-valid |
remote error: tls: internal error | peer alert | Peer-side failure, not a cert problem on this end; retry and, if persistent on Cloud, open a support ticket |
Failed reaching server: last connection error | Temporal client diagnostic | Often an expired TLS cert per the troubleshooting guide; run the expiry checks in Expired or not-yet-valid |
For TLS errors that only appear after the handshake (gRPC UNAUTHENTICATED, PERMISSION_DENIED), jump to authentication.md. For UNAVAILABLE without a TLS-layer cause, back off to connectivity.md.
Requirements and compatibility
Written for: OpenAI Codex, Claude Code
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Claude Code frontmatter, Needs an API key
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- Temporal
- Publisher
- OpenAI (Official)
- Distributed by
- OpenAI
- Source
- openai/plugins · plugins/temporal/skills/temporal-ops · commit d420d13
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from plugins/temporal/LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Temporal or OpenAI.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download temporal-ops.zip