Azure Aigateway

MicrosoftOfficialCreated by Microsoft

Skill
Tool integration
Cloud and DevOps
MIT
Low risk

From the author

Configure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbreak detection, add Azure OpenAI backend, add AI Foundry model, test AI gateway, LLM policies, configure AI backend, token metrics, AI cost control, convert API to MCP, import OpenAPI to gateway.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides the configuration patterns and policy references needed to transform Azure API Management (APIM) into a robust AI Gateway. It enables centralized governance for AI models, including Azure OpenAI and AI Foundry, by implementing essential features like semantic caching, token usage tracking, load balancing, and content safety filtering. The skill covers the full lifecycle from backend setup and managed identity authentication to advanced policy application and troubleshooting.

What it does

Connects AI models as backends within Azure API Management.
Implements cost control through token limits and semantic caching.
Enforces security policies like jailbreak detection and content safety.
Enables load balancing across multiple AI backend instances.
Provides observability by emitting token usage metrics.
Facilitates MCP tool protection and rate limiting.

When to use it

When you need to govern and secure access to AI models in production.
When you want to implement cost controls for LLM usage.
When you need to load balance requests across multiple AI regions.
When you need to protect AI agents with content filtering and jailbreak detection.
When you want to convert existing APIs into MCP-compatible tools.

Advantages

Centralized management of AI policies and security.
Significant cost savings via semantic caching.
Improved reliability through load balancing and retry logic.
Enhanced security using managed identities for backend authentication.
Detailed observability into token consumption and model performance.

Usage examples

“Add a new Azure OpenAI backend to my existing APIM instance.”: The agent provides the CLI commands to create the backend and the XML policy snippet to authenticate using managed identity.
“How do I implement semantic caching to save on AI costs?”: The agent explains how to apply the azure-openai-semantic-cache-lookup/store policies within the APIM inbound processing block.
“Set up load balancing for my AI models across two regions.”: The agent provides a policy pattern that uses a variable to rotate between backend URLs based on request hash.
“How can I protect my AI gateway from jailbreak attempts?”: The agent guides you to apply the llm-content-safety policy to filter harmful content and detect malicious inputs.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/azure-aigateway/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip azure-aigateway.zip -d .agents/skills

Documentation by Microsoft

Azure AI Gateway

Configure Azure API Management (APIM) as an AI Gateway for governing AI models, MCP tools, and agents.

To deploy APIM, use the azure-prepare skill. See APIM deployment guide.

When to Use This Skill

Category	Triggers
Model Governance	"semantic caching", "token limits", "load balance AI", "track token usage"
Tool Governance	"rate limit MCP", "protect my tools", "configure my tool", "convert API to MCP"
Agent Governance	"content safety", "jailbreak detection", "filter harmful content"
Configuration	"add Azure OpenAI backend", "configure my model", "add AI Foundry model"
Testing	"test AI gateway", "call OpenAI through gateway"

Quick Reference

Policy	Purpose	Details
`azure-openai-token-limit`	Cost control	Model Policies
`azure-openai-semantic-cache-lookup/store`	60-80% cost savings	Model Policies
`azure-openai-emit-token-metric`	Observability	Model Policies
`llm-content-safety`	Safety & compliance	Agent Policies
`rate-limit-by-key`	MCP/tool protection	Tool Policies

Get Gateway Details

# Get gateway URL
az apim show --name <apim-name> --resource-group <rg> --query "gatewayUrl" -o tsv

# List backends (AI models)
az apim backend list --service-name <apim-name> --resource-group <rg> \
  --query "[].{id:name, url:url}" -o table

# Get subscription key
az apim subscription keys list \
  --service-name <apim-name> --resource-group <rg> --subscription-id <sub-id>

Test AI Endpoint

GATEWAY_URL=$(az apim show --name <apim-name> --resource-group <rg> --query "gatewayUrl" -o tsv)

curl -X POST "${GATEWAY_URL}/openai/deployments/<deployment>/chat/completions?api-version=2024-02-01" \
  -H "Content-Type: application/json" \
  -H "Ocp-Apim-Subscription-Key: <key>" \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'

Common Tasks

Add AI Backend

See references/patterns.md for full steps.

# Discover AI resources
az cognitiveservices account list --query "[?kind=='OpenAI']" -o table

# Create backend
az apim backend create --service-name <apim> --resource-group <rg> \
  --backend-id openai-backend --protocol http --url "https://<aoai>.openai.azure.com/openai"

# Grant access (managed identity)
az role assignment create --assignee <apim-principal-id> \
  --role "Cognitive Services User" --scope <aoai-resource-id>

Apply AI Governance Policy

Recommended policy order in <inbound>:

Authentication - Managed identity to backend
Semantic Cache Lookup - Check cache before calling AI
Token Limits - Cost control
Content Safety - Filter harmful content
Backend Selection - Load balancing
Metrics - Token usage tracking

See references/policies.md for complete example.

Troubleshooting

Issue	Solution
Token limit 429	Increase `tokens-per-minute` or add load balancing
No cache hits	Lower `score-threshold` to 0.7
Content false positives	Increase category thresholds (5-6)
Backend auth 401	Grant APIM "Cognitive Services User" role

See references/troubleshooting.md for details.

References

Detailed Policies - Full policy examples
Configuration Patterns - Step-by-step patterns
Troubleshooting - Common issues
AI-Gateway Samples
GenAI Gateway Docs

SDK Quick References

Content Safety: Python | TypeScript
API Management: Python | .NET

Files

ATTRIBUTION.mdAdded by AutoGineer977 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md4.9 KiB
references/auth-best-practices.md6.1 KiB
references/patterns.md6.6 KiB
references/policies.md9.2 KiB
references/sdk/azure-ai-contentsafety-py.md1.3 KiB
references/sdk/azure-ai-contentsafety-ts.md1.4 KiB
references/sdk/azure-mgmt-apimanagement-dotnet.md1.3 KiB
references/sdk/azure-mgmt-apimanagement-py.md1.0 KiB
references/troubleshooting.md7.7 KiB

references/patterns.md

AI Gateway Configuration Patterns

Step-by-step patterns for configuring Azure API Management as an AI Gateway.

Pattern 1: Add AI Model Backend

Connect Azure OpenAI or AI Foundry models to your APIM instance.

Prerequisites

APIM instance deployed (use azure-prepare skill to deploy APIM — see APIM deployment guide)
Azure OpenAI or AI Foundry resource provisioned
System-assigned or user-assigned managed identity enabled on APIM

Steps

1. Discover AI Resources

# Find Azure OpenAI resources
az cognitiveservices account list --query "[?kind=='OpenAI'].{name:name, rg:resourceGroup, endpoint:properties.endpoint}" -o table

# Find AI Foundry resources (if using)
az cognitiveservices account list --query "[?kind=='AIServices'].{name:name, rg:resourceGroup}" -o table

2. Enable Managed Identity on APIM

# Enable system-assigned identity
az apim update --name <apim-name> --resource-group <rg> --set identity.type=SystemAssigned

# Get principal ID
PRINCIPAL_ID=$(az apim show --name <apim-name> --resource-group <rg> --query "identity.principalId" -o tsv)

3. Grant RBAC Access

AOAI_ID=$(az cognitiveservices account show --name <aoai-name> --resource-group <rg> --query id -o tsv)

az role assignment create \
  --assignee "$PRINCIPAL_ID" \
  --role "Cognitive Services User" \
  --scope "$AOAI_ID"

4. Create Backend

az apim backend create \
  --service-name <apim-name> \
  --resource-group <rg> \
  --backend-id openai-backend \
  --protocol http \
  --url "https://<aoai-name>.openai.azure.com/openai"

5. Import API (OpenAPI Spec)

# Import the Azure OpenAI API specification
az apim api import \
  --service-name <apim-name> \
  --resource-group <rg> \
  --api-id azure-openai-api \
  --path "openai" \
  --specification-format OpenApi \
  --specification-url "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/2024-02-01/inference.json" \
  --service-url "https://<aoai-name>.openai.azure.com/openai"

6. Set Backend Policy

Add managed identity authentication in <inbound>:

<inbound>
    <base />
    <set-backend-service backend-id="openai-backend" />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
</inbound>

Pattern 2: Load Balance Across Multiple AI Backends

Distribute requests across multiple Azure OpenAI instances for higher throughput.

Steps

1. Create Multiple Backends

# Primary region
az apim backend create --service-name <apim> --resource-group <rg> \
  --backend-id openai-eastus --protocol http \
  --url "https://<aoai-eastus>.openai.azure.com/openai"

# Secondary region
az apim backend create --service-name <apim> --resource-group <rg> \
  --backend-id openai-westus --protocol http \
  --url "https://<aoai-westus>.openai.azure.com/openai"

2. Create Backend Pool

Using APIM backend pool (preview) or policy-based load balancing:

<inbound>
    <base />
    <set-variable name="backendUrl" value="@{
        var backends = new [] {
            "https://aoai-eastus.openai.azure.com",
            "https://aoai-westus.openai.azure.com"
        };
        var hash = Math.Abs(context.RequestId.GetHashCode());
        var index = hash % backends.Length;
        return backends[index];
    }" />
    <set-backend-service base-url="@((string)context.Variables["backendUrl"] + "/openai")" />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
</inbound>

3. Add Circuit Breaker (Retry on 429)

<retry condition="@(context.Response.StatusCode == 429)" count="3" interval="10" delta="5" max-interval="30" first-fast-retry="false">
    <set-variable name="backendUrl" value="@{
        var backends = new [] {
            "https://aoai-eastus.openai.azure.com",
            "https://aoai-westus.openai.azure.com"
        };
        var currentIndex = Array.IndexOf(backends, (string)context.Variables["backendUrl"]);
        return backends[(currentIndex + 1) % backends.Length];
    }" />
    <set-backend-service base-url="@((string)context.Variables["backendUrl"] + "/openai")" />
    <forward-request />
</retry>

Pattern 3: Convert API to MCP Tool

Expose an existing API through APIM as an MCP-compatible tool for AI agents.

Steps

Import API into APIM using OpenAPI spec
Add rate limiting to protect the tool endpoint
Add content safety to filter harmful inputs
Generate MCP manifest pointing to the APIM endpoint

<!-- Rate limit MCP tool calls -->
<inbound>
    <base />
    <rate-limit-by-key calls="10" renewal-period="60"
        counter-key="@(context.Request.Headers.GetValueOrDefault("X-Agent-Id", "anonymous"))" />
</inbound>

Pattern 4: Add Streaming Support

Configure APIM to properly handle Server-Sent Events (SSE) for streaming AI responses.

<inbound>
    <base />
    <set-backend-service backend-id="openai-backend" />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
</inbound>
<outbound>
    <base />
    <set-header name="Content-Type" exists-action="override">
        <value>@(context.Request.Body.As<JObject>()["stream"]?.Value<bool>() == true
            ? "text/event-stream" : "application/json")</value>
    </set-header>
</outbound>

Note: Semantic caching and token metrics policies are NOT compatible with streaming responses. Use non-streaming for cost control scenarios.

Pattern 5: Multi-Tenant AI Gateway

Isolate tenants with per-client rate limiting and tracking.

<inbound>
    <base />
    <!-- Extract tenant from subscription or header -->
    <set-variable name="tenantId" value="@(context.Subscription.Id)" />

    <!-- Per-tenant token limit -->
    <azure-openai-token-limit
        tokens-per-minute="10000"
        counter-key="@((string)context.Variables["tenantId"])"
        estimate-prompt-tokens="true" />

    <!-- Per-tenant metrics -->
    <azure-openai-emit-token-metric namespace="ai-gateway">
        <dimension name="Tenant" value="@((string)context.Variables["tenantId"])" />
        <dimension name="API" value="@(context.Api.Name)" />
    </azure-openai-emit-token-metric>

    <set-backend-service backend-id="openai-backend" />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
</inbound>

Next Steps

Apply governance policies to your configured backends
Review troubleshooting for common configuration issues

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs a command-line tool

The author says: Requires Azure CLI (az) for configuration and testing

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills · .github/plugins/azure-skills/skills/azure-aigateway · commit 0696600

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from .github/plugins/azure-skills/LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

microsoft/azure-skills

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download azure-aigateway.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Microsoft

Azure AI Gateway

When to Use This Skill

Quick Reference

Get Gateway Details

Test AI Endpoint

Common Tasks

Add AI Backend

Apply AI Governance Policy

Troubleshooting

References

SDK Quick References

Files

references/patterns.md

AI Gateway Configuration Patterns

Pattern 1: Add AI Model Backend

Prerequisites

Steps

1. Discover AI Resources

2. Enable Managed Identity on APIM

3. Grant RBAC Access

4. Create Backend

5. Import API (OpenAPI Spec)

6. Set Backend Policy

Pattern 2: Load Balance Across Multiple AI Backends

Steps

1. Create Multiple Backends

2. Create Backend Pool

3. Add Circuit Breaker (Retry on 429)

Pattern 3: Convert API to MCP Tool

Steps

Pattern 4: Add Streaming Support

Pattern 5: Multi-Tenant AI Gateway

Next Steps

Requirements and compatibility

Security

Attribution and licence

Related skills

Modal

Cloud Infrastructure Network Engineer

Deployment Validation Cloud Architect

Airflow Dag Patterns

Cost Optimization

Hybrid Cloud Networking

Download ZIP