Azure Aigateway

MicrosoftOfficialCreated by Microsoft

Skill
Tool integration
Cloud and DevOps
MIT
Low risk

From the author

Configure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbreak detection, add Azure OpenAI backend, add AI Foundry model, test AI gateway, LLM policies, configure AI backend, token metrics, AI cost control, convert API to MCP, import OpenAPI to gateway.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides the configuration patterns and policy references needed to transform Azure API Management (APIM) into a robust AI Gateway. It enables centralized governance for AI models, including Azure OpenAI and AI Foundry, by implementing essential features like semantic caching, token usage tracking, load balancing, and content safety filtering. The skill covers the full lifecycle from backend setup and managed identity authentication to advanced policy application and troubleshooting.

What it does

Connects AI models as backends within Azure API Management.
Implements cost control through token limits and semantic caching.
Enforces security policies like jailbreak detection and content safety.
Enables load balancing across multiple AI backend instances.
Provides observability by emitting token usage metrics.
Facilitates MCP tool protection and rate limiting.

When to use it

When you need to govern and secure access to AI models in production.
When you want to implement cost controls for LLM usage.
When you need to load balance requests across multiple AI regions.
When you need to protect AI agents with content filtering and jailbreak detection.
When you want to convert existing APIs into MCP-compatible tools.

Advantages

Centralized management of AI policies and security.
Significant cost savings via semantic caching.
Improved reliability through load balancing and retry logic.
Enhanced security using managed identities for backend authentication.
Detailed observability into token consumption and model performance.

Usage examples

“Add a new Azure OpenAI backend to my existing APIM instance.”: The agent provides the CLI commands to create the backend and the XML policy snippet to authenticate using managed identity.
“How do I implement semantic caching to save on AI costs?”: The agent explains how to apply the azure-openai-semantic-cache-lookup/store policies within the APIM inbound processing block.
“Set up load balancing for my AI models across two regions.”: The agent provides a policy pattern that uses a variable to rotate between backend URLs based on request hash.
“How can I protect my AI gateway from jailbreak attempts?”: The agent guides you to apply the llm-content-safety policy to filter harmful content and detect malicious inputs.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/azure-aigateway/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip azure-aigateway.zip -d .agents/skills

Documentation by Microsoft

Azure AI Gateway

Configure Azure API Management (APIM) as an AI Gateway for governing AI models, MCP tools, and agents.

To deploy APIM, use the azure-prepare skill. See APIM deployment guide.

When to Use This Skill

Category	Triggers
Model Governance	"semantic caching", "token limits", "load balance AI", "track token usage"
Tool Governance	"rate limit MCP", "protect my tools", "configure my tool", "convert API to MCP"
Agent Governance	"content safety", "jailbreak detection", "filter harmful content"
Configuration	"add Azure OpenAI backend", "configure my model", "add AI Foundry model"
Testing	"test AI gateway", "call OpenAI through gateway"

Quick Reference

Policy	Purpose	Details
`azure-openai-token-limit`	Cost control	Model Policies
`azure-openai-semantic-cache-lookup/store`	60-80% cost savings	Model Policies
`azure-openai-emit-token-metric`	Observability	Model Policies
`llm-content-safety`	Safety & compliance	Agent Policies
`rate-limit-by-key`	MCP/tool protection	Tool Policies

Get Gateway Details

# Get gateway URL
az apim show --name <apim-name> --resource-group <rg> --query "gatewayUrl" -o tsv

# List backends (AI models)
az apim backend list --service-name <apim-name> --resource-group <rg> \
  --query "[].{id:name, url:url}" -o table

# Get subscription key
az apim subscription keys list \
  --service-name <apim-name> --resource-group <rg> --subscription-id <sub-id>

Test AI Endpoint

GATEWAY_URL=$(az apim show --name <apim-name> --resource-group <rg> --query "gatewayUrl" -o tsv)

curl -X POST "${GATEWAY_URL}/openai/deployments/<deployment>/chat/completions?api-version=2024-02-01" \
  -H "Content-Type: application/json" \
  -H "Ocp-Apim-Subscription-Key: <key>" \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'

Common Tasks

Add AI Backend

See references/patterns.md for full steps.

# Discover AI resources
az cognitiveservices account list --query "[?kind=='OpenAI']" -o table

# Create backend
az apim backend create --service-name <apim> --resource-group <rg> \
  --backend-id openai-backend --protocol http --url "https://<aoai>.openai.azure.com/openai"

# Grant access (managed identity)
az role assignment create --assignee <apim-principal-id> \
  --role "Cognitive Services User" --scope <aoai-resource-id>

Apply AI Governance Policy

Recommended policy order in <inbound>:

Authentication - Managed identity to backend
Semantic Cache Lookup - Check cache before calling AI
Token Limits - Cost control
Content Safety - Filter harmful content
Backend Selection - Load balancing
Metrics - Token usage tracking

See references/policies.md for complete example.

Troubleshooting

Issue	Solution
Token limit 429	Increase `tokens-per-minute` or add load balancing
No cache hits	Lower `score-threshold` to 0.7
Content false positives	Increase category thresholds (5-6)
Backend auth 401	Grant APIM "Cognitive Services User" role

See references/troubleshooting.md for details.

References

Detailed Policies - Full policy examples
Configuration Patterns - Step-by-step patterns
Troubleshooting - Common issues
AI-Gateway Samples
GenAI Gateway Docs

SDK Quick References

Content Safety: Python | TypeScript
API Management: Python | .NET

Files

ATTRIBUTION.mdAdded by AutoGineer977 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md4.9 KiB
references/auth-best-practices.md6.1 KiB
references/patterns.md6.6 KiB
references/policies.md9.2 KiB
references/sdk/azure-ai-contentsafety-py.md1.3 KiB
references/sdk/azure-ai-contentsafety-ts.md1.4 KiB
references/sdk/azure-mgmt-apimanagement-dotnet.md1.3 KiB
references/sdk/azure-mgmt-apimanagement-py.md1.0 KiB
references/troubleshooting.md7.7 KiB

references/policies.md

AI Gateway Policies

Complete reference for Azure API Management AI governance policies.

Policy Placement Order

Recommended order in <inbound> section:

1. Authentication (managed identity)
2. Semantic Cache Lookup
3. Token Rate Limiting
4. Content Safety
5. Backend Selection / Load Balancing
6. Token Metrics

Model Policies

Token Rate Limiting

Control costs by limiting token consumption per minute.

<azure-openai-token-limit
    tokens-per-minute="50000"
    counter-key="@(context.Subscription.Id)"
    estimate-prompt-tokens="true"
    tokens-consumed-header-name="x-tokens-consumed"
    remaining-tokens-header-name="x-tokens-remaining" />

Attribute	Purpose	Default
`tokens-per-minute`	Max tokens per counter window	Required
`counter-key`	Grouping key (subscription, IP, custom)	Required
`estimate-prompt-tokens`	Count prompt tokens toward limit	`true`
`tokens-consumed-header-name`	Response header with consumed count	—
`remaining-tokens-header-name`	Response header with remaining count	—

Usage tiers example:

<!-- Free tier: 5K TPM -->
<azure-openai-token-limit tokens-per-minute="5000"
    counter-key="@("free-" + context.Subscription.Id)"
    estimate-prompt-tokens="true" />

<!-- Premium tier: 100K TPM -->
<azure-openai-token-limit tokens-per-minute="100000"
    counter-key="@("premium-" + context.Subscription.Id)"
    estimate-prompt-tokens="true" />

Semantic Caching

Cache AI responses for semantically similar prompts. Saves 60-80% on repeated queries.

Lookup (in <inbound>):

<azure-openai-semantic-cache-lookup
    score-threshold="0.8"
    embeddings-backend-id="embeddings-backend"
    embeddings-backend-auth="system-assigned" />

Store (in <outbound>):

<azure-openai-semantic-cache-store duration="3600" />

Attribute	Purpose	Recommended
`score-threshold`	Similarity threshold (0-1)	0.8 (lower = more cache hits)
`embeddings-backend-id`	Backend for embedding generation	Required
`embeddings-backend-auth`	Auth to embeddings backend	`system-assigned`
`duration`	Cache TTL in seconds	3600 (1 hour)

Prerequisites:

An embeddings model deployed (e.g., text-embedding-ada-002)
A separate backend pointing to the embeddings endpoint
Azure Cache for Redis Enterprise with RediSearch module (for vector storage)

# Create embeddings backend
az apim backend create --service-name <apim> --resource-group <rg> \
  --backend-id embeddings-backend --protocol http \
  --url "https://<aoai>.openai.azure.com/openai"

Note: Semantic caching is NOT compatible with streaming responses ("stream": true).

Token Metrics

Emit token usage metrics for monitoring and chargeback.

<azure-openai-emit-token-metric namespace="ai-gateway">
    <dimension name="Subscription" value="@(context.Subscription.Id)" />
    <dimension name="API" value="@(context.Api.Name)" />
    <dimension name="Model" value="@(context.Request.Headers.GetValueOrDefault("x-model", "unknown"))" />
    <dimension name="Operation" value="@(context.Operation.Id)" />
</azure-openai-emit-token-metric>

Emits to Azure Monitor with these metrics:

Total Tokens — prompt + completion combined
Prompt Tokens — input tokens
Completion Tokens — output tokens

Query token usage (KQL):

customMetrics
| where name == "Total Tokens"
| extend Subscription = tostring(customDimensions["Subscription"])
| summarize TotalTokens = sum(value) by Subscription, bin(timestamp, 1h)
| order by TotalTokens desc

Agent Policies

Content Safety

Filter harmful, violent, or inappropriate content from AI inputs and outputs.

<!-- In <inbound> -->
<llm-content-safety backend-id="contentsafety-backend">
    <category name="Hate" threshold="4" />
    <category name="Sexual" threshold="4" />
    <category name="SelfHarm" threshold="4" />
    <category name="Violence" threshold="4" />
</llm-content-safety>

Category	Description	Threshold Range
Hate	Discrimination, slurs	0 (block all) - 6 (allow most)
Sexual	Explicit content	0-6
SelfHarm	Self-injury content	0-6
Violence	Violent content	0-6

Prerequisites:

Azure AI Content Safety resource deployed
Backend configured for the Content Safety endpoint:

az apim backend create --service-name <apim> --resource-group <rg> \
  --backend-id contentsafety-backend --protocol http \
  --url "https://<contentsafety>.cognitiveservices.azure.com"

Jailbreak Detection

Block prompt injection attacks that attempt to bypass AI safety guardrails.

<llm-content-safety backend-id="contentsafety-backend">
    <category name="Hate" threshold="4" />
    <category name="Sexual" threshold="4" />
    <category name="SelfHarm" threshold="4" />
    <category name="Violence" threshold="4" />
    <!-- Jailbreak detection is automatic when content safety is enabled -->
</llm-content-safety>

Custom response for blocked content:

<on-error>
    <base />
    <choose>
        <when condition="@(context.LastError.Source == "llm-content-safety")">
            <return-response>
                <set-status code="400" reason="Content Filtered" />
                <set-body>{"error": "Request blocked by content safety policy"}</set-body>
            </return-response>
        </when>
    </choose>
</on-error>

Tool Policies

Request Rate Limiting

Protect MCP tools and API endpoints from excessive requests.

<!-- Per-agent rate limiting -->
<rate-limit-by-key calls="30" renewal-period="60"
    counter-key="@(context.Request.Headers.GetValueOrDefault("X-Agent-Id", "anonymous"))"
    remaining-calls-header-name="x-ratelimit-remaining"
    retry-after-header-name="Retry-After" />

<!-- Per-subscription rate limiting -->
<rate-limit-by-key calls="100" renewal-period="60"
    counter-key="@(context.Subscription.Id)" />

Combining Policies

Complete inbound policy example with all governance layers:

<policies>
    <inbound>
        <base />

        <!-- 1. Authentication -->
        <authentication-managed-identity resource="https://cognitiveservices.azure.com" />

        <!-- 2. Semantic Cache Lookup -->
        <azure-openai-semantic-cache-lookup
            score-threshold="0.8"
            embeddings-backend-id="embeddings-backend"
            embeddings-backend-auth="system-assigned" />

        <!-- 3. Token Rate Limiting -->
        <azure-openai-token-limit
            tokens-per-minute="50000"
            counter-key="@(context.Subscription.Id)"
            estimate-prompt-tokens="true" />

        <!-- 4. Content Safety -->
        <llm-content-safety backend-id="contentsafety-backend">
            <category name="Hate" threshold="4" />
            <category name="Sexual" threshold="4" />
            <category name="SelfHarm" threshold="4" />
            <category name="Violence" threshold="4" />
        </llm-content-safety>

        <!-- 5. Backend Selection -->
        <set-backend-service backend-id="openai-backend" />

        <!-- 6. Token Metrics -->
        <azure-openai-emit-token-metric namespace="ai-gateway">
            <dimension name="Subscription" value="@(context.Subscription.Id)" />
            <dimension name="API" value="@(context.Api.Name)" />
        </azure-openai-emit-token-metric>
    </inbound>

    <backend>
        <forward-request timeout="120" />
    </backend>

    <outbound>
        <base />
        <!-- Cache store (after successful response) -->
        <azure-openai-semantic-cache-store duration="3600" />
    </outbound>

    <on-error>
        <base />
        <choose>
            <when condition="@(context.LastError.Source == "azure-openai-token-limit")">
                <return-response>
                    <set-status code="429" reason="Token Limit Exceeded" />
                    <set-header name="Retry-After" exists-action="override">
                        <value>60</value>
                    </set-header>
                    <set-body>{"error": "Token rate limit exceeded. Try again later."}</set-body>
                </return-response>
            </when>
        </choose>
    </on-error>
</policies>

Policy Quick-Decision Table

Need	Policy	Section
Control token spend	`azure-openai-token-limit`	`<inbound>`
Cache similar prompts	`azure-openai-semantic-cache-lookup/store`	`<inbound>` / `<outbound>`
Track token usage	`azure-openai-emit-token-metric`	`<inbound>`
Block harmful content	`llm-content-safety`	`<inbound>`
Rate limit API calls	`rate-limit-by-key`	`<inbound>`
Authenticate to backend	`authentication-managed-identity`	`<inbound>`
Load balance backends	`set-backend-service` + retry	`<inbound>`

References

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs a command-line tool

The author says: Requires Azure CLI (az) for configuration and testing

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills · .github/plugins/azure-skills/skills/azure-aigateway · commit 0696600

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from .github/plugins/azure-skills/LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

microsoft/azure-skills

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download azure-aigateway.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Microsoft

Azure AI Gateway

When to Use This Skill

Quick Reference

Get Gateway Details

Test AI Endpoint

Common Tasks

Add AI Backend

Apply AI Governance Policy

Troubleshooting

References

SDK Quick References

Files

references/policies.md

AI Gateway Policies

Policy Placement Order

Model Policies

Token Rate Limiting

Semantic Caching

Token Metrics

Agent Policies

Content Safety

Jailbreak Detection

Tool Policies

Request Rate Limiting

Combining Policies

Policy Quick-Decision Table

References

Requirements and compatibility

Security

Attribution and licence

Related skills

Modal

Cloud Infrastructure Network Engineer

Deployment Validation Cloud Architect

Airflow Dag Patterns

Cost Optimization

Hybrid Cloud Networking

Download ZIP