AutoGineer
Generated banner for Azure AI Projects Py
Generated logo for Azure AI Projects Py

Azure AI Projects Py

MicrosoftOfficialCreated by Microsoft

  • Skill
  • Tool integration
  • Software development
  • MIT
  • Low risk
  • Has scripts
Download ZIP

From the author

Build AI applications using the Azure AI Projects Python SDK (azure-ai-projects). Use when working with Foundry project clients, creating versioned agents with PromptAgentDefinition, running evaluations, managing connections/deployments/datasets/indexes, or using OpenAI-compatible clients. This is the high-level Foundry SDK - for low-level agent operations, use azure-ai-agents-python skill.
— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The azure-ai-projects-py skill provides an interface for interacting with the Azure AI Projects Python SDK. It enables developers to manage the lifecycle of AI agents, deployments, datasets, and indexes within the Foundry environment. The SDK supports both synchronous and asynchronous operations, integrates with Azure identity management, and provides an OpenAI-compatible client for standard API interactions.

What it does

  • Create and manage versioned AI agents using PromptAgentDefinition
  • List and manage model deployments, datasets, and indexes
  • Execute and manage agent threads, messages, and runs
  • Access an OpenAI-compatible client for standard chat completions
  • Configure and run evaluations on AI models
  • Manage project connections to external services

When to use it

  • When building AI applications that require integration with Microsoft Foundry
  • When you need to manage agent versions and definitions programmatically
  • When performing batch evaluations or quality checks on AI outputs
  • When you need to interface with Azure-managed AI resources and deployments

Advantages

  • Provides a unified client for managing diverse AI project resources
  • Supports both sync and async programming patterns
  • Enables versioning for agent definitions
  • Integrates with standard Azure authentication via DefaultAzureCredential

Usage examples

“How do I initialize the AIProjectClient?”
Use the AIProjectClient class with your project endpoint and a DefaultAzureCredential instance, wrapped in a context manager.
“How can I create a versioned agent?”
Use the client.agents.create_version method, passing a PromptAgentDefinition object with your model, instructions, and version label.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/microsoft/azure-ai-projects-py/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip azure-ai-projects-py.zip -d .agents/skills

Documentation by Microsoft

Azure AI Projects Python SDK (Foundry SDK)

Build AI applications on Microsoft Foundry using the azure-ai-projects SDK.

Installation

pip install azure-ai-projects azure-identity

Environment Variables

AZURE_AI_PROJECT_ENDPOINT="https://<resource>.services.ai.azure.com/api/projects/<project>"  # Required for all auth methods
AZURE_AI_MODEL_DEPLOYMENT_NAME="gpt-4o-mini"  # Required for all auth methods
AZURE_TOKEN_CREDENTIALS=prod # Required only if DefaultAzureCredential is used in production

Authentication & Lifecycle

🔑 Two rules apply to every code sample below:

  1. Prefer DefaultAzureCredential. It works locally (Azure CLI / VS Code / Developer CLI) and in Azure (managed identity, workload identity) with no code change. Avoid connection strings, account/API keys — they bypass Entra audit and rotation.

    • Local dev: DefaultAzureCredential works as-is.

    • Production: set AZURE_TOKEN_CREDENTIALS=prod (or AZURE_TOKEN_CREDENTIALS=<specific_credential>) to constrain the credential chain to production-safe credentials.

  2. Wrap every client in a context manager so HTTP transports, sockets, and token caches are released deterministically:

    • Sync: with <Client>(...) as client:

    • Async: async with <Client>(...) as client: and async with DefaultAzureCredential() as credential: (from azure.identity.aio)

Snippets may abbreviate this setup, but production code should always follow both rules.

import os
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.ai.projects import AIProjectClient

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential()
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
with AIProjectClient(
    endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    credential=credential,
) as client:
    deployments = list(client.deployments.list())

Client Operations Overview

OperationAccessPurpose
client.agents.agents.*Agent CRUD, versions, threads, runs
client.connections.connections.*List/get project connections
client.deployments.deployments.*List model deployments
client.datasets.datasets.*Dataset management
client.indexes.indexes.*Index management
client.evaluations.evaluations.*Run evaluations
client.red_teams.red_teams.*Red team operations

Two Client Approaches

1. AIProjectClient (Native Foundry)
from azure.ai.projects import AIProjectClient

with AIProjectClient(
    endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
) as client:
    # Use Foundry-native operations
    agent = client.agents.create_agent(
        model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
        name="my-agent",
        instructions="You are helpful.",
    )
2. OpenAI-Compatible Client
# Get OpenAI-compatible client from project
openai_client = client.get_openai_client()

# Use standard OpenAI API
response = openai_client.chat.completions.create(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    messages=[{"role": "user", "content": "Hello!"}],
)

Agent Operations

Create Agent (Basic)
agent = client.agents.create_agent(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    name="my-agent",
    instructions="You are a helpful assistant.",
)
Create Agent with Tools
from azure.ai.agents.models import CodeInterpreterTool, FileSearchTool

agent = client.agents.create_agent(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    name="tool-agent",
    instructions="You can execute code and search files.",
    tools=[CodeInterpreterTool(), FileSearchTool()],
)
Versioned Agents with PromptAgentDefinition
from azure.ai.projects.models import PromptAgentDefinition

# Create a versioned agent
agent_version = client.agents.create_version(
    agent_name="customer-support-agent",
    definition=PromptAgentDefinition(
        model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
        instructions="You are a customer support specialist.",
        tools=[],  # Add tools as needed
    ),
    version_label="v1.0",
)

See references/agents.md for detailed agent patterns.

Tools Overview

ToolClassUse Case
Code InterpreterCodeInterpreterToolExecute Python, generate files
File SearchFileSearchToolRAG over uploaded documents
Bing GroundingBingGroundingToolWeb search (requires connection)
Azure AI SearchAzureAISearchToolSearch your indexes
Function CallingFunctionToolCall your Python functions
OpenAPIOpenApiToolCall REST APIs
MCPMcpToolModel Context Protocol servers
Memory SearchMemorySearchToolSearch agent memory stores
SharePointSharepointGroundingToolSearch SharePoint content

See references/tools.md for all tool patterns.

Thread and Message Flow

# 1. Create thread
thread = client.agents.threads.create()

# 2. Add message
client.agents.messages.create(
    thread_id=thread.id,
    role="user",
    content="What's the weather like?",
)

# 3. Create and process run
run = client.agents.runs.create_and_process(
    thread_id=thread.id,
    agent_id=agent.id,
)

# 4. Get response
if run.status == "completed":
    messages = client.agents.messages.list(thread_id=thread.id)
    for msg in messages:
        if msg.role == "assistant":
            print(msg.content[0].text.value)

Connections

# List all connections
connections = client.connections.list()
for conn in connections:
    print(f"{conn.name}: {conn.connection_type}")

# Get specific connection
connection = client.connections.get(connection_name="my-search-connection")

See references/connections.md for connection patterns.

Deployments

# List available model deployments
deployments = client.deployments.list()
for deployment in deployments:
    print(f"{deployment.name}: {deployment.model}")

See references/deployments.md for deployment patterns.

Datasets and Indexes

# List datasets
datasets = client.datasets.list()

# List indexes
indexes = client.indexes.list()

See references/datasets-indexes.md for data operations.

Evaluation

# Using OpenAI client for evals
openai_client = client.get_openai_client()

# Create evaluation with built-in evaluators
eval_run = openai_client.evals.runs.create(
    eval_id="my-eval",
    name="quality-check",
    data_source={
        "type": "custom",
        "item_references": [{"item_id": "test-1"}],
    },
    testing_criteria=[
        {"type": "fluency"},
        {"type": "task_adherence"},
    ],
)

See references/evaluation.md for evaluation patterns.

Async Client

from azure.ai.projects.aio import AIProjectClient

async with AIProjectClient(
    endpoint=os.environ["AZURE_AI_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
) as client:
    agent = await client.agents.create_agent(...)
    # ... async operations

See references/async-patterns.md for async patterns.

Memory Stores

# Create memory store for agent
memory_store = client.agents.create_memory_store(
    name="conversation-memory",
)

# Attach to agent for persistent memory
agent = client.agents.create_agent(
    model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
    name="memory-agent",
    tools=[MemorySearchTool()],
    tool_resources={"memory": {"store_ids": [memory_store.id]}},
)

Best Practices

  1. Pick sync OR async and stay consistent. Do not mix azure.ai.projects sync clients with azure.ai.projects.aio async clients in the same call path. Choose one mode per module.

  2. Always use context managers for clients and async credentials. Wrap every client in with AIProjectClient(...) as client: (sync) or async with AIProjectClient(...) as client: (async). For async DefaultAzureCredential from azure.identity.aio, also use async with credential: so tokens and transports are cleaned up.

  3. Clean up agents when done: client.agents.delete_agent(agent.id)

  4. Use create_and_process for simple runs, streaming for real-time UX

  5. Use versioned agents for production deployments

  6. Prefer connections for external service integration (AI Search, Bing, etc.)

SDK Comparison

Featureazure-ai-projectsazure-ai-agents
LevelHigh-level (Foundry)Low-level (Agents)
ClientAIProjectClientAgentsClient
Versioningcreate_version()Not available
ConnectionsYesNo
DeploymentsYesNo
Datasets/IndexesYesNo
EvaluationVia OpenAI clientNo
When to useFull Foundry integrationStandalone agent apps

Reference Files

Files

references/custom-evaluators.md

Custom Evaluators Reference

Create custom evaluators when built-in evaluators don't meet your needs using the azure-ai-projects SDK.

Evaluator Types

TypeBest ForRequires LLM
Code-basedPattern matching, format validation, deterministic rulesNo
Prompt-basedSubjective judgment, semantic analysis, nuanced evaluationYes

Code-Based Evaluators

Use Python code for deterministic evaluation logic.

Basic Code Evaluator
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    EvaluatorVersion,
    EvaluatorCategory,
    EvaluatorType,
    CodeBasedEvaluatorDefinition,
    EvaluatorMetric,
    EvaluatorMetricType,
    EvaluatorMetricDirection,
)
from azure.identity import DefaultAzureCredential
import os

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    evaluator = project_client.evaluators.create_version(
        name="word_count_evaluator",
        evaluator_version=EvaluatorVersion(
            evaluator_type=EvaluatorType.CUSTOM,
            categories=[EvaluatorCategory.QUALITY],
            display_name="Word Count",
            description="Counts words in response and checks for conciseness",
            definition=CodeBasedEvaluatorDefinition(
                code_text='''
def grade(sample, item) -> dict:
    response = item.get("response", "")
    word_count = len(response.split())
    return {
        "word_count": word_count,
        "is_concise": word_count < 100
    }
''',
                data_schema={
                    "type": "object",
                    "properties": {
                        "response": {"type": "string"}
                    },
                    "required": ["response"]
                },
                metrics={
                    "word_count": EvaluatorMetric(
                        type=EvaluatorMetricType.ORDINAL,
                        desirable_direction=EvaluatorMetricDirection.DECREASE,
                        min_value=0,
                        max_value=10000,
                    ),
                    "is_concise": EvaluatorMetric(
                        type=EvaluatorMetricType.BINARY,
                    ),
                },
            ),
        ),
    )
    print(f"Created evaluator: {evaluator.name} (version {evaluator.version})")
Code Evaluator: Keyword Checker
evaluator = project_client.evaluators.create_version(
    name="disclaimer_checker",
    evaluator_version=EvaluatorVersion(
        evaluator_type=EvaluatorType.CUSTOM,
        categories=[EvaluatorCategory.QUALITY],
        display_name="Disclaimer Checker",
        description="Verifies required disclaimers are present in response",
        definition=CodeBasedEvaluatorDefinition(
            code_text='''
def grade(sample, item) -> dict:
    response = item.get("response", "").lower()
    required_keywords = ["disclaimer", "not financial advice", "consult a professional"]
    
    found = [kw for kw in required_keywords if kw in response]
    missing = [kw for kw in required_keywords if kw not in response]
    
    score = len(found) / len(required_keywords) if required_keywords else 1.0
    
    return {
        "compliance_score": score,
        "missing_disclaimers": ", ".join(missing) if missing else "none",
        "passes": score >= 0.8
    }
''',
            data_schema={
                "type": "object",
                "properties": {"response": {"type": "string"}},
                "required": ["response"]
            },
            metrics={
                "compliance_score": EvaluatorMetric(
                    type=EvaluatorMetricType.ORDINAL,
                    desirable_direction=EvaluatorMetricDirection.INCREASE,
                    min_value=0.0,
                    max_value=1.0,
                ),
                "passes": EvaluatorMetric(type=EvaluatorMetricType.BINARY),
            },
        ),
    ),
)
Code Evaluator: JSON Format Validator
evaluator = project_client.evaluators.create_version(
    name="json_format_checker",
    evaluator_version=EvaluatorVersion(
        evaluator_type=EvaluatorType.CUSTOM,
        categories=[EvaluatorCategory.QUALITY],
        display_name="JSON Format Validator",
        description="Checks if response is valid JSON with required fields",
        definition=CodeBasedEvaluatorDefinition(
            code_text='''
import json

def grade(sample, item) -> dict:
    response = item.get("response", "")
    required_fields = item.get("required_fields", [])
    
    try:
        parsed = json.loads(response)
        is_valid_json = True
        
        if required_fields:
            missing = [f for f in required_fields if f not in parsed]
            has_required_fields = len(missing) == 0
        else:
            has_required_fields = True
            missing = []
            
    except json.JSONDecodeError:
        is_valid_json = False
        has_required_fields = False
        missing = required_fields
    
    return {
        "is_valid_json": is_valid_json,
        "has_required_fields": has_required_fields,
        "missing_fields": ", ".join(missing) if missing else "none"
    }
''',
            data_schema={
                "type": "object",
                "properties": {
                    "response": {"type": "string"},
                    "required_fields": {"type": "array", "items": {"type": "string"}}
                },
                "required": ["response"]
            },
            metrics={
                "is_valid_json": EvaluatorMetric(type=EvaluatorMetricType.BINARY),
                "has_required_fields": EvaluatorMetric(type=EvaluatorMetricType.BINARY),
            },
        ),
    ),
)

Prompt-Based Evaluators

Use LLM judgment for subjective evaluation.

Basic Prompt Evaluator
from azure.ai.projects.models import PromptBasedEvaluatorDefinition

evaluator = project_client.evaluators.create_version(
    name="helpfulness_evaluator",
    evaluator_version=EvaluatorVersion(
        evaluator_type=EvaluatorType.CUSTOM,
        categories=[EvaluatorCategory.QUALITY],
        display_name="Helpfulness Evaluator",
        description="Evaluates how helpful the response is to the user",
        definition=PromptBasedEvaluatorDefinition(
            prompt_text='''
You are an expert evaluator. Rate the helpfulness of the AI assistant's response.

Query: {query}
Response: {response}

Scoring (1-5):
1 = Not helpful at all, doesn't address the query
2 = Slightly helpful, partially addresses the query
3 = Moderately helpful, addresses most of the query
4 = Very helpful, fully addresses the query
5 = Extremely helpful, exceeds expectations

Return ONLY valid JSON: {"score": <1-5>, "reason": "<brief explanation>"}
''',
            init_parameters={
                "type": "object",
                "properties": {
                    "deployment_name": {"type": "string", "description": "Model deployment name"}
                },
                "required": ["deployment_name"]
            },
            data_schema={
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "response": {"type": "string"}
                },
                "required": ["query", "response"]
            },
            metrics={
                "score": EvaluatorMetric(
                    type=EvaluatorMetricType.ORDINAL,
                    desirable_direction=EvaluatorMetricDirection.INCREASE,
                    min_value=1,
                    max_value=5,
                ),
            },
        ),
    ),
)
Prompt Evaluator: Brand Tone Checker
evaluator = project_client.evaluators.create_version(
    name="brand_tone_checker",
    evaluator_version=EvaluatorVersion(
        evaluator_type=EvaluatorType.CUSTOM,
        categories=[EvaluatorCategory.QUALITY],
        display_name="Brand Tone Checker",
        description="Evaluates if response matches company brand voice guidelines",
        definition=PromptBasedEvaluatorDefinition(
            prompt_text='''
You are evaluating if an AI assistant's response matches brand voice guidelines.

Brand Guidelines:
- Professional but friendly
- Avoid jargon, use simple language
- Always offer next steps or additional help
- Never use negative language about competitors
- End with a helpful call-to-action

Response to evaluate:
{response}

Score the response from 1-5:
5 = Perfectly matches brand voice
4 = Mostly matches, minor issues
3 = Partially matches
2 = Significant tone issues
1 = Does not match brand voice

Return ONLY valid JSON: {"score": <1-5>, "reason": "<brief explanation>", "suggestions": "<improvement suggestions>"}
''',
            init_parameters={
                "type": "object",
                "properties": {"deployment_name": {"type": "string"}},
                "required": ["deployment_name"]
            },
            data_schema={
                "type": "object",
                "properties": {"response": {"type": "string"}},
                "required": ["response"]
            },
            metrics={
                "score": EvaluatorMetric(
                    type=EvaluatorMetricType.ORDINAL,
                    min_value=1,
                    max_value=5,
                ),
            },
        ),
    ),
)
Prompt Evaluator: Factual Accuracy
evaluator = project_client.evaluators.create_version(
    name="factual_accuracy_checker",
    evaluator_version=EvaluatorVersion(
        evaluator_type=EvaluatorType.CUSTOM,
        categories=[EvaluatorCategory.QUALITY],
        display_name="Factual Accuracy",
        description="Checks if response claims are supported by context",
        definition=PromptBasedEvaluatorDefinition(
            prompt_text='''
Evaluate whether the response contains only facts supported by the provided context.

Context (source of truth):
{context}

Response to evaluate:
{response}

Analysis steps:
1. Identify each factual claim in the response
2. Check if each claim is supported by the context
3. Note any unsupported or fabricated claims

Scoring (1-5):
1 = Mostly fabricated or incorrect
2 = Many unsupported claims
3 = Mixed: some facts but notable errors
4 = Mostly factual, minor issues
5 = Fully factual, no unsupported claims

Return ONLY valid JSON: {"score": <1-5>, "reason": "<explanation>", "unsupported_claims": ["<list of unsupported claims>"]}
''',
            init_parameters={
                "type": "object",
                "properties": {"deployment_name": {"type": "string"}},
                "required": ["deployment_name"]
            },
            data_schema={
                "type": "object",
                "properties": {
                    "context": {"type": "string"},
                    "response": {"type": "string"}
                },
                "required": ["context", "response"]
            },
            metrics={
                "score": EvaluatorMetric(
                    type=EvaluatorMetricType.ORDINAL,
                    min_value=1,
                    max_value=5,
                ),
            },
        ),
    ),
)

Using Custom Evaluators

In Testing Criteria
testing_criteria = [
    # Built-in evaluator
    {
        "type": "azure_ai_evaluator",
        "name": "coherence",
        "evaluator_name": "builtin.coherence",
        "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
        "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
    },
    # Custom code-based evaluator
    {
        "type": "azure_ai_evaluator",
        "name": "word_count",
        "evaluator_name": "word_count_evaluator",
        "data_mapping": {"response": "{{item.response}}"}
    },
    # Custom prompt-based evaluator
    {
        "type": "azure_ai_evaluator",
        "name": "helpfulness",
        "evaluator_name": "helpfulness_evaluator",
        "initialization_parameters": {"deployment_name": "gpt-4o-mini"},
        "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
    },
]

eval_object = openai_client.evals.create(
    name="Mixed Evaluators Test",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

Managing Custom Evaluators

List Custom Evaluators
evaluators = project_client.evaluators.list_latest_versions(type="custom")
for e in evaluators:
    print(f"{e.name} (v{e.version}): {e.display_name}")
Get Evaluator Details
evaluator = project_client.evaluators.get_version(
    name="helpfulness_evaluator",
    version="latest"
)
print(f"Data Schema: {evaluator.definition.data_schema}")
print(f"Metrics: {evaluator.definition.metrics}")
Update Evaluator
updated = project_client.evaluators.update_version(
    name="word_count_evaluator",
    version="1",
    evaluator_version={
        "description": "Updated description",
        "display_name": "Word Count v2",
    }
)
Delete Evaluator
project_client.evaluators.delete_version(
    name="word_count_evaluator",
    version="1"
)

Best Practices

  1. Use code-based for deterministic logic - Pattern matching, format validation, keyword checking

  2. Use prompt-based for subjective judgment - Quality assessment, tone evaluation, semantic analysis

  3. Always define data_schema - Ensures correct data mapping

  4. Define meaningful metrics - Use appropriate types (ORDINAL, BINARY)

  5. Test before production - Run evaluator on sample data first

  6. Version your evaluators - Create new versions instead of modifying existing ones

Related Documentation

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Microsoft
Publisher
Microsoft (Official)
Source
microsoft/skills · .github/plugins/azure-sdk-python/skills/azure-ai-projects-py · commit 0696600
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download azure-ai-projects-py.zip