Azure AI Voicelive Java

MicrosoftOfficialCreated by Microsoft

Skill
Tool integration
Software development
MIT
No issues found

From the author

Azure AI VoiceLive SDK for Java. Real-time bidirectional voice conversations with AI assistants using WebSocket. Triggers: "VoiceLiveClient java", "voice assistant java", "real-time voice java", "audio streaming java", "voice activity detection java".

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The Azure AI VoiceLive SDK for Java provides developers with the tools to build bidirectional voice-enabled AI applications. It leverages WebSocket technology to facilitate real-time interaction between users and AI assistants. The SDK supports various authentication methods, including API keys and DefaultAzureCredential, and offers configurable session options such as Voice Activity Detection (VAD), audio format settings, and support for both OpenAI and Azure-specific voice models.

What it does

Establishes WebSocket connections for bidirectional voice streaming.
Supports server-side Voice Activity Detection (VAD) configuration.
Enables integration with OpenAI and Azure voice models.
Provides mechanisms for function calling within voice sessions.
Handles audio input transcription using Whisper models.
Facilitates event-driven communication for session management.

When to use it

When building Java applications that require real-time voice interaction with AI.
When you need to integrate custom voice models into an AI assistant.
When implementing voice-based function calling in a Java environment.
When you need to manage audio streaming sessions with specific VAD parameters.

Advantages

Native Java support for Azure AI voice services.
Flexible authentication options including managed identities.
Granular control over audio input and output configurations.
Supports reactive programming patterns for event handling.

Usage examples

“How do I initialize the VoiceLiveAsyncClient in Java?”: The client is initialized using the VoiceLiveClientBuilder, providing an endpoint and an AzureKeyCredential or a TokenCredential.
“How can I configure VAD for my voice session?”: You can use the ServerVadTurnDetection class to set thresholds, silence duration, and interruption behavior before sending a session update event.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/azure-ai-voicelive-java/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip azure-ai-voicelive-java.zip -d .agents/skills

Documentation by Microsoft

Azure AI VoiceLive SDK for Java

Real-time, bidirectional voice conversations with AI assistants using WebSocket technology.

Installation

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-voicelive</artifactId>
    <version>1.0.0-beta.2</version>
</dependency>

Environment Variables

AZURE_VOICELIVE_ENDPOINT=https://<resource>.openai.azure.com/ # Required for all auth methods
AZURE_VOICELIVE_API_KEY=<your-api-key> # Only required for AzureKeyCredential auth
AZURE_TOKEN_CREDENTIALS=prod  # Required only if DefaultAzureCredential is used in production

Authentication

API Key

import com.azure.ai.voicelive.VoiceLiveAsyncClient;
import com.azure.ai.voicelive.VoiceLiveClientBuilder;
import com.azure.core.credential.AzureKeyCredential;

VoiceLiveAsyncClient client = new VoiceLiveClientBuilder()
    .endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT"))
    .credential(new AzureKeyCredential(System.getenv("AZURE_VOICELIVE_API_KEY")))
    .buildAsyncClient();

DefaultAzureCredential (Recommended)

import com.azure.core.credential.TokenCredential;
import com.azure.identity.AzureIdentityEnvVars;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.identity.ManagedIdentityCredentialBuilder;

TokenCredential credential = new DefaultAzureCredentialBuilder()
    .requireEnvVars(AzureIdentityEnvVars.AZURE_TOKEN_CREDENTIALS)
    .build();
// Or use a specific credential directly in production:
// See https://learn.microsoft.com/java/api/overview/azure/identity-readme?view=azure-java-stable#credential-classes
// TokenCredential credential = new ManagedIdentityCredentialBuilder().build();

VoiceLiveAsyncClient client = new VoiceLiveClientBuilder()
    .endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT"))
    .credential(credential)
    .buildAsyncClient();

Key Concepts

Concept	Description
`VoiceLiveAsyncClient`	Main entry point for voice sessions
`VoiceLiveSessionAsyncClient`	Active WebSocket connection for streaming
`VoiceLiveSessionOptions`	Configuration for session behavior

Audio Requirements

Sample Rate: 24kHz (24000 Hz)
Bit Depth: 16-bit PCM
Channels: Mono (1 channel)
Format: Signed PCM, little-endian

Core Workflow

1. Start Session

import reactor.core.publisher.Mono;

client.startSession("gpt-4o-realtime-preview")
    .flatMap(session -> {
        System.out.println("Session started");
        
        // Subscribe to events
        session.receiveEvents()
            .subscribe(
                event -> System.out.println("Event: " + event.getType()),
                error -> System.err.println("Error: " + error.getMessage())
            );
        
        return Mono.just(session);
    })
    .block();

2. Configure Session Options

import com.azure.ai.voicelive.models.*;
import java.util.Arrays;

ServerVadTurnDetection turnDetection = new ServerVadTurnDetection()
    .setThreshold(0.5)                    // Sensitivity (0.0-1.0)
    .setPrefixPaddingMs(300)              // Audio before speech
    .setSilenceDurationMs(500)            // Silence to end turn
    .setInterruptResponse(true)           // Allow interruptions
    .setAutoTruncate(true)
    .setCreateResponse(true);

AudioInputTranscriptionOptions transcription = new AudioInputTranscriptionOptions(
    AudioInputTranscriptionOptionsModel.WHISPER_1);

VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
    .setInstructions("You are a helpful AI voice assistant.")
    .setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)))
    .setModalities(Arrays.asList(InteractionModality.TEXT, InteractionModality.AUDIO))
    .setInputAudioFormat(InputAudioFormat.PCM16)
    .setOutputAudioFormat(OutputAudioFormat.PCM16)
    .setInputAudioSamplingRate(24000)
    .setInputAudioNoiseReduction(new AudioNoiseReduction(AudioNoiseReductionType.NEAR_FIELD))
    .setInputAudioEchoCancellation(new AudioEchoCancellation())
    .setInputAudioTranscription(transcription)
    .setTurnDetection(turnDetection);

// Send configuration
ClientEventSessionUpdate updateEvent = new ClientEventSessionUpdate(options);
session.sendEvent(updateEvent).subscribe();

3. Send Audio Input

byte[] audioData = readAudioChunk(); // Your PCM16 audio data
session.sendInputAudio(BinaryData.fromBytes(audioData)).subscribe();

4. Handle Events

session.receiveEvents().subscribe(event -> {
    ServerEventType eventType = event.getType();
    
    if (ServerEventType.SESSION_CREATED.equals(eventType)) {
        System.out.println("Session created");
    } else if (ServerEventType.INPUT_AUDIO_BUFFER_SPEECH_STARTED.equals(eventType)) {
        System.out.println("User started speaking");
    } else if (ServerEventType.INPUT_AUDIO_BUFFER_SPEECH_STOPPED.equals(eventType)) {
        System.out.println("User stopped speaking");
    } else if (ServerEventType.RESPONSE_AUDIO_DELTA.equals(eventType)) {
        if (event instanceof SessionUpdateResponseAudioDelta) {
            SessionUpdateResponseAudioDelta audioEvent = (SessionUpdateResponseAudioDelta) event;
            playAudioChunk(audioEvent.getDelta());
        }
    } else if (ServerEventType.RESPONSE_DONE.equals(eventType)) {
        System.out.println("Response complete");
    } else if (ServerEventType.ERROR.equals(eventType)) {
        if (event instanceof SessionUpdateError) {
            SessionUpdateError errorEvent = (SessionUpdateError) event;
            System.err.println("Error: " + errorEvent.getError().getMessage());
        }
    }
});

Voice Configuration

OpenAI Voices

// Available: ALLOY, ASH, BALLAD, CORAL, ECHO, SAGE, SHIMMER, VERSE
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
    .setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)));

Azure Voices

// Azure Standard Voice
options.setVoice(BinaryData.fromObject(new AzureStandardVoice("en-US-JennyNeural")));

// Azure Custom Voice
options.setVoice(BinaryData.fromObject(new AzureCustomVoice("myVoice", "endpointId")));

// Azure Personal Voice
options.setVoice(BinaryData.fromObject(
    new AzurePersonalVoice("speakerProfileId", PersonalVoiceModels.PHOENIX_LATEST_NEURAL)));

Function Calling

VoiceLiveFunctionDefinition weatherFunction = new VoiceLiveFunctionDefinition("get_weather")
    .setDescription("Get current weather for a location")
    .setParameters(BinaryData.fromObject(parametersSchema));

VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
    .setTools(Arrays.asList(weatherFunction))
    .setInstructions("You have access to weather information.");

Best Practices

Use async client — VoiceLive requires reactive patterns
Configure turn detection for natural conversation flow
Enable noise reduction for better speech recognition
Handle interruptions gracefully with setInterruptResponse(true)
Use Whisper transcription for input audio transcription
Close sessions properly when conversation ends

Error Handling

session.receiveEvents()
    .doOnError(error -> System.err.println("Connection error: " + error.getMessage()))
    .onErrorResume(error -> {
        // Attempt reconnection or cleanup
        return Flux.empty();
    })
    .subscribe();

Reference Links

Resource	URL
GitHub Source	https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-voicelive
Samples	https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-voicelive/src/samples

Files

ATTRIBUTION.mdAdded by AutoGineer976 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md7.9 KiB
references/examples.md24.3 KiB

references/examples.md

Azure AI VoiceLive Java SDK - Examples

Comprehensive code examples for the Azure AI VoiceLive SDK for Java.

Maven Dependency
Client Creation
Session Management
Session Configuration
Audio Streaming
Event Handling
Voice Configuration
Function Calling
Error Handling
Complete Application Example

Maven Dependency

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-voicelive</artifactId>
    <version>1.0.0-beta.2</version>
</dependency>

<!-- For DefaultAzureCredential -->
<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-identity</artifactId>
    <version>1.14.2</version>
</dependency>

<!-- Reactor for reactive streams -->
<dependency>
    <groupId>io.projectreactor</groupId>
    <artifactId>reactor-core</artifactId>
    <version>3.6.0</version>
</dependency>

Client Creation

With API Key

import com.azure.ai.voicelive.VoiceLiveAsyncClient;
import com.azure.ai.voicelive.VoiceLiveClientBuilder;
import com.azure.core.credential.AzureKeyCredential;

String endpoint = System.getenv("AZURE_VOICELIVE_ENDPOINT");
String key = System.getenv("AZURE_VOICELIVE_API_KEY");

VoiceLiveAsyncClient client = new VoiceLiveClientBuilder()
    .endpoint(endpoint)
    .credential(new AzureKeyCredential(key))
    .buildAsyncClient();

With DefaultAzureCredential (Recommended)

import com.azure.identity.DefaultAzureCredentialBuilder;

VoiceLiveAsyncClient client = new VoiceLiveClientBuilder()
    .endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT"))
    .credential(new DefaultAzureCredentialBuilder().build())
    .buildAsyncClient();

Session Management

Start Session

import com.azure.ai.voicelive.VoiceLiveSessionAsyncClient;
import reactor.core.publisher.Mono;

String modelDeployment = "gpt-4o-realtime-preview";

client.startSession(modelDeployment)
    .flatMap(session -> {
        System.out.println("Session started successfully");
        
        // Subscribe to events
        session.receiveEvents()
            .subscribe(
                event -> handleEvent(event),
                error -> System.err.println("Error: " + error.getMessage()),
                () -> System.out.println("Session ended")
            );
        
        return Mono.just(session);
    })
    .subscribe();

Basic Session Flow

client.startSession("gpt-4o-realtime-preview")
    .flatMap(session -> {
        // 1. Configure session
        return configureSession(session)
            .then(Mono.just(session));
    })
    .flatMap(session -> {
        // 2. Start receiving events
        session.receiveEvents()
            .subscribe(event -> processEvent(event));
        
        // 3. Send audio
        return sendAudioStream(session);
    })
    .subscribe(
        result -> System.out.println("Audio sent"),
        error -> System.err.println("Error: " + error.getMessage())
    );

Session Configuration

Configure with Turn Detection

import com.azure.ai.voicelive.models.*;
import com.azure.core.util.BinaryData;
import java.util.Arrays;

// Server-side Voice Activity Detection (VAD)
ServerVadTurnDetection turnDetection = new ServerVadTurnDetection()
    .setThreshold(0.5)                    // Sensitivity (0.0-1.0)
    .setPrefixPaddingMs(300)              // Audio to keep before speech
    .setSilenceDurationMs(500)            // Silence to end turn
    .setInterruptResponse(true)           // Allow user interruptions
    .setAutoTruncate(true)
    .setCreateResponse(true);

// Input transcription
AudioInputTranscriptionOptions transcription = new AudioInputTranscriptionOptions(
    AudioInputTranscriptionOptionsModel.WHISPER_1);

// Session options
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
    .setInstructions("You are a helpful AI voice assistant. Be concise and friendly.")
    .setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)))
    .setModalities(Arrays.asList(InteractionModality.TEXT, InteractionModality.AUDIO))
    .setInputAudioFormat(InputAudioFormat.PCM16)
    .setOutputAudioFormat(OutputAudioFormat.PCM16)
    .setInputAudioSamplingRate(24000)
    .setInputAudioNoiseReduction(new AudioNoiseReduction(AudioNoiseReductionType.NEAR_FIELD))
    .setInputAudioEchoCancellation(new AudioEchoCancellation())
    .setInputAudioTranscription(transcription)
    .setTurnDetection(turnDetection);

// Send configuration
ClientEventSessionUpdate updateEvent = new ClientEventSessionUpdate(options);
session.sendEvent(updateEvent).subscribe();

Configure for Text-Only Response

VoiceLiveSessionOptions textOnlyOptions = new VoiceLiveSessionOptions()
    .setInstructions("You are a helpful assistant.")
    .setModalities(Arrays.asList(InteractionModality.TEXT))  // Text only
    .setInputAudioFormat(InputAudioFormat.PCM16)
    .setInputAudioSamplingRate(24000)
    .setTurnDetection(new ServerVadTurnDetection()
        .setCreateResponse(true));

session.sendEvent(new ClientEventSessionUpdate(textOnlyOptions)).subscribe();

Audio Streaming

Audio Requirements

Sample Rate: 24kHz (24000 Hz)
Bit Depth: 16-bit PCM
Channels: Mono (1 channel)
Format: Signed PCM, little-endian

Send Audio Input

import com.azure.core.util.BinaryData;

// Read audio from microphone or file (PCM16 format)
byte[] audioData = readAudioChunk();  // Your audio data

// Send audio to session
session.sendInputAudio(BinaryData.fromBytes(audioData))
    .subscribe(
        () -> System.out.println("Audio sent"),
        error -> System.err.println("Error sending audio: " + error.getMessage())
    );

Stream Audio from Microphone

import javax.sound.sampled.*;
import java.util.concurrent.atomic.AtomicBoolean;

public void streamFromMicrophone(VoiceLiveSessionAsyncClient session) {
    AtomicBoolean recording = new AtomicBoolean(true);
    
    // Audio format: 24kHz, 16-bit, mono, signed, little-endian
    AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
    
    try {
        DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);
        TargetDataLine microphone = (TargetDataLine) AudioSystem.getLine(info);
        microphone.open(format);
        microphone.start();
        
        // Stream in chunks
        byte[] buffer = new byte[4800];  // 100ms of audio at 24kHz
        
        Thread audioThread = new Thread(() -> {
            while (recording.get()) {
                int bytesRead = microphone.read(buffer, 0, buffer.length);
                if (bytesRead > 0) {
                    byte[] audioChunk = new byte[bytesRead];
                    System.arraycopy(buffer, 0, audioChunk, 0, bytesRead);
                    
                    session.sendInputAudio(BinaryData.fromBytes(audioChunk))
                        .subscribe();
                }
            }
            microphone.close();
        });
        
        audioThread.start();
        
        // Stop after some time or user action
        Thread.sleep(30000);
        recording.set(false);
        
    } catch (Exception e) {
        System.err.println("Error streaming audio: " + e.getMessage());
    }
}

Commit Audio Buffer

Signal end of user input:

session.sendEvent(new ClientEventInputAudioBufferCommit())
    .subscribe();

Clear Audio Buffer

Cancel current input:

session.sendEvent(new ClientEventInputAudioBufferClear())
    .subscribe();

Event Handling

Handle All Event Types

import com.azure.ai.voicelive.models.*;

private void handleEvent(ServerEvent event) {
    ServerEventType eventType = event.getType();
    
    switch (eventType) {
        case SESSION_CREATED:
            handleSessionCreated((SessionCreatedEvent) event);
            break;
            
        case SESSION_UPDATED:
            System.out.println("Session configuration updated");
            break;
            
        case INPUT_AUDIO_BUFFER_SPEECH_STARTED:
            System.out.println("User started speaking");
            break;
            
        case INPUT_AUDIO_BUFFER_SPEECH_STOPPED:
            System.out.println("User stopped speaking");
            break;
            
        case INPUT_AUDIO_BUFFER_COMMITTED:
            System.out.println("Audio buffer committed");
            break;
            
        case CONVERSATION_ITEM_CREATED:
            handleConversationItemCreated(event);
            break;
            
        case RESPONSE_CREATED:
            System.out.println("Response generation started");
            break;
            
        case RESPONSE_AUDIO_DELTA:
            handleAudioDelta(event);
            break;
            
        case RESPONSE_AUDIO_TRANSCRIPT_DELTA:
            handleTranscriptDelta(event);
            break;
            
        case RESPONSE_TEXT_DELTA:
            handleTextDelta(event);
            break;
            
        case RESPONSE_DONE:
            System.out.println("Response complete");
            break;
            
        case ERROR:
            handleError(event);
            break;
            
        default:
            System.out.println("Event: " + eventType);
    }
}

private void handleSessionCreated(SessionCreatedEvent event) {
    System.out.println("Session created: " + event.getSession().getId());
}

private void handleAudioDelta(ServerEvent event) {
    if (event instanceof ResponseAudioDeltaEvent) {
        ResponseAudioDeltaEvent audioEvent = (ResponseAudioDeltaEvent) event;
        byte[] audioData = audioEvent.getDelta();
        
        // Play audio through speakers
        playAudio(audioData);
    }
}

private void handleTranscriptDelta(ServerEvent event) {
    if (event instanceof ResponseAudioTranscriptDeltaEvent) {
        ResponseAudioTranscriptDeltaEvent transcriptEvent = (ResponseAudioTranscriptDeltaEvent) event;
        System.out.print(transcriptEvent.getDelta());  // Print as it arrives
    }
}

private void handleTextDelta(ServerEvent event) {
    if (event instanceof ResponseTextDeltaEvent) {
        ResponseTextDeltaEvent textEvent = (ResponseTextDeltaEvent) event;
        System.out.print(textEvent.getDelta());
    }
}

private void handleError(ServerEvent event) {
    if (event instanceof ErrorEvent) {
        ErrorEvent errorEvent = (ErrorEvent) event;
        System.err.println("Error: " + errorEvent.getError().getMessage());
    }
}

Play Audio Response

import javax.sound.sampled.*;

private SourceDataLine speakerLine;

private void initializeSpeaker() throws LineUnavailableException {
    AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
    DataLine.Info info = new DataLine.Info(SourceDataLine.class, format);
    speakerLine = (SourceDataLine) AudioSystem.getLine(info);
    speakerLine.open(format);
    speakerLine.start();
}

private void playAudio(byte[] audioData) {
    if (speakerLine != null) {
        speakerLine.write(audioData, 0, audioData.length);
    }
}

private void closeSpeaker() {
    if (speakerLine != null) {
        speakerLine.drain();
        speakerLine.close();
    }
}

Voice Configuration

OpenAI Voices

// Available voices: ALLOY, ASH, BALLAD, CORAL, ECHO, SAGE, SHIMMER, VERSE
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
    .setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)));

// Other voices
// OpenAIVoiceName.ASH      - Warm, friendly
// OpenAIVoiceName.BALLAD   - Expressive, dramatic
// OpenAIVoiceName.CORAL    - Clear, professional
// OpenAIVoiceName.ECHO     - Soft, calming
// OpenAIVoiceName.SAGE     - Wise, measured
// OpenAIVoiceName.SHIMMER  - Bright, energetic
// OpenAIVoiceName.VERSE    - Neutral, versatile

Azure Neural Voices

// Azure Standard Voice
AzureStandardVoice standardVoice = new AzureStandardVoice("en-US-JennyNeural");
options.setVoice(BinaryData.fromObject(standardVoice));

// Azure Custom Voice
AzureCustomVoice customVoice = new AzureCustomVoice("myCustomVoice", "endpointId");
options.setVoice(BinaryData.fromObject(customVoice));

// Azure Personal Voice
AzurePersonalVoice personalVoice = new AzurePersonalVoice(
    "speakerProfileId", 
    PersonalVoiceModels.PHOENIX_LATEST_NEURAL
);
options.setVoice(BinaryData.fromObject(personalVoice));

Function Calling

Define Functions

import java.util.Map;

// Define function schema
Map<String, Object> weatherParams = Map.of(
    "type", "object",
    "properties", Map.of(
        "location", Map.of(
            "type", "string",
            "description", "City name, e.g., San Francisco"
        ),
        "unit", Map.of(
            "type", "string",
            "enum", new String[]{"celsius", "fahrenheit"},
            "description", "Temperature unit"
        )
    ),
    "required", new String[]{"location"}
);

VoiceLiveFunctionDefinition weatherFunction = new VoiceLiveFunctionDefinition("get_weather")
    .setDescription("Get current weather for a location")
    .setParameters(BinaryData.fromObject(weatherParams));

// Add to session options
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
    .setInstructions("You are a helpful assistant with access to weather information.")
    .setTools(Arrays.asList(weatherFunction));

Handle Function Calls

private void handleFunctionCall(ServerEvent event) {
    if (event instanceof ResponseFunctionCallArgumentsDoneEvent) {
        ResponseFunctionCallArgumentsDoneEvent funcEvent = 
            (ResponseFunctionCallArgumentsDoneEvent) event;
        
        String functionName = funcEvent.getName();
        String arguments = funcEvent.getArguments();
        String callId = funcEvent.getCallId();
        
        System.out.println("Function call: " + functionName);
        System.out.println("Arguments: " + arguments);
        
        // Execute function and get result
        String result = executeFunction(functionName, arguments);
        
        // Send result back
        ConversationItemFunctionCallOutput output = new ConversationItemFunctionCallOutput(
            callId,
            result
        );
        
        session.sendEvent(new ClientEventConversationItemCreate(output))
            .then(session.sendEvent(new ClientEventResponseCreate()))
            .subscribe();
    }
}

private String executeFunction(String name, String arguments) {
    if ("get_weather".equals(name)) {
        // Parse arguments and call weather API
        return "{\"temperature\": 72, \"condition\": \"sunny\"}";
    }
    return "{\"error\": \"Unknown function\"}";
}

Error Handling

Handle Connection Errors

session.receiveEvents()
    .doOnError(error -> {
        System.err.println("Connection error: " + error.getMessage());
        
        if (error instanceof java.net.SocketException) {
            System.err.println("Network disconnected - attempting reconnect");
            reconnect();
        }
    })
    .onErrorResume(error -> {
        // Log and continue or terminate
        return Flux.empty();
    })
    .subscribe(event -> handleEvent(event));

Handle API Errors

private void handleError(ServerEvent event) {
    if (event instanceof ErrorEvent) {
        ErrorEvent errorEvent = (ErrorEvent) event;
        ErrorDetails error = errorEvent.getError();
        
        System.err.println("Error type: " + error.getType());
        System.err.println("Error message: " + error.getMessage());
        System.err.println("Error code: " + error.getCode());
        
        // Handle specific errors
        String errorType = error.getType();
        if ("invalid_request_error".equals(errorType)) {
            System.err.println("Invalid request - check parameters");
        } else if ("rate_limit_error".equals(errorType)) {
            System.err.println("Rate limited - slow down requests");
        } else if ("server_error".equals(errorType)) {
            System.err.println("Server error - retry later");
        }
    }
}

Complete Application Example

import com.azure.ai.voicelive.VoiceLiveAsyncClient;
import com.azure.ai.voicelive.VoiceLiveClientBuilder;
import com.azure.ai.voicelive.VoiceLiveSessionAsyncClient;
import com.azure.ai.voicelive.models.*;
import com.azure.core.util.BinaryData;
import com.azure.identity.DefaultAzureCredentialBuilder;
import reactor.core.Disposable;

import javax.sound.sampled.*;
import java.util.Arrays;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;

public class VoiceAssistant {
    
    private final VoiceLiveAsyncClient client;
    private VoiceLiveSessionAsyncClient session;
    private SourceDataLine speakerLine;
    private TargetDataLine microphoneLine;
    private AtomicBoolean running = new AtomicBoolean(false);
    private CountDownLatch sessionLatch;
    
    public VoiceAssistant() {
        this.client = new VoiceLiveClientBuilder()
            .endpoint(System.getenv("AZURE_VOICELIVE_ENDPOINT"))
            .credential(new DefaultAzureCredentialBuilder().build())
            .buildAsyncClient();
    }
    
    public void start() throws Exception {
        sessionLatch = new CountDownLatch(1);
        running.set(true);
        
        // Initialize audio
        initializeAudio();
        
        // Start session
        client.startSession("gpt-4o-realtime-preview")
            .doOnNext(s -> {
                this.session = s;
                System.out.println("Session started");
            })
            .flatMap(this::configureSession)
            .subscribe(
                session -> {
                    // Start receiving events
                    session.receiveEvents()
                        .subscribe(
                            this::handleEvent,
                            error -> {
                                System.err.println("Error: " + error.getMessage());
                                stop();
                            },
                            () -> {
                                System.out.println("Session ended");
                                sessionLatch.countDown();
                            }
                        );
                    
                    // Start microphone streaming
                    startMicrophoneStream();
                },
                error -> {
                    System.err.println("Failed to start session: " + error.getMessage());
                    sessionLatch.countDown();
                }
            );
        
        System.out.println("Voice assistant started. Speak into the microphone...");
        System.out.println("Press Enter to stop.");
        
        // Wait for user to press Enter
        System.in.read();
        stop();
        
        sessionLatch.await();
    }
    
    private reactor.core.publisher.Mono<VoiceLiveSessionAsyncClient> configureSession(
            VoiceLiveSessionAsyncClient session) {
        
        ServerVadTurnDetection turnDetection = new ServerVadTurnDetection()
            .setThreshold(0.5)
            .setPrefixPaddingMs(300)
            .setSilenceDurationMs(500)
            .setInterruptResponse(true)
            .setAutoTruncate(true)
            .setCreateResponse(true);
        
        VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
            .setInstructions("You are a helpful voice assistant. Keep responses concise.")
            .setVoice(BinaryData.fromObject(new OpenAIVoice(OpenAIVoiceName.ALLOY)))
            .setModalities(Arrays.asList(InteractionModality.TEXT, InteractionModality.AUDIO))
            .setInputAudioFormat(InputAudioFormat.PCM16)
            .setOutputAudioFormat(OutputAudioFormat.PCM16)
            .setInputAudioSamplingRate(24000)
            .setInputAudioTranscription(new AudioInputTranscriptionOptions(
                AudioInputTranscriptionOptionsModel.WHISPER_1))
            .setTurnDetection(turnDetection);
        
        return session.sendEvent(new ClientEventSessionUpdate(options))
            .thenReturn(session);
    }
    
    private void initializeAudio() throws LineUnavailableException {
        AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
        
        // Speaker
        DataLine.Info speakerInfo = new DataLine.Info(SourceDataLine.class, format);
        speakerLine = (SourceDataLine) AudioSystem.getLine(speakerInfo);
        speakerLine.open(format);
        speakerLine.start();
        
        // Microphone
        DataLine.Info micInfo = new DataLine.Info(TargetDataLine.class, format);
        microphoneLine = (TargetDataLine) AudioSystem.getLine(micInfo);
        microphoneLine.open(format);
        microphoneLine.start();
    }
    
    private void startMicrophoneStream() {
        Thread micThread = new Thread(() -> {
            byte[] buffer = new byte[4800];  // 100ms chunks
            
            while (running.get()) {
                int bytesRead = microphoneLine.read(buffer, 0, buffer.length);
                if (bytesRead > 0 && session != null) {
                    byte[] chunk = new byte[bytesRead];
                    System.arraycopy(buffer, 0, chunk, 0, bytesRead);
                    session.sendInputAudio(BinaryData.fromBytes(chunk)).subscribe();
                }
            }
        });
        micThread.setDaemon(true);
        micThread.start();
    }
    
    private void handleEvent(ServerEvent event) {
        switch (event.getType()) {
            case INPUT_AUDIO_BUFFER_SPEECH_STARTED:
                System.out.println("\n[You are speaking...]");
                break;
                
            case INPUT_AUDIO_BUFFER_SPEECH_STOPPED:
                System.out.println("[Processing...]");
                break;
                
            case RESPONSE_AUDIO_DELTA:
                if (event instanceof ResponseAudioDeltaEvent) {
                    byte[] audio = ((ResponseAudioDeltaEvent) event).getDelta();
                    speakerLine.write(audio, 0, audio.length);
                }
                break;
                
            case RESPONSE_AUDIO_TRANSCRIPT_DELTA:
                if (event instanceof ResponseAudioTranscriptDeltaEvent) {
                    System.out.print(((ResponseAudioTranscriptDeltaEvent) event).getDelta());
                }
                break;
                
            case RESPONSE_DONE:
                System.out.println("\n");
                break;
                
            case ERROR:
                if (event instanceof ErrorEvent) {
                    System.err.println("Error: " + ((ErrorEvent) event).getError().getMessage());
                }
                break;
                
            default:
                // Ignore other events
                break;
        }
    }
    
    public void stop() {
        running.set(false);
        
        if (microphoneLine != null) {
            microphoneLine.stop();
            microphoneLine.close();
        }
        
        if (speakerLine != null) {
            speakerLine.drain();
            speakerLine.stop();
            speakerLine.close();
        }
        
        System.out.println("Voice assistant stopped");
    }
    
    public static void main(String[] args) {
        try {
            VoiceAssistant assistant = new VoiceAssistant();
            assistant.start();
        } catch (Exception e) {
            System.err.println("Error: " + e.getMessage());
            e.printStackTrace();
        }
    }
}

Environment Variables

AZURE_VOICELIVE_ENDPOINT=https://<resource>.openai.azure.com/
AZURE_VOICELIVE_API_KEY=<your-api-key>

# For DefaultAzureCredential
AZURE_CLIENT_ID=<service-principal-client-id>
AZURE_CLIENT_SECRET=<service-principal-secret>
AZURE_TENANT_ID=<tenant-id>

Best Practices

Use async client — VoiceLive requires reactive patterns for real-time streaming
Configure turn detection — Tune VAD settings for natural conversation flow
Enable noise reduction — Use AudioNoiseReduction for better speech recognition
Handle interruptions — Set setInterruptResponse(true) for natural conversations
Use Whisper transcription — Enable for input audio transcription
Close sessions properly — Clean up resources when conversation ends
Buffer audio appropriately — Send audio in ~100ms chunks for optimal latency
Implement error recovery — Handle connection drops with reconnection logic

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills · .github/plugins/azure-sdk-java/skills/azure-ai-voicelive-java · commit 0696600

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download azure-ai-voicelive-java.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Microsoft

Azure AI VoiceLive SDK for Java

Installation

Environment Variables

Authentication

API Key

DefaultAzureCredential (Recommended)

Key Concepts

Audio Requirements

Core Workflow

1. Start Session

2. Configure Session Options

3. Send Audio Input

4. Handle Events

Voice Configuration

OpenAI Voices

Azure Voices

Function Calling

Best Practices

Error Handling

Reference Links

Files

references/examples.md

Azure AI VoiceLive Java SDK - Examples

Table of Contents

Maven Dependency

Client Creation

With API Key

With DefaultAzureCredential (Recommended)

Session Management

Start Session

Basic Session Flow

Session Configuration

Configure with Turn Detection

Configure for Text-Only Response

Audio Streaming

Audio Requirements

Send Audio Input

Stream Audio from Microphone

Commit Audio Buffer

Clear Audio Buffer

Event Handling

Handle All Event Types

Play Audio Response

Voice Configuration

OpenAI Voices

Azure Neural Voices

Function Calling

Define Functions

Handle Function Calls

Error Handling

Handle Connection Errors

Handle API Errors

Complete Application Example

Environment Variables

Best Practices

Requirements and compatibility

Security

Attribution and licence

Related skills

Implement Spec

MCP Builder

Schema

Postgresql Table Design

SQL Optimization Patterns

API Scaffolding Fastapi Pro

Download ZIP