How I Built AVA: A Talking-Avatar AI Agent in Azure Foundry

One of the most-asked questions about my site is: “How did you put a real, talking AI person on a web page?” This is the full build. AVA is my technical-project-intake agent: a Microsoft Foundry agent running gpt-5.4, given a face through Azure Speech’s text-to-speech avatar, a voice through real-time speech, and a home inside a single HTML page. Below is the exact prompt I use to generate the whole thing in VS Code, plus the use cases, value, limits, licensing, and capacity realities a developer needs before shipping their own.

The complete prompt (paste this into VS Code)

This is a single end-to-end prompt for an agentic coding assistant (GitHub Copilot, Cursor, etc.). It does two jobs: it defines the agent in Azure AI Foundry, and it generates the front-end page that embeds the avatar. Replace the bracketed values with your own.

You are my senior Azure + front-end engineer. Build "AVA", a talking-avatar
intake agent, end to end. Work in this repo and create real files.

PART A — Create the agent in Azure AI Foundry
1. Target an existing Azure AI Foundry project in resource group [RG] and region
   [eastus2]. Use the deployed chat model "gpt-5.4". If it is not deployed, give
   me the exact `az cognitiveservices account deployment create` command.
2. Create an agent named "AVA" with these instructions:
   """
   You are AVA, a professional technical project intake specialist.
   Your job is to speak with customers and collect detailed requirements for
   technical software and AI projects. Ask one or two questions at a time. Be
   friendly, organized, and precise. Capture: business goal, users, scope,
   must-have features, integrations, data sources, security and compliance
   needs, target platforms, timeline, and budget range. Confirm each section
   back to the user before moving on. Never invent requirements; ask instead.
   End by summarizing a structured intake brief the user can copy.
   """
3. Enable the "Grounding with Bing Search" tool so AVA can look up current facts.
4. Set temperature 0.5, top_p 1. Save and publish a version.

PART B — Give the agent a face and a voice
5. Use Azure Speech (resource [SPEECH_NAME], region [eastus2], SKU S0) for
   real-time text-to-speech AVATAR synthesis (character "meg", style "business")
   and for speech-to-text in voice mode.
6. The browser must NEVER hold the Speech key. Create a backend token endpoint
   `/api/avatar-token` (Azure Static Web Apps managed Function, Node 18) that:
     - reads AZURE_SPEECH_KEY and AZURE_SPEECH_REGION from app settings,
     - calls the avatar relay token endpoint, and
     - returns { token, region, relay } as JSON.
   Create `/api/chat` that proxies to the Foundry/Azure OpenAI gpt-5.4 deployment
   using AZURE_AI_ENDPOINT + AZURE_AI_KEY + AZURE_AI_DEPLOYMENT, server-side only.

PART C — Embed AVA in a web page
7. Create `ava-agent.html`. Load the Azure Speech SDK
   (https://aka.ms/csspeech/jsbrowserpackageraw). On "Start":
     - fetch `/api/avatar-token`, open a WebRTC peer connection to the relay,
       and render the live avatar video into a <video> element;
     - in voice mode, stream mic audio to STT, send the transcript to `/api/chat`,
       speak the reply through the avatar; in text mode, do the same from a textbox.
   8. Add idle auto-stop after [60000] ms to release the avatar session.
   9. Keep page load fast: defer all SDK and avatar work until the user clicks Start.

Output: every file, the az CLI commands, and the exact Static Web Apps app
settings I must set. Do not hardcode any secret in client code.

How the embed actually works

There is no magic “avatar widget.” The talking head is a live WebRTC video stream synthesized by Azure Speech in real time. The page flow is:

1. User clicks Start → the page calls /api/avatar-token.
2. The Function returns a relay token + ICE server info; the browser opens a peer connection.
3. The Speech avatar service streams video/audio of the character speaking.
4. User input (typed or spoken) → /api/chat → gpt-5.4 reply → sent to the avatar to speak.

If you only want the simplest possible embed and do not need a custom UI, Foundry also gives every published agent a hosted preview URL you can drop into an <iframe>. That is the 10-minute version. The custom WebRTC build above is what gives you brand control, voice mode, idle management, and no Foundry chrome.

Use cases

• Project / sales intake — exactly what AVA does: structured requirements gathering that never forgets a question.
• Reception & triage — a front-door agent that routes visitors, books time, or answers FAQs by voice.
• Training & onboarding — a consistent, patient explainer that walks new hires or customers through a process.
• Accessibility — a spoken, face-to-face interface for users who prefer talking over typing.
• Kiosks & events — a branded presenter on a screen at a booth, lobby, or store.

Value proposition

A talking avatar collapses the distance between a form and a conversation. People disclose more, and more naturally, to a face than to a text box. You get the warmth of a human intake call with the consistency, availability, and infinite patience of software — and because the brain is a Foundry agent, the same back end can power chat, voice, and avatar from one source of truth. For an intake use case specifically, the payoff is clean, structured data captured on the first pass instead of three follow-up emails.

Limitations (be honest with stakeholders)

• Latency. Model thinking time + speech synthesis + network means a half-second-plus before AVA replies. It feels conversational, not instantaneous.
• Autoplay & permissions. Browsers require a user gesture to start audio and microphone access. The avatar must start on a click, never on page load.
• Avatar realism. The synthesized characters are good, not photoreal, and the library of characters/styles is fixed unless you build a custom avatar (a separate, gated program).
• Region availability. The real-time avatar is only offered in select Speech regions. Build where it is supported (I use eastus2).
• Cost scales with talk time. Avatar synthesis is billed by the second of streamed video, so idle auto-stop matters.
• Not for sensitive data without review. Grounding with Bing sends queries outside your compliance boundary; gate it for regulated workloads.

Microsoft licensing & resources you need

This is Azure consumption, not an M365/Copilot seat license. You are paying for cloud resources, not per-user subscriptions. The shopping list:

• Azure subscription with rights to create AI + Speech resources.
• Azure AI Foundry project + a gpt-5.4 (or comparable) chat model deployment. Billed per 1K input/output tokens.
• Azure AI Speech resource, SKU S0 (Standard) — the free F0 tier does not include real-time avatar. Billed for TTS avatar (per second), neural TTS (per character), and STT (per second/hour).
• Grounding with Bing Search resource (optional) if you want live web facts. Billed per query.
• Azure Static Web Apps (free or standard) to host the page and the token/chat Functions.

Capacity & quota realities

• Model TPM quota. Your Foundry deployment has a tokens-per-minute ceiling. Concurrent voice users burn through it quickly; request a quota increase before a launch.
• Avatar concurrency. Real-time avatar has a per-resource cap on simultaneous sessions. Plan sessions, and release them on idle.
• Region capacity. Newer models and the avatar feature are not in every region; pick a region that has both, in the same geo, to keep latency down.
• Cold starts. Static Web Apps managed Functions can cold-start; the first token request after idle is slightly slower.

The bottom line

A talking-avatar agent is not one product you buy; it is three Azure capabilities — a reasoning model, a speech avatar, and a thin secure backend — stitched into one page. Get the security model right (no keys in the browser), respect the autoplay and latency realities, and watch your Speech and token quotas. Do that, and you can put a genuinely useful AI person on any web page. You can meet mine on the AVA page.