MirrorMatch

MirrorMatch is a real-time emotional intelligence layer for AI voice agents. It sits alongside a live customer service call and gives the agent a continuous read on the caller’s emotional state — updated on every turn — then tells them exactly what to do about it, out loud.

What it does: You play the angry customer. A Tavus AI avatar plays the customer service agent trying to de-escalate you. MirrorMatch listens to every utterance, classifies your emotion in real time, drives a live coaching signal, and speaks urgent guidance aloud when the call escalates.

EMOTIONAL ACCURACY
Every spoken turn is transcribed via the Web Speech API and classified by HuggingFace’s j-hartmann/emotion-english-distilroberta-base — a DistilRoBERTa model fine-tuned on 7 emotion labels (anger, joy, sadness, disgust, fear, neutral, surprise). Scores feed live bars carrying both a psychological label and a business KPI: “Anger · Escalation Risk”, “Joy · Satisfaction”. A keyword heuristic fallback guarantees a classification even when the model API is cold.

REAL-TIME ADAPTATION
A Live Coaching Signal panel updates every spoken turn — Level 0 (maintain flow) → Level 1 (empathize) → Level 2 (resolve concretely now) → Level 3 (route to human). The EmotionOrb visualizes intensity continuously: color, glow radius, and pulse speed (2.5s calm → 0.6s critical) all encode emotion. When the level rises, ElevenLabs speaks the coaching action aloud so the agent never has to look away from the customer.

MULTILINGUAL CAPABILITY
ElevenLabs TTS uses eleven_multilingual_v2 (29 languages) with voice stability modulated by emotion — higher anger lowers stability for more urgent delivery. The DistilRoBERTa classifier generalizes across Latin-script languages, and the Tavus CVI avatar supports multilingual personas out of the box.

CREATIVE EXPRESSION
A live Emotion Arc chart (Recharts) builds across the call — anger, frustration, neutral, and joy as a multi-line emotional trajectory. The EmotionOrb reacts to every turn, and ElevenLabs voices coaching cues for a fully multimodal experience.

REDIS — hot path and rate limiting
Redis (Upstash) is the real-time backbone. The emotion arc is a sorted set keyed by timestamp, peak anger is a ZREVRANGE lookup, and turns and escalations are atomic INCR counters read back into a live “REDIS · N turns · M escalations · peak X%” chip in the dashboard header. Redis also powers per-IP rate limiting on every API endpoint — 40 emotion classifications per minute and 3 Tavus sessions per minute per IP — using the same atomic counter pattern. Hot path, live readback, and rate control all in one store.

SUPABASE — durable archive
Supabase persists every session and turn permanently for post-call QA replay and SQL queries across historical calls. Redis expires after an hour; Supabase keeps everything.

TECH STACK

• Tavus CVI (/v2/conversations iframe — live AI avatar agent)
• ElevenLabs TTS (eleven_multilingual_v2, emotion-modulated stability, spoken coaching alerts)
• Redis / Upstash (sorted-set emotion arc, atomic counters, live state, rate limiting, readback)
• HuggingFace Inference API (j-hartmann/emotion-english-distilroberta-base + keyword fallback)
• Web Speech API (continuous real-time transcription with interim streaming)
• Supabase (durable session + turn archive)
• Next.js 14 App Router + TypeScript + Tailwind (fully serverless, no backend process)
• Recharts (live Emotion Arc) · Vercel (deployment)

None. All code written during the hackathon.