Children of the Fields

We put MCP Apps in a WebXR application, so for the first time, Claude has the ability to alter mixed reality environments.

This allows communication with to an AI model via speech, that helps you display the status of NYC subways and Citi Bike stations in augmented reality.

All of the apps are MCP Apps — dynamically generated at runtime, per each user request.

Note, my voice was not picked up in the demo video, but the LLM is prompted with speech. The Quest 3 just doesn’t pick my microphone up when screen recording.

All code was written during the hackathon.

We used Deepgram for speech to text, Eleven Labs for text to speech, and Anthropic for LLM inferences/tool calls.