Testing LLMs in Web Applications

I want to talk about testing LLMs in production web applications. There are a lot of ways to integrate your LLMs into apps: REST API requests, SDKs, etc. but not a lot of easy-to-use tooling to make testing straightforward and high-signal. I’m collaborating with my MIT roommate on a unit testing library - Poyro - that makes it dead-simple to test LLMs that work as part of Node-based web applications (e.g. Next.js, Express, etc.). I wanted to share how we’re thinking about testing and how other web app engineers might as well.