Emmet
Emmet lets users have interview-style conversations with the experts in science and math about their papers and save them as videos.
YouTube Video
Project Description
Emmet is a platform to learn complex technical content and teach it to others. Users upload scientific papers and engage a conversational AI in learning about them and generating informative visualizations. We also want to add a feature to export chat sessions as videos.
Our code is fully functional with sub-second responses, our project is creative, and we think it has major world impact given the number of SWEs who need to learn more ML, quickly. It aligns with the theme of the hackathon by using the Realtime API, two fine-tuned models for the reasoning required to generate SVG illustrations, and a custom computer vision model to extract data from research papers. We introduce reasoning using a two-step CoT to generate SVG visualizations.
Technologies:
- OpenAI Realtime API for the conversation engine over WebRTC
- Finetuned OpenAI models for SVG generation
- React with Shadcn for the frontend
- Clojure for the backend
- Docker for managing postgres and S3
- Grobid for parsing research papers
Prior Work
We started with an idea and two empty starter repos.