Bitter Lessons

As an on-call engineer or SRE we receive automated phone calls simply informing us something is wrong, and we get out of bed at 2am and begin digging in. What if that first touchpoint was far more powerful? That phone call should have tons of context and can be used to initially triage.

We have a working demo of receiving alerts that immediately can be conversed with to dig into the issue. It also can go off and use reasoning to figure out what information it needs for a root cause analysis and can search the service graph by itself simply by knowing what APIs to call.

We think this can substantially improve the life of someone on-call and save valuable minutes that are currently used for context aggregation during an incident. We use Datadog in this demo, but created scaffolding that can be used by any monitoring tools (reference any APIs from Sentry, NewRelic, etc.)