Shipping an AI agent you can trust
I simplified how teams see an AI agent's rules: a Library for every SOP and test case, and a stage-by-stage canvas in place of the engineering team's hairball graph.
The problem
An agent's rules lived in YAML files only engineers could read, and their graph of them was a hairball nobody else could follow.
Context
Only the engineers could see the rules. Rifa's agents follow 80 to 90 SOPs at once, and they lived in YAML files the tech team kept.
Rifa's agents call and chat with people about debts they owe, for US collections agencies, under strict law. Each agent follows SOPs: written rules for each situation.
Rules that interact
80 to 90 SOPs run at once, and some override others.
Testers aren't engineers
QA and operations people check the agent, not the code.
Behind it, the engineers kept the structure in YAML files, and their own picture of it was a graph of everything at once.
QA couldn't answer the question that mattered most: why did this call break here? They asked engineering in Slack, or on a call.
What I shipped
The Library: everything in one place. Every SOP, test case, prompt and test run, instead of files only the tech team opened.
Anyone could see what each rule says, which tests check it, and how the last runs went.
Three questions
The hairball was three questions drawn at once: what comes next, which rule wins in a tie, and what can interrupt.
Only the first belongs on the canvas. Which rule wins moved to each SOP's own page, and interruptions became a layer you switch on.
The call by stage
The call, stage by stage. Four columns in the order a person follows a call, from greeting to payment link.
Greeting and verification always sit in their stage. Select a path, and every SOP it touches lights up.
Select one SOP, and the canvas shows what it touches: the next step, what can interrupt it, and what is always on.
Interrupts as a layer
Compliance can cut in anywhere, so it's a layer you switch on, not more lines on the map.
Switch on interrupts, and the compliance-sensitive SOPs that can cut in from any stage appear: bankruptcy, cease communication, an attorney, a dispute, a death.
Trade-off: the canvas is for reading. People change SOPs in the Library, and the canvas follows.
Designing for model limits
The agent gets things wrong, so people need to see where. Testing caught real failures like these.
A death misread as the account holder's. A mention of consolidating debts read as credit counselling. Each one is a rule firing where it shouldn't, and the canvas and Library are where people find which one.
Across every client, in 30 days, the AI flagged 578 chats; people audited 398, and 90 of those failed.
Impact
A library people use, and a map the team chose. Agent Studio is still in development.
Shipped
The Library: SOPs, test cases, prompts and runs in one place.
Chosen
In June the team chose the stage canvas over the hairball.
Next
By October engineers were building it to follow a live call.
Reflection
Show one question at a time.
Draw the questions, not the data. The hairball held every fact and answered nothing. Splitting it by question made it readable.
Hear the testers myself. Their needs reached me secondhand. Next time I'd sit in.