Proving an AI agent follows the rules
I turned scattered rules, tests and requests into a dashboard with a clear information architecture, so auditors can check every call an AI agent makes.
The problem
A collections agency is liable for what Rifa's AI says.
Rifa had to prove compliance with dozens of written rules, on every call, at a volume nobody could review.
Context
Rifa's AI talks to people about their debts, and the agency answers for every word. Nobody can read it all: about 240,000 calls in 30 days.
Each agent follows dozens of written rules, SOPs, so the checking runs as a loop.
Client names are withheld under NDA.
What I shipped
One dashboard, from the rule to the verdict. I designed it in Figma as Rifa's only designer, and the engineering team built it in stages.
What should it do?
The Library: every rule, its tests and the prompts behind them.
Does it do it?
Runs test the library, and each failure names the check it broke.
What did it do?
Every call and chat, with transcript, events and the AI's reasoning.
Was it right?
Passing is one tap. Failing needs a reason and a severity.
One library
Rules, tests and change requests were spread across six tabs, a bell menu and a floating button. I sorted them into three places, one question each.
Rules and tests open in a drawer, not a modal, so the list stays in view.
I designed with real data. One tile reads 78 rules with 77 showing; I left the gap visible, to be fixed at the source.
The voice screens lost three columns I'd invented. The data didn't have them.
329 change requests moved from a bell menu to their own Inbox, each with an owner and a link to its rule.
The agencies push back in the Customer Portal I designed: they raise an issue straight to the engineering team, follow it across the board, then approve or reopen the fix.
I handed off 16 screens in September 2026, and they were built that month.
Reasoning beside chat
A transcript shows what the AI said, not why. So its reasoning and tool calls sit beside the conversation, each call marked with whether it worked.
Reviewers can tell a wrong decision from a failed tool: nearly a quarter of logged failures were platform faults.
Pass, fail, severity
Failing a conversation takes longer than passing it, on purpose. Before, failures couldn't be counted or traced to a cause.
Passing is one click. Failing won't submit without a reason of at least 20 characters and a severity.
An agency's QA reviewer asked to see severity, so I drew two ways: four pills, or a thin bar.
The agency's long QA form became a scorecard: 36 questions marked yes, no or n/a, and a pass or fail.
Designing for model limits
Show the AI's work, including where it might be wrong. The AI flags, people decide: the chats it marks are audited first.
In 30 days it flagged 578 chats. People audited 69%, and 77% of those passed.
Real-time transcripts get words wrong, so a label says so, with a switch to speech-to-text. It stayed an experiment.
My rule for the AI assistants Rifa later added: each inherits its user's permissions and holds no role of its own.
Impact
In daily use, and built from the design. My Figma variables became DESIGN.md, which every pull request is checked against.
- 2,845chats audited by people, 30 days to 3 Nov 2026
- 315issues on the Customer Portal's board, to Nov 2026
- 45releases tracked for clients, Feb–Nov 2026
Roles and permissions were built months later from my annotations, and the code quotes my reasons.
The code never names me. It just says "the designer".
In November 2026 an audit failed a chat where the bot asked a caller on a spouse's phone to verify. Marked Urgent, it was hotfixed the next day.
Reflection
Agree what severity means before designing how it looks.
Define severity first. Rifa's Urgent meant "system unusable"; the agency's reviewers used it for any compliance concern.
Design to the data you have. My first voice screens showed columns the product didn't record. Next time I'd read the live data first.