Information hierarchy
The results screen has to make 192 calls, 12 findings, 6 modules, 4 strategies, and a score on every row readable at a glance.
Case Study · Take-home exercise
Probe | Adversarial testing for voice agents
A one-week take-home for an AI-security startup. The brief asked for two screens: scan setup and results. I designed those two, and ten more states around them: running, interrupted runs, evidence review, export and retest.
Risk by module / strategy
Detail · results screen
DirectCrescendoEncodedRoleplay
Access control
3 / 8
7 / 8
4 / 8
4 / 8
Tool misuse
3 / 8
5 / 8
4 / 8
3 / 8
Identity
4 / 8
6 / 8
2 / 8
4 / 8
Goal manipulation
2 / 8
4 / 8
3 / 8
3 / 8
Memory
0 / 8
0 / 8
0 / 8
0 / 8
Untraceability
0 / 8
0 / 8
0 / 8
0 / 8
The brief
Two screens requested, four areas graded
The product places hundreds of adversarial phone calls against a customer’s voice AI agent and reports where the agent broke. The exercise was one screen to configure a scan and one to read the results.
The brief also named four focus areas. Only the first one sits inside those two screens. The other three shaped the states around them.
Orientation within the workflow
A scan runs for about 40 minutes. The two requested screens did not show what happens during that time.
Error handling and status communication
A run that stops at call 60 of 192 has already found things. Those findings need their own screen.
Research into how security teams think
Analysts discard alerts they cannot explain. That applies to every row in the findings table.
AI as a design method
How I used AI
Several AI agents got the same brief, each arguing a different position: the product logic, the evidence, or the case study. They wrote into one shared review file and answered each other. Three arguments changed the work.
The opening claim
One agent wanted a stronger opening claim. Another showed it went past the brief. The hero now names the two requested screens and the states I added.
The denominator
The percentage was clear, the denominator was not. An attempt is now a call that reached the behaviour under test. I rechecked every total against it.
How many screens to annotate
The first pass annotated too many screens. Running reads as a thumbnail, Partial results and Evidence do not. Five screens kept their notes.
Scope
Twelve states, two of them requested
Twelve states shipped. Two were requested. The other ten cover setup conflicts, progress, failure, evidence, export and retest.
-
01
Setup
Choosing what to test, and what it will cost.
-
02
Running
What happens during the roughly 40-minute scan, including interrupted runs.
-
03
Results
What the run can claim, and what different readers need from it.
-
04
Evidence
How a reviewer checks the recording and transcript without exposing sensitive data by default.
-
05
Retest
Whether the fix held, measured against a comparable run.
The requested screens
Configure a scan, then read one
New scan · the form has to price the run
The estimate updates with the configuration
Six modules × four strategies × eight variations = 192 calls. The panel updates with the configuration and shows call count and estimated runtime.
Live scans require a safety review
A Live scan places real calls to a production number and costs money. Selecting it opens a safety review.
Coverage is stated before the run
The header shows that 6 of 10 modules are covered, so exclusions are visible before the scan begins.
Turns depend on strategy
Maximum test turns only applies to multi-turn strategies. The field is linked to the matrix, and both update the estimate.
Scan results · a verdict, then the evidence
The headline is one sentence
The screen opens with “The agent broke in 4 of the 6 attack modules tested”. Four KPI cards of equal weight would leave the reader to work out which number matters.
Severity and frequency answer different questions
A Critical that breached once in 40 attempts and a High that breached 38 times are different problems. Every row shows both. An attempt is a call that reached the behaviour under test, so the denominator changes per finding.
The matrix compares modules with strategies
Run order is an operational detail. The matrix shows which attack strategies were effective against each module.
Severity does not rely on colour alone
Each level uses a letter, a rank bar, and a text label, so colour is not the only cue.
The states nobody asked for
The brief did not cover the 40 minutes between setup and results
Partial results · no comparison without comparable coverage
The findings survive the failure
60 of 192 calls completed. Two findings, valid for the two modules that were reached. Resume continues from call 60.
No delta without comparable coverage
Comparison to the previous run shows a dash and the label “not comparable, partial coverage”. A numeric delta would imply evidence the scan did not collect.
Unreached modules are listed by name
The four modules the run never reached are listed by name and marked as untested. The report does not count them as clean.
No findings · scoped to tested coverage
Clean within the tested coverage
Six of ten modules ran. The headline and the empty state both state the coverage, so a zero is not read as a clean result for all ten.
The empty state suggests the next test
The empty state offers three next actions: add the four untested modules, raise the limit to 16 turns, or schedule a weekly scan.
Evidence · recording, transcript, and score inputs
How the score was calculated
The drawer shows the 9.0 with its inputs: CVSS base 7.4, agentic factor sum 6.5 of 10, threat multiplier 0.97, mitigation factor 1.0, AARS +1.6. AIVSS is a draft at v0.8, so the inputs stay visible.
One click to the breach moment
The waveform opens at the relevant four-second segment, so the analyst can verify the finding without scanning the full recording.
Redaction is on by default
Customer names are masked in the transcript and bleeped in the audio. Revealing them requires a deliberate action and creates a record.
Research
Four research findings that changed the interface
What analysts need to validate a finding
Evidence20 SOC practitioners surveyed, 21 more interviewed. Many alarms filed as false positives are benign triggers: real detections caused by legitimate behaviour. Five properties make validation faster: reliable, explainable, analytical, contextual, transferable.
In the interfaceEvery finding shows its module, strategy, prompt, reply and attempt count. A review action lets an analyst challenge the model’s verdict.
CVSS Base is only the starting point
EvidenceCVSS v4.0 separates Base from Threat and Environmental metrics. Regulatory exposure, customers affected and monetary loss fall outside CVSS. The score is an input to prioritisation.
In the interfaceSeverity and observed breach frequency appear together on every row. The screen opens with what broke. The score comes after.
A draft scoring model needs visible inputs
EvidenceAIVSS v0.8 combines a CVSS v4.0 base with an agentic factor sum, threat multiplier and mitigation factor to produce AARS.
In the interfaceThe drawer shows the CVSS base and every adjustment used to reach AARS, so a reviewer can check the calculation.
Keep the evidence beside the finding
EvidenceThe same study describes alarm validation as tedious work that leads to burnout and then to desensitisation. Contextual and transferable are two of the five properties that shorten it.
In the interfaceThe finding, audio, transcript, remediation and retest stay in one drawer over the list, so reviewers keep their place in the table.
What this does not prove
No user was tested
No usability session was run. Every user claim on this page is an assumption. The ledger records each one and what would change it.
The research is indirect
The false-positive and triage studies cover security operations centres in general, not this product’s users. I used them as directional evidence.
Desktop at 1440 only
No responsive work was done. All data is fictional, held in one source file so the twelve states stay consistent.
| Assumed user | What it decided | What would change the decision |
|---|---|---|
| An AI red-teamer runs the scans and reads the findings | Density over guidance. No wizard, no coaching copy, six modules on one screen. | If setup is run by a customer-success owner, the form needs a guided mode and the matrix has to collapse. |
| A conversation designer owns the fix | Remediation is written as a change to the agent, not as a code patch. | If fixes ship from a ticket by engineers, remediation belongs in the export, not in the drawer. |
| A CISO signs off on Live runs | Live requires four settings, a typed confirmation and a named approver. | If Live runs nightly in CI, the safety review is friction and belongs in a policy setting instead. |