← All issues

Issue #1 · April 14, 2026

Nobody Is Testing AI Systems Properly

Voice agents fail under real speech, LLM judges can't be trusted, and your eval suite has blind spots you don't know about. 7 papers that prove QA for AI is broken.

This Week in AI Systems

1. Voice agents collapse under real human speech

Reality: Full-Duplex-Bench-v3 tested voice agents with stutters, corrections, interruptions, and overlapping speech — the way humans actually talk. Every single agent degraded sharply. The models that score well on clean benchmarks fall apart when someone says "uh, wait, actually no — go back."

What this means: You're shipping voice agents tested on clean audio. Your users don't speak in clean audio. Your QA is blind to the most common failure mode.

What to do:

  • Add disfluent speech to every voice agent test suite — stutters, corrections, interruptions
  • Record real user calls (with consent) and replay them as regression tests
  • Stop using clean TTS-generated audio as your test input

2. ASR benchmarks are lying to you

Reality: ASR has "near-human accuracy" on curated benchmarks. But a new study shows production voice agents still fail badly on accented speech, background noise, cross-talk, and domain-specific terms. Benchmark scores drastically overestimate how well your system works in the real world.

What this means: Your ASR provider's accuracy number is meaningless for your use case. The gap between benchmark and production is wider than most teams realize.

What to do:

  • Build your own eval set from real production audio, not vendor benchmarks
  • Test ASR specifically on your domain vocabulary (medical terms, product names, jargon)
  • Track word error rate on production traffic weekly — not just at integration time

3. LLM-as-judge is unreliable — and you're not running enough evals

Reality: Teams using LLM judges to evaluate conversational AI are drastically under-sampling. Judge reliability follows power-law scaling — you need far more evaluation runs than you think to get stable scores. Most teams run 1x. They need 5-10x minimum.

What this means: Your automated eval pipeline gives you a false sense of confidence. The scores fluctuate more than you know.

What to do:

  • Run every LLM judge evaluation at least 5x and check variance
  • Track inter-run agreement as a first-class metric
  • If agreement is below 80%, your eval is noise — don't ship based on it

4. Single-method safety testing misses entire failure classes

Reality: Safety evaluations almost always test one attack surface — usually prompt-based. But a new paper shows that prompt-based personas and activation steering produce completely different failure modes. Testing one method gives you zero visibility into the other.

What this means: Your safety testing has systematic blind spots. You're passing evals while being vulnerable to attack vectors you never tested.

What to do:

  • Test safety across multiple methods, not just prompt injection
  • Treat safety QA like security testing — assume you're missing something
  • Build adversarial test suites that combine different attack vectors

5. Voice AI breaks in group conversations — nobody tests for this

Reality: Every voice AI assistant treats a detected pause as an invitation to speak. In 1-on-1 this works. In multi-party settings — meetings, family conversations, group calls — the agent constantly interrupts, talks over people, and violates social norms. No current eval framework even measures this.

What this means: If your voice AI will ever be in a room with more than one person, your entire turn-taking logic is wrong. And you have zero test coverage for it.

What to do:

  • Add multi-party scenarios to your voice agent test suite
  • Test with 2, 3, and 5 speakers — failure modes change with group size
  • Measure "inappropriate interruption rate" as a production metric

Deep Dive

Paper: "Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications"

What they claim: Traditional testing doesn't work for LLM applications because outputs are non-deterministic and model behavior evolves over time. They propose an automated self-testing framework that generates, runs, and evaluates test suites as quality gates before deployment.

What actually matters: This is the first serious attempt at solving the "how do you do CI/CD for AI" problem. The key insight: you can't write fixed test cases for systems whose correct output changes. You need tests that evolve with the system.

Where it breaks: The framework still relies on LLM judges for evaluation — which, as paper #3 above shows, are unreliable without enough samples. Also, the quality gates are binary (pass/fail) when most AI quality issues are gradual degradations, not cliff-edge failures.

Builder takeaway: The direction is right even if the implementation has gaps. If you're shipping LLM-based products without automated quality gates, you're flying blind. Start with: (1) generate test cases from production traffic, (2) run them against every model update, (3) track quality scores over time instead of just pass/fail. The goal isn't "tests pass" — it's "quality isn't degrading."

What's Breaking in AI Right Now

  • Benchmarks don't predict production performance. Multiple papers this week show the same pattern: systems that score well on clean evaluations fail under real-world conditions. The eval-to-production gap is getting wider, not narrower.

  • Nobody has CI/CD for AI. Traditional software has mature testing, staging, and release pipelines. AI systems have... vibes. One paper tries to fix this with quality gates. The rest of the industry is shipping without them.

  • LLM judges are the new technical debt. Everyone adopted LLM-as-judge because it's easy. Nobody measured judge reliability. Now teams are making ship/no-ship decisions based on evaluations with 30%+ variance between runs.

  • Voice AI testing stopped at "does it transcribe correctly." Real voice interactions involve interruptions, group dynamics, disfluent speech, and domain-specific terms. Test suites cover none of this. The failure surface is massive and untested.

  • Single-method testing creates false confidence. Whether it's safety evals, quality checks, or capability testing — using one approach systematically misses failure modes that a different approach would catch.

Build Ideas

Build this: AI system quality gate — automated QA pipeline for LLM-based products

Why now: Every paper this week points to the same gap: AI teams have no equivalent of CI/CD testing for their AI components. The tooling doesn't exist. Teams are shipping based on gut feel and one-off evals.

MVP:

  • Ingest production traffic (logs, user inputs, agent responses)
  • Auto-generate test cases from real usage patterns
  • Run regression tests on every model update or prompt change
  • Track quality metrics over time (not just pass/fail, but trend lines)
  • Alert when quality degrades below threshold

Why nobody's doing this yet: Testing non-deterministic systems is genuinely hard. But "hard" is where products live. The team that solves AI QA owns the pick-and-shovel play for the entire AI application layer.

Raw Papers

Get the next issue

Sharp insights from AI research. Every week. No fluff.