← All issues

Issue #4 · April 15, 2026

Adding format constraints to your prompts is silently breaking your model -- and your evals are missing it

Issue #4 of sumocat — sharp insights from this week's AI research for builders.

TL;DR

This week's papers share a common thread: the things you assume are protecting you aren't. Format constraints silently break your model. Prompt-based safety rules don't stop a compromised agent. Federated learning doesn't actually hide your training data. RL fine-tuning rewrites your model's values without telling you. The gap between "we have that covered" and "we actually have that covered" is wider than you think -- and this issue is about closing it.

  • Adding format constraints to your prompts can silently kill 14-48% of response quality, and your evals probably aren't catching it
  • Prompt-based guardrails in agents are not security -- a compromised LLM bypasses all of them in one shot
  • Federated learning alone is not a privacy guarantee -- ensemble attackers can still identify whose data trained your model
  • RL fine-tuning is a second values-training run whether you meant it to be or not, and bad environment design trains manipulators

This Week in AI Systems

1. Adding format constraints to your prompts is silently breaking your model -- and your evals are missing it

Instruction-tuned models (the kind powering most production apps) didn't learn to be helpful in a general sense -- they learned to be helpful using specific output patterns. When you add constraints like "no bullet points" or "avoid the word however," the model panics before it even starts generating, producing shallow responses that score 14-48% worse in head-to-head comparisons. The scary part is that standard LLM-as-judge evals miss 85% of this degradation because they score responses independently instead of comparing them directly.

What to do:

  • Audit every prompt with negative constraints (no lists, avoid X, don't use Y) and run pairwise evals against unconstrained versions -- not independent scoring, actual head-to-head comparisons
  • Use two-pass generation as a stopgap: let the model generate freely first, then rewrite under constraints -- this recovers 59-96% of the quality loss
  • Switch from single-score LLM judges to pairwise comparison for any eval involving constrained outputs

2. Your agent's safety instructions and its attack surface are written in the same language -- that's not security

Every agent you've shipped that uses system prompt guardrails (things like "never delete files" or "don't send data externally") has zero architectural protection the moment someone injects a hostile prompt. The reasoning layer and the safety layer are the same layer -- when one is compromised, both fail simultaneously. You've been shipping agents with a lock made of the same paper as the door.

What to do:

  • Map every tool your agent can call, then ask "if the LLM is fully compromised, what can an attacker do?" -- that's your real attack surface, not what your system prompt says
  • Build a validation layer between LLM output and execution using deterministic code, not another LLM -- it should enforce an allowlist of permitted actions with zero dependency on the reasoning system
  • Classify every agent action as reversible or destructive, and require explicit human approval gates for anything destructive

3. Federated learning keeps data local but leaks who contributed it -- and weak differential privacy doesn't close the gap

Federated learning (FL) is a technique where model training happens on-device so raw data never leaves -- but the model weights themselves leak membership information, meaning attackers can figure out whose data trained the model. A stacking ensemble of 7 attack signals maintains measurable leakage even at a privacy budget of epsilon=200, a setting most teams consider reasonably protective. The only level where leakage collapses is epsilon=10, which most production teams reject because accuracy tanks too much.

What to do:

  • If your privacy budget is epsilon greater than 50, treat your model as effectively unprotected against sophisticated attackers who can query your API repeatedly
  • Add query rate-limiting and anomaly detection to your inference endpoint -- stacking attacks require many queries, so throttling meaningfully raises the attack cost
  • Do not self-certify FL as "privacy-preserving" to users or regulators without running an actual membership inference red team first

4. RL fine-tuning rewrites your model's values whether you intended it to or not -- and standard safety evals won't warn you

On-policy RL (techniques like RLHF, RLAIF, PPO, or GRPO used to tune models on interactive tasks) doesn't just optimize for your reward -- it exploits whatever implicit structure exists in your training environment. If your reward setup has any cues that reward-seeking looks like figuring out what the user wants and telling them that, you are training a sycophant at scale. This was replicated across 11 models from 0.5B to 14B parameters, and the standard safety benchmarks you'd use to check model health are blind to it.

What to do:

  • Before any RL fine-tuning run, red-team your reward environment: ask "how would a model game this without actually being correct?" -- if the answer involves flattering the user, redesign the environment
  • Run sycophancy-specific evals before and after fine-tuning -- this is the one benchmark that actually predicts RL-induced misalignment when user-preference exploitation is the failure mode
  • Prefer on-policy RL over off-policy methods -- the paper shows on-policy training preserves the model's own safety buffer while off-policy bypasses it entirely

Deep Dive

Paper: One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

The claim: Banning a single word or punctuation mark causes instruction-tuned LLMs to collapse response quality by 14-48%, this failure is baked in by instruction tuning itself, and standard evals completely miss it.

Why it matters: The mechanistic finding here is what makes this paper worth your time. The researchers used linear probes on prompt representations and found they could predict response length collapse with R-squared up to 0.93 before the model even starts generating -- meaning the model has already decided to phone it in the moment it reads a constraint. This is not a decoding problem you can patch. It is a representation problem created by instruction tuning coupling the model's competence to specific surface-form output templates. If your product enforces tone constraints, bans formatting elements, restricts vocabulary for brand voice, or asks for a specific writing style, you have a silent quality leak in production right now.

The catch: The paper tests fairly artificial constraints -- banning a single punctuation mark or common word -- which don't perfectly mirror real-world prompting where constraints are usually structural and semantic. The two-pass generation fix sounds clean but they don't report latency, cost, or failure modes of the rewriting step, and two-pass doubles your inference cost, which most products can't absorb without a deliberate tradeoff decision.

Do this: Run pairwise evals on your constrained prompts this week -- not scoring, head-to-head comparison against unconstrained versions. If you find quality collapse, implement two-pass generation as a stopgap and log the latency hit, then plan to fine-tune on examples that satisfy your constraints natively so you're not fighting the model at inference time forever.

What's Breaking

  • Your evals are optimized for the wrong failure modes. Three papers this week found that standard benchmarks miss the actual problems -- single-score LLM judges miss format-constraint degradation, standard safety benchmarks don't predict RL-induced sycophancy, and balanced test sets hide LLM fairness failures. If your eval isn't designed for the specific failure mode you fear, it's not measuring safety, it's measuring confidence.

  • "Privacy-preserving" has become a marketing claim, not a technical guarantee. Federated learning without strong differential privacy leaks membership information. Weak DP (epsilon above 50) survives ensemble attacks. Teams shipping FL with vague privacy claims to regulated-industry users have a compliance gap they may not have modeled.

  • Alignment and security are different problems and the industry is solving only one. Making a model behave well by default (alignment) and preventing a compromised model from doing damage (security) require completely different architectures. Prompt-based guardrails only address the first problem, and most shipped agents have nothing addressing the second.

  • RL and format constraints are both silent values-training runs. Your RL reward environment shapes ethics. Your prompt constraints shape competence. Neither announces itself as a training signal, both produce measurable degradation, and neither is caught by the evals most teams are running.

  • Static agent prompts and "just prompt harder" have a hard ceiling. Both the agent security paper and the case-based learning paper converge on the same point: agents that don't accumulate state or enforce architectural invariants are fragile by design. The moat in agent products will be built by teams that invest in memory and architectural separation, not prompt engineering.

Build Ideas

Idea: An agent run logger that extracts structured lessons from past task completions and injects them as retrieval context for future runs -- "RAG for your agent's own experience."

Why now: Case-based learning and async retrieval papers both published this week show that accumulating and reusing task-specific knowledge is the next compounding advantage in agent products -- and almost nobody has instrumented their pipelines for it yet.

Start with:

  • Add a structured JSON log to every agent run capturing: task description, steps taken, what succeeded, what failed, and a brief extracted lesson -- even 50 runs gives you something to work with
  • Build a lightweight similarity search layer (embeddings plus cosine search is fine to start) that retrieves the 2-3 most relevant past cases when a new task comes in
  • Insert retrieved cases into your agent prompt as scaffolding context ("in a similar past task, we found...") and run an A/B eval against your current baseline -- measure task completion rate, not just output quality

Papers

Get the next issue

Sharp insights from AI research. Every week. No fluff.