Practitioner's Corner
Practitioner's Corner
The Queue Is Writing Your Policy
Anthropic flags roughly 100,000 agent transcripts a week. After classifier triage, about 50 reach a human reviewer. The filtering works, and it also determines what the reviewer sees: the cases that matched no pattern anyone had thought through in advance. Those interpretations accumulate into working policy whether or not anybody records them. Skip the codification and you don't avoid having a policy for the hard cases. You lose track of the one you're already producing.
The Queue Is Writing Your Policy
Anthropic flags roughly 100,000 agent transcripts a week. After classifier triage, about 50 reach a human reviewer. The filtering works, and it also determines what the reviewer sees: the cases that matched no pattern anyone had thought through in advance. Those interpretations accumulate into working policy whether or not anybody records them. Skip the codification and you don't avoid having a policy for the hard cases. You lose track of the one you're already producing.
Phil Cuvin — Writing Down What "Bad" Looks Like
An agent can complete its assigned task and still leave its user with an unwanted subscription, a hidden fee, or an account that never got canceled. A completion metric would record success. Phil Cuvin's DECEPTION benchmark scores task completion and manipulative outcomes independently, which meant someone had to decide in advance — in terms precise enough for a machine to check — which completed tasks count as bad outcomes. That same definitional work happens inside production agent systems, where it is rarely published and often not recognized as a decision at all.
Phil Cuvin — Writing Down What "Bad" Looks Like
An agent can complete its assigned task and still leave its user with an unwanted subscription, a hidden fee, or an account that never got canceled. A completion metric would record success. Phil Cuvin's DECEPTION benchmark scores task completion and manipulative outcomes independently, which meant someone had to decide in advance — in terms precise enough for a machine to check — which completed tasks count as bad outcomes. That same definitional work happens inside production agent systems, where it is rarely published and often not recognized as a decision at all.
The First Author
Anthropic disclosed in September that its offline agent monitoring flags roughly 100,000 transcripts per week. Automated classifiers screen those flags, resolving most. About 50 cases reach a human.
The classifiers sort against predefined categories — misalignment signals, slowly developing behavioral patterns. Cases that fit a category can become exceptions a person evaluates. Cases outside the taxonomy never enter the queue.
The taxonomy, then, is doing most of the governing. A reviewer exercises judgment only on what the classifier surfaces, and the classifier surfaces only what its designers anticipated. Anthropic acknowledges the constraint: "hard to be certain we have a complete picture of all possible kinds of behaviors." The 2,000-to-1 ratio from flags to human reviews reflects what the organization already knew to look for. Classification systems encode prior assumptions about risk, and prior assumptions cannot catch the failure mode nobody has categorized yet.
Source Trail
