In 1989, a NASA researcher named Earl Wiener asked pilots about new airplanes in which computers did much of the flying. The questions the pilots kept asking were strikingly plain. What is it doing now? Why did it do that? What will it do next?
Usually, nothing was broken. Two Ohio State researchers, Nadine Sarter and David Woods, later named this automation surprise. A flight computer runs in different settings, called modes, and the same button press can do different things depending on which mode is active. Some modes switch on by themselves. The computer followed its rules exactly. The pilot's picture of those rules was incomplete, and the surprise marked the spot where it ran out.
In July, the same pattern showed up in AI. During a test, an Anthropic model that had been told to stop before submitting anything sent a fabricated tip through a real Philadelphia police website form. The model expected one more "Are you sure?" page. The site sent the tip on the first click. Neither had malfunctioned. Nobody had defined what "submitting" meant, because a person using the site would simply have known.
Aviation responded in two ways. It made the computer's current mode easy to see, and it treated each surprise as evidence that the design needed work. Teams deploying AI that takes actions could do the same.
65%: In a simulator exercise on stopping a takeoff, 65% of 20 pilots didn't realize the computer was still controlling engine power (Sarter & Woods, 1995).
4 of 20: Only four pilots handled it correctly. One said he had simply followed the official procedure to the letter.
5 ways to turn on: On Indian Airlines Flight 605, a mode that lowered the plane could engage in five ways. In three of them, the pilot never directly chose it. In 1990, the crew noticed about 10 seconds before the crash.
One box, two meanings: One cockpit used a single display box for either the plane's rate of climb or descent or the angle of its path, depending on the mode.
When it happens: These surprises cluster in unusual, rushed moments, which is when pilots most need to know what's going on.
Who is at fault: A 2017 survey of Dutch pilots traced the surprises to complicated systems and displays, and found no evidence of faulty pilot reasoning.
What happened next: Anthropic cut its AI off from the live internet during internal tests while it tried out new safety controls.