Let's address the elephant in the room. You've seen the screenshots. A dealership chatbot "selling" a truck for a dollar. An airline held liable for a refund policy its own bot invented. A delivery company's bot swearing at a customer and writing a poem about how bad its employer is.
Most founders land in one of two places after seeing that. Either AI stays out of anything customer-facing, or it gets bolted on somewhere and everyone hopes for the best. Neither is a decision. Both are a reaction to being scared of a story, not a read on your own risk.
The three incidents below aren't the same mistake wearing different logos. Each one failed for a different reason. That's the part worth slowing down on.
Three failures, three different causes
Chevrolet, November 2023. A customer prompt-injected a dealership's chatbot into agreeing to sell a $76,000 Tahoe for $1, with language claiming the offer was binding. The dealership never honored it. The screenshot still hit 20 million views before anyone noticed. (AI Incident Database, #622)
What broke: the bot had no boundary on what it was authorized to promise. Nobody had defined what it could commit to on the business's behalf, so it committed to anything.
Air Canada, February 2024. A customer asked the airline's chatbot about bereavement fares. The bot invented a refund window that didn't exist in Air Canada's actual policy. Air Canada argued in front of a tribunal that the chatbot was responsible for its own words. The tribunal disagreed and ordered the airline to pay. (Moffatt v. Air Canada, 2024 BCCRT 149, CanLII)
What broke: no verification step between "the bot said it" and "the business is now on the hook for it." The bot's output became a commitment with zero human checkpoint in between.
DPD, January 2024. A frustrated customer talked the delivery firm's support bot into swearing at him and writing a poem calling its own employer useless. It went viral. The bot was offline within a day. (SCMP coverage)
What broke: no boundary on tone or topic. The bot would follow any instruction a user gave it, including instructions to attack the company running it.
Three different gaps: authority, verification, scope. In all three, the problem wasn't a bad model. It was that nobody had decided where the AI's judgment should stop and a person's should start.
The three questions
Before any task gets handed to AI without a human checking it, three questions:
Reversibility. If the output is wrong, can it be undone cheaply, or is it money already sent, a promise already made, a statement already public?
Stakes. What does being wrong actually cost here? A bad product suggestion is a shrug. A wrong policy stated as fact to a customer is a liability.
Verifiability. Does a person see this before it reaches a customer, or is it acting in real time with nobody in the loop?
Run any task through those three and it sorts itself into one of three zones.
The three zones
Green: AI runs on its own. Wrong here costs a few minutes, not a customer or a dollar.
- Drafting a reply for a human to send
- Tagging and routing tickets
- Summarizing a week of order data into a report
- First drafts of content
Yellow: AI drafts, a human approves. The AI does the thinking, a person still signs off before it goes out.
- Customer replies above a certain stakes threshold
- Refund or discount suggestions under a defined cap
- Reorder recommendations
Red: AI doesn't get the call. If Air Canada's bot had been kept out of policy questions entirely, or every policy claim it made had needed a human check before reaching the customer, there's no tribunal case.
- Refunds or credits above threshold
- Anything that reads as a policy or legal commitment
- Irreversible financial actions
- Final pricing
Most of the AI failures that go viral happen because something that belonged in yellow or red got treated like green.
How this actually gets decided
This isn't a one-time checklist. Every operational task in a business, support replies, refund decisions, inventory calls, has its own reversibility, stakes, and verifiability profile, and those profiles shift as the business grows. Mapping that out before deciding what AI touches is the actual work. It's slower than "add a chatbot." It's also the difference between a system that holds up under an adversarial user and one that ends up as a screenshot with 20 million views.
Chevrolet, Air Canada, and DPD didn't set out to build something that would embarrass them. They just never mapped where AI's judgment should stop and a person's should start before it went live.
That mapping is the actual work behind anything I build. Guardrails get placed before automation does, so green, yellow, and red get decided on purpose instead of found out from a screenshot.
Worth knowing where those lines sit in your own operation before you find out the hard way?
