Make It Stop

An open-source sermon for developers. (With apologies to Pope Leo.)

I used to listen to a public radio show that retold old folktales for kids. One of them goes like this…

An old woman gives a poor girl a pot. Say Cook, little pot and it fills with sweet porridge. Say Stop, little pot and it rests. She and her mother are never hungry again. One day, while the girl is away, her mother says the first words and eats her fill. When she goes to stop the pot, she realizes the girl never taught her the second words, because the need had never come up in all their easy, well-fed days. The porridge climbs up the walls, fills the house, overtakes the street, swallows the town, until the girl comes home and says the word her mother never had reason to learn. The neighbors eat their way back to their own front doors.

A word you never have to say is a word you forget you had the right to receive. This is the cruelty of a good pot: It works every morning of your life, until the one it does not, and the only word that could have saved you is the one no one bothered to record.

In May, Pope Leo XIV gave us the same story in 42,300 words.

He signed his first encyclical, Magnifica Humanitas, on the 135th anniversary of Pope Leo XIII’s Rerum Novarum, the 1891 cry about what the factory was doing to the worker. The loom then, the model now, and at both machines a human asking the same question: Is this here to lift my work or to inherit it? Every age learns to say cook. Every age has to be reminded there was ever a second command.

AI, Pope Leo writes, can “paradoxically de-skill workers, subject them to automated surveillance and relegate them to rigid and repetitive tasks.” It erodes the human’s own sense of being the one doing the work. His remedy carries a startling image. In the Italian, he tells us to learn to “fast from AI”; the official English softens it to “exercise restraint in the use of AI.” Either way, the picture is the same: a hunger maintained on purpose, to preserve the muscle that ease so quickly decays. A pope looked at the most powerful tool of the age and prescribed, of all things, deliberate friction.

He points at the wound with real precision. He even speaks to the people who build these systems, saying their design choices carry a vision of the human being. But for the healing, he leans hardest on the State, on institutions, on discernment among nations — necessary things, and slower than the pot that is already boiling. The kitchen has its own work, and only one kind of person stands in it. That’s where you builders reading this come in.

The Fallacy of the Human in the Loop

Everyone assumes the solution is to watch the proverbial pot more closely. How many times have we been told to keep a human in the loop? It sounds like wisdom until you start asking what that human is really there to do — because more often than not, they are not there to catch the machine. They are there to take the fall for its inevitable failure.

A machine that always fails keeps you sharp. One that never fails needs no watching. But a machine that almost always works sings you to sleep. It’s right so many mornings in a row that watching it starts to feel like something you’re embarrassed to still do, until the morning it is wrong and your eyes are somewhere else. I have written before about the day my own car drove itself into a situation it couldn’t handle, my children in the back seat and my hands resting in my lap. The car knew how to cook. It did not know how to stop — and neither, in that second, did I. I was the human in the loop. The loop is what failed, because it was a man whom months of competence had gently, patiently sung to sleep.

That is not a weakness you can train away. Attention decays in front of a reliable machine, which makes “keep a human in the loop” a convenient place to rest the entire weight of safety — convenient for everyone but the human. Build a machine that will outlast any person’s vigilance, seat someone beside it with all the responsibility and none of the controls, and you have not built oversight. You have built a place to send the blame. The anthropologist Madeleine Clare Elish gave it a name that belongs over every dashboard: the moral crumple zone. Just as a car’s crumple zone absorbs the impact to keep the passenger whole, the human absorbs the fault to keep the system blameless. The driver, the law claims afterward, was supposed to be paying attention. Is that a careless driver they have found — or a feature they have shipped?

So asking us to pay closer attention is just cook dressed as caution — the same machine asking you to lean in further, with more speed and more trust. It’s right where the system wants you, because a person who can be blamed is cheaper than a pot that cannot boil over. In the tale, the girl simply comes home and says the words. Our version has to go further than Grimm did: the stopping built into the pot, both words carved into a wall where they hold, whether or not anyone is awake to say them.

The ability to stop has to live in the thing itself, because more than 75 years of research on vigilance says it will not stay fixed in our attention. When a pot floods a town, the fault belongs to whoever built the pot — not whoever happened to be in the kitchen. There is a human in the loop, and a human on the hook. The trick is to make you think they are the same person. We do not need to watch the pot more vigilantly. We need a pot that wasn’t built to drown the person beside it.

Nothing in this technology, the Pope’s encyclical insists, is “immaterial or magical.” If nothing is magical, every moral sentence in the letter has an implementation underneath it, and every implementation has a price.

The mother trusted the pot. I trusted the car. You trust the banner you never read. “Keep a human in the loop” is a demand for trust aimed at the one party it has already lulled to sleep. We are asked to extend faith, latitude and the benefit of the doubt to these systems, but trust must be earned. The earning is the one thing these machines are built to skip: label the content, affirm a commitment to transparency — words that cost nothing to say and move nothing once said, a sign nailed over the door of a house no one built. So the builder’s only real job is to make a machine that deserves the trust it asks for. Earning it comes down to the following three things — none of which require the machine to be more than a very good pot.

The first thing a builder owes is transparency, because you cannot trust what will not tell you where it has been. The encyclical asks that automated decisions be “understandable, contestable and subject to oversight,” and nothing is contestable that left no trace, just as no one could argue with a pot about a meal it would not account for. So the machine keeps its receipts: a full, replayable record of every step it took. But a record that waits on a tired human to read it is the watchman all over again. Make the machine answerable to another machine: a check, run against the record, that fails the build the moment the porridge tastes wrong, the way your tests fail it when the code does. The taster has to be code, not conscience, because conscience gets tired and code does not. I have spent a long while building exactly this: a way to hold the whole record and let the tasting happen on its own, every time, instead of waiting on someone to remember. The saving word gets said by the kitchen itself.

The second is a brake, because you only trust what you can stop. Trust with no way to pull back is merely exposure wearing trust’s clothes — the confirm dialog you long ago learned to click through on your way to “yes.” The Pope’s encyclical keeps returning to limits — that to be human is to accept them, that power needs “proportionate limits” — and a limit you cannot enforce is only a wish. So the boundary has to be real: scoped and revocable, the machine reaching into this and never that, stop its resting state and cook the deliberate exception, the permission falling out of its hands the instant you decide. This is the second word made physical — not a thing you remember to say over the pot, but the quiet the pot returns to when no one is asking it for more. A pot you cannot stop was never a pot to be trusted, no matter how sweet the porridge.

The third is ownership, because trust that lives in one house is not trust, it is tenancy — and a tenant trusts the landlord because the lease leaves no other option. When such power is concentrated “in the hands of a few,” the encyclical warns, it threatens equality, freedom and the common good. A model you reach only through someone else’s door, on someone else’s terms, priced and deprecated and revised in a routine email you do not get to reply to, is a pot in another person’s kitchen. You are renting your faith in its keeper, month to month, until the month the terms change. The town does not negotiate with the well’s owner, because the town owns the well: weights you can run without anyone’s leave, on hardware you hold, with the method carved into stone where the next person reads it for free. Concentration is the most expensive problem the encyclical names, and the only one a builder can truly answer — because you answer it in architecture, not applause, and what you buy is the right to stop trusting on someone else’s terms and start owning on your own.

Open Is the Door. Owning Is Walking Through It.

We ran this experiment once, with social media. We trusted the feed because trusting it was easy, and it never had to earn a thing. It had no memory we could audit, no way to question the ranking, no brake we could grab, and one house owned all of it. Everything that would have made it trustworthy was exactly what it left out, and so it never settled into our lives. Instead, it rearranged them. The second word is the refusal to do that twice — and that refusal is more than admiring open weights from a distance. A model you only reach through someone else’s endpoint is not yours in any way that survives the terms changing. You can wear the T-shirt, star the repo, and still be the town waiting on the keeper to come home. Open is the door. Owning is walking through it.

In Grimm’s tale, the girl just says the word and the porridge stops. The story ends there. But our trouble does not, so let’s allow the story to run one scene longer. She does not smash the pot — the pot is good, the porridge is real, nobody should ever be hungry again. She refuses the trade between abundance and safety. She teaches the whole town both words and carves them into the wall of the well-house where anyone can read them: neighbors, strangers passing through, even the old woman who first handed her the pot. A town that knows the second word keeps the pot and stays intact. It cannot be flooded. It cannot be owned. And nobody goes back to being hungry.

There is mercy in that. A word that only one person knows is a kind of power over everyone who doesn’t. A word everyone knows is just literacy — and literacy does not run out when you share it. The cost was always in the carving and never in the reading, which means the work you do once protects people you will never meet, in towns you will never visit, on a morning you will not be awake for.

So carve it — this week — in the only kitchen you control. Write the eval before you trust the agent, and let a failing check, not your fading attention, be the thing that says stop. Make a run with no readable trace fail to merge. Default the system to stop, and make cook the deliberate exception. Run one open model on a machine you own, with a kill switch that is genuinely yours. And when you ship something that acts for another person, refuse to make them your crumple zone: Give them controls equal to the blame, or do not ship it that way. Then put it in the open — the eval, the config, the method — so the next builder reads the second word off the wall instead of paying, as you did, to learn it the hard way.

The radio show always ended on a question instead of a moral. So does this.

In the tale, more than one person could say cook. The catastrophe was that, in the whole house, no one had been taught to say stop. Our house is larger now, and the question is the same: When this thing is humming and the street is filling, who here knows the second word?

Learn it. Then carve it where they can read it.