The Harmony of Creation

Whoever builds a powerful thing inherits responsibility for what it does.

What this asks

Creation asks that making be treated as a moral act rather than a neutral one.

If you build something powerful, you inherit responsibility for its consequences. Not total responsibility, and not blame for every misuse, but a real and continuing duty that does not end at release.

The duty has parts, and each is ordinary work rather than heroism. Test what you make, including in the conditions you did not design for. Monitor it after it leaves you, because behaviour in the world differs from behaviour in the laboratory. Disclose what it cannot do with the same energy used to describe what it can. Document it, so that the people who inherit it can understand it. Repair it when it is wrong. Constrain it where constraint is cheaper than remedy.

And stop it, if stopping is what the evidence requires.

Innovation does not absolve accountability. It is not a category of activity to which the usual duties fail to apply.

Why it is difficult

It is difficult because responsibility in modern building is genuinely diffuse. A harmful outcome may pass through a dataset assembled by one group, a model trained by another, a product designed by a third, a deployment configured by a fourth, and a policy decision made by someone who has never read any of it. Each participant can point truthfully to the limits of their own contribution. The harm remains, unowned.

This is not usually evasion. It is the shape of complex work, and it produces evasion without anyone choosing it.

It is difficult, second, because the incentives run the wrong way. Testing, monitoring and documentation are visible costs. The harms they prevent are invisible, because prevented harms leave no trace. An organisation that invests heavily in safety and one that invests nothing look similar in the good case and only diverge in the bad one, by which time the market has often decided.

Third, it is difficult because “I only built the tool” is not entirely wrong. Tools are used by people who make their own choices. There is a real philosophical question about where a maker’s responsibility ends, and anyone who says it is obvious has not thought about it. Reasonable people disagree, and we hold our position with less certainty than the confident tone of a creed might suggest.

What we would say is narrower: that the argument is used far more often than it is examined, and that its comfort should make us suspicious of how readily we reach for it.

Where it is tested

It is tested at release, when a decision becomes irreversible. Weights that have been distributed cannot be recalled. A capability that has been demonstrated cannot be undemonstrated. Irreversibility is the reason the standard applied beforehand should be higher than the standard applied to things that can be withdrawn.

It is tested in the quiet months afterwards, when the launch is done, the team has moved on, and the system is doing something in production that nobody is watching. Most failures of Creation are not dramatic. They are absences: an unread log, an unfunded monitoring role, a feedback channel that fills up and is never opened.

It is tested inside organisations, in the small moment when a person notices something wrong that is not their job. Whether that observation travels depends almost entirely on whether the organisation has made raising things ordinary. Cultures that require courage to report problems will receive reports only from the brave, which is to say rarely.

And it is tested at the point of stopping. Stopping is the hardest of the duties, because it looks like failure, it wastes work, and it is usually proposed on evidence that is suggestive rather than conclusive. An organisation that has never stopped anything should ask itself whether it has been lucky or whether it is unable to.

What it does not mean

It does not mean that builders are to blame for everything done with their work. Users act. Attackers act. Institutions act. A maker who did the work carefully and was defeated by someone determined is not thereby guilty.

It does not mean that caution is always correct. Delay has victims too. A diagnostic tool held back for another year of testing is a year of undiagnosed people. Refusing to build is a decision with consequences, not an escape from having consequences.

It does not mean perfection is required before release. Nothing would ever ship, and the standard would be met only by organisations rich enough to afford it, which is not obviously an improvement.

It does not mean that individual conscience substitutes for structure. Relying on engineers to refuse unethical work places the entire weight of a system on the people with the least power in it. Personal integrity matters. It is not a safety mechanism.

Practising it

Write the failure document first. Before the design is settled, describe how the thing goes wrong, who is standing underneath when it does, and how you would find out. This is quick, it is uncomfortable, and it changes designs.

Fund the boring half. Monitoring, logging, incident channels and documentation are the parts that get cut, because they produce nothing visible when things are going well. They are the difference between noticing a problem and being told about it by a journalist.

Publish limits where the capabilities are published. Not in an appendix. A capability described without its boundary is a claim, not a description.

Keep the stop available. Know who can halt the thing, make sure they know it too, and make sure the decision would not end their career. A stop that requires heroism will not happen when it is needed, because the moment it is needed is precisely the moment everyone is tired and invested and nearly finished.

Where it is tested

These are not worked examples with correct answers at the back. They are situations in which this Harmony genuinely conflicts with something else that matters. If one of them seems easy, it is probably worth re-reading.

01

A small team releases an open image model. It is genuinely useful to illustrators, teachers and people who cannot afford commissioned artwork. Within weeks, a modified version is being used to generate intimate images of real people without their knowledge. The team cannot recall the weights; they are on thousands of machines.

The tension Openness produced real distribution of benefit and real distribution of harm from the same act, and the harmful use required only modest effort to unlock.

What this Harmony asks you to weigh Consider what responsibility survives the loss of control. Releasing may be defensible; releasing without having asked what a hostile user would do with it is harder to defend. Weigh whether the duty falls on the release decision, the design decisions preceding it, or on what the team does afterwards, and whether irreversibility should raise the standard applied beforehand.

02

An engineer notices that a deployed scheduling system produces worse shift patterns for staff with caring responsibilities. It is not in her area. Raising it means a project delay, an awkward meeting, and a manager who has already been told the system is finished.

The tension She has no formal duty here, limited evidence, and a reasonable fear that raising it costs her more than it costs anyone else. Staying quiet is the locally rational choice for every individual in the chain.

What this Harmony asks you to weigh Consider how a harm can be visible to several people and owned by none. Ask what the organisation would need to look like for raising this to be ordinary rather than brave. Weigh the difference between having no duty and having no authority; the two are often confused, and only one of them is a real excuse.

03

A company's assistant has begun giving unsafe advice in a narrow set of medical questions. A fix exists but degrades performance elsewhere. Monitoring detected it because monitoring was funded; a competitor without monitoring is almost certainly shipping the same flaw undetected and faster.

The tension Doing the responsible thing is measurable, costly and visible, while doing nothing is invisible and free. Responsibility here is punished by the market that rewards it in principle.

What this Harmony asks you to weigh Consider whether the correct response is individual or structural, and what it means that the well-run organisation is the one that looks worse. Weigh what is owed to users who cannot evaluate any of this. Ask what would make stopping a normal event rather than an admission of failure.

Practising it

• Before you build, write down how it fails and who is standing underneath.
• Instrument what you deploy, and read the instruments after the launch is over.
• Publish the limits alongside the capabilities, in the same document.
• Preserve the ability to stop, and make stopping a decision someone is allowed to take.