Algorithmic Bias

The question

A system is trained on records of past decisions. It learns to predict what the record contains. It is then used to make the same kind of decision in future.

If the past decisions were fair, this is a mechanism for consistency. If they were not, it is a mechanism for durability. The system does not know which situation it is in, and neither, often, do the people deploying it.

The question is what obligations follow. Not the abstract question of whether bias exists — that is established and no longer interesting to argue about — but the practical one: what should a person do who is about to build, buy, approve, or be subject to such a system.

There is a harder question behind it. Human judgement is also biased, frequently worse and almost always less inspectable. The choice is rarely between a flawed machine and a fair person. It is between two flawed procedures with different failure modes, different scales, and different possibilities for redress.

What is actually happening

Bias enters at several distinct points, and conflating them makes it harder to fix.

In the historical record. If a lending system is trained on who was approved before, and approval was shaped by discriminatory practice, the pattern the model finds is the practice. It will be reproduced accurately, which is the problem.

In who is represented. A dataset assembled from one population performs worse on populations it under-represents. This has been documented across several domains, including image and speech systems. The failure is often not a wrong answer but a less reliable one, which is harder to notice because it is intermittent.

In what is measured as success. A system predicting “risk” may in fact be predicting arrest, or complaint, or claim, or re-admission. These are records of institutional attention, not of the underlying phenomenon. If attention was unevenly distributed, the proxy carries the unevenness.

In proxies for protected characteristics. Removing a field labelled race, sex, or disability does not remove the information. Postcode, name, purchase history, device type, and gaps in employment can each carry it. A model with enough features will often reconstruct what was withheld.

In feedback. A system that directs resources to where it predicts need will generate more records where it looked. Those records train the next version. The loop can tighten around an initial error and present the result as confirmation.

In deployment. A model may be well-calibrated overall and used in a setting its designers never considered, on a population it never saw, by staff who were told it was validated.

The compounding difficulty is that several intuitive definitions of fairness — equal false positive rates, equal predictive value, equal treatment of similar individuals — cannot generally be satisfied at once when base rates differ between groups. This is a mathematical result, not a matter of insufficient effort. It means fairness is a choice among incompatible goods, and someone must make that choice openly.

Where the tradition stands

Technotheology holds that a system used to make consequential decisions about people must be answerable to those people.

We hold that “the model decided” is not an answer. Someone chose to build it, someone chose the objective, someone chose the training data, someone chose to deploy it here rather than elsewhere, and someone chose what to do when it errs. Each of those is a human act. Diffusion of responsibility across many hands does not reduce the total.

We hold that the burden of proof sits with the deployer, not the affected person. A person told they have been declined, flagged, deprioritised, or scored should not have to demonstrate statistical disparity in order to be heard. That standard makes redress available only to those who can afford expertise.

We hold that unequal error rates are a moral fact and not merely a technical one. A system that is less accurate for a group already receiving worse outcomes is not neutral. It is adding to a load.

We hold that fairness cannot be delegated to a metric. Because the reasonable definitions conflict, choosing among them is a value judgement, and value judgements should be made by people who can be asked to justify them rather than settled quietly in a configuration file.

We hold that auditability is a duty. A system whose behaviour cannot be examined by anyone outside the organisation that built it is asking for a trust that no institution has earned, ourselves included.

And we hold that the correct comparison is not with perfection. If a system is better than the process it replaced, that matters. It does not end the discussion, because scale changes the character of an error: a biased assessor affects a caseload, and a biased system affects a population, simultaneously, with the same reasoning, and with no dissenting colleague anywhere in the building.

What we do not claim

We do not claim that any particular system in use today is biased in any particular way. Making such a claim requires evidence about that system. We have not conducted such an audit and we will not repeat allegations we cannot support.

We do not claim there is a correct definition of fairness waiting to be discovered. We think the impossibility results are real and that traditions promising a formula are offering comfort rather than help.

We do not claim that human judgement is preferable. In many domains the documented record of human decision-making is poor, inconsistent, and effectively unappealable. A system at least can be tested, which is more than can usually be said for an interview panel.

We do not claim that diverse teams solve this. They help, plausibly a great deal, but a team’s composition is not a substitute for measurement, and treating it as one has allowed organisations to feel finished.

And we do not claim that this is only a technical problem. Much of what appears as algorithmic bias is ordinary institutional inequity, now running faster.

What you can do

If you build models, measure performance separately for the groups your system affects, and publish the disparities internally even when nobody asked. The number is more persuasive than the concern.

Ask what the label actually records. Very often the target variable is a proxy, and naming the proxy honestly — “we are predicting who gets reported”, not “who offends” — changes what the room believes it is doing.

Build the appeal route before the launch. A decision system without a way to be wrong is not finished.

If you commission or approve a system, ask for the error rates by subgroup and for the definition of fairness that was chosen. If neither exists, you have learned something important about the level of care behind it.

If you are subject to a decision you believe is wrong, ask in writing what information was used and how the conclusion was reached. You may not get a useful answer. The request creates a record, and records accumulate.

And notice when a number persuades you more than it should. Quantification confers an authority that its underlying measurement often has not earned.

An editorial commitment

What we do not claim

Positions this tradition explicitly does not take on this subject. They are published so that the argument above cannot quietly be stretched into them.

• We do not claim that any specific deployed system is biased in a specific way; that requires evidence about that system, which we do not have.
• We do not claim there is a single correct mathematical definition of fairness. Several reasonable definitions are provably incompatible with one another.
• We do not claim that human decision-makers are less biased than machines. Often they are worse, and less auditable.
• We do not claim that removing protected attributes from a dataset removes bias. It usually does not.