Human control over AI is more than an engineering problem

In a post on X, Sam Altman argued that humans must never cede control to superintelligence, stating that OpenAI opposes any future where people only perform tasks artificial systems decide to reward. The argument is incoherent. It rests on the premise that we can somehow build superintelligence and maintain control, while ignoring that artificial intelligence systems already decide what gets rewarded just by existing and disrupting the status quo. In many industries, the reward system is heavily oriented toward whoever can use AI the most and the best. We are already in the scenario he advises against; now it is just a debate over whether we consider current systems superintelligence.

It is also a classic case of saying one thing while doing another. Altman claims to hold human agency precious, even as his company specifically incentivizes maximum usage of its products. In market reality, that agency is largely an illusion. Humans can choose to opt out of the economy, but they are punished severely for doing so in most societies. Developers can opt out of using AI coding tools, but they will be fired, as with rare exceptions their colleagues will outcompete them at every measurable KPI and monopolize promotions and job opportunities. Warning about hypothetical machines directing human work ignores the fact that software is already setting the terms of what gets rewarded today.