In 2023, a large language model passed a professional legal examination. Not barely. It scored in the ninetieth percentile. The achievement was reported with the mixture of awe and anxiety that greets every new proof that machines can do what we assumed only humans could do. And then something interesting happened: nothing changed. No law firm fired its associates. No bar association revised its standards. No court accepted a brief authored by a machine and submitted without a human lawyer's name attached.
Why not? The machine had demonstrated the competence. If the examination measures readiness to practise law, and the machine passed, why does the legal profession carry on as though the result is impressive but irrelevant?
There are straightforward answers. Regulations require a natural person to hold a practising licence; no statute contemplates a machine as a member of the bar. Liability demands someone who can be sued when advice goes wrong, and a neural network cannot appear as a defendant. Professional institutions, from training pipelines to courtroom conventions, were built around human practitioners and do not reorganise overnight. These are real constraints, not pretexts. But they invite a further question: why are legal systems structured this way? Why does the law insist on a human being at the centre of the process, even when a machine can match or exceed the human's technical performance?
Two centuries before anyone imagined a machine that could reason about law, a philosopher in Königsberg asked the question that explains why the legal profession is right to hesitate. Immanuel Kant reduced the entire vocation of philosophy to four questions: What can I know? What ought I to do? What may I hope? What is the human being? The first three are famous. The fourth is often treated as a footnote. But Kant insisted that the first three "relate to the last" (Kant, Logic, AA 9:25). They are not four parallel inquiries. They are three questions orbiting a fourth, the way planets orbit a star. Take away the star and the orbits collapse.
What does it mean to know something? That depends on what kind of being is doing the knowing. What does it mean to act rightly? That depends on what kind of being bears the obligation. What may we hope for? That depends on what kind of being is doing the hoping, and under what constraints of time, mortality, and uncertainty.
Here is the claim this section develops: no matter how capable artificial systems become, Kant's fourth question does not go away. It gets more urgent. And the reason is not sentimental. It is structural.
The exam is designed to test whether a person is ready to practise law. A machine passes it. But "ready to practise law" turns out to mean something different from "able to answer legal questions correctly." The lawyer stands in a particular relationship to her client: she owes a duty of care, she can be disbarred for negligence, she enters a courtroom as an officer of the court whose word carries weight because she has staked her professional life on it. The machine has no client. It has no career to lose. It occupies no standpoint from which passing the exam matters.
This is not a complaint about machines. It is an observation about what competence actually is. The examination assumes a background of commitments, vulnerabilities, and responsibilities that give the knowledge its point. Strip away that background and you have correct answers, which is a real achievement, but not the same achievement the exam was designed to certify.
Kant had a framework for this distinction, though he never imagined the case. He distinguished between causes and reasons. Causes explain events under natural laws: the billiard ball moves because it was struck. Reasons justify actions under norms: the lawyer advises her client because she has an obligation to act in the client's interest. A system that produces correct legal analysis is operating in the space of causes. It generates outputs because of how it was trained. A lawyer who produces the same analysis is operating in the space of reasons. She can be asked why she reached that conclusion, and her answer must satisfy not just a statistical benchmark but the professional and moral standards of her community.
Conflating the two is easy and increasingly common. When a system performs well, it is natural to say it "understands" or "reasons" or "knows." Sometimes this language is harmless shorthand. Sometimes it obscures the fact that performance and justification are different things, and that the difference matters enormously when something goes wrong.
The Difference Between Being Described and Being Formed
Kant drew another distinction that turns out to be surprisingly relevant. He separated physiological anthropology (what nature makes of the human being) from pragmatic anthropology (what the human being, as a free agent, makes of himself) (Kant, Anthropology, AA 7:119–120).
Physiological anthropology studies us the way a biologist studies any organism: inputs, outputs, mechanisms, constraints. This is the register in which most AI capability research operates. It asks what humans can do, measures how well machines replicate those abilities, and tracks where the crossover points fall. It is useful work. But it addresses only half the picture.
Pragmatic anthropology asks a different question: given what we are, what should we become? This is the question of character formation, judgment, education, civic life. It treats the human being not as a fixed object to be described but as an ongoing project to be shaped.
Think about what happens when a physician consults a diagnostic AI. The system flags a lesion as likely malignant. The physician must now decide: follow the recommendation, order further tests, or override based on clinical experience. This decision is not a performance benchmark. It is an exercise of judgment, which means taking responsibility for the outcome in a way the system cannot. The physician who always follows the algorithm and the physician who always overrides it have both stopped exercising judgment. They have become, in different ways, appendages of the system rather than agents using it.
Pragmatic anthropology insists that this capacity for judgment is not given. It is cultivated, through training, through experience, through the kind of reflective self-correction that Kant called Bildung. If hybrid systems demand more of human judgment rather than less (and the evidence from high-stakes medical, legal, and military domains suggests they do), then the cultivation of that judgment is not a secondary concern. It is the primary one. A society that invests billions in machine capability and nothing in the human capacity to use it wisely has misunderstood what the problem actually is.
The Thought Experiment That Settles Nothing
Here is a thought experiment that circulates in AI ethics circles: suppose a system emerges that exceeds human performance across every measurable cognitive domain. Mathematics, medicine, law, scientific discovery, strategic planning. It is better than any human at all of them. What then?
The definition of "general intelligence" shifts with each new capability milestone, and what seems like science fiction in one decade becomes a product demonstration in the next. But the instability of the definition does not weaken the thought experiment. It strengthens it, because it means we cannot wait for a settled definition before asking what follows.
The interesting thing about this thought experiment is how little it settles. If you believe that moral status tracks capability, then the arrival of a superior intelligence means humans lose their privileged position. We become, in the scheme of things, what animals are to us: less capable, and therefore (on this view) less worthy of moral consideration.
Kant rejected this logic before it was applied to machines. He rejected it when it appeared in the aristocratic presumptions of his own century, where birth and rank were treated as marks of higher worth. His argument was that the dignity of persons does not rest on what they can do but on what they are: beings capable of autonomy, of giving themselves the moral law, of being ends in themselves rather than merely means to someone else's purposes (Kant, Groundwork, AA 4:428–434).
A child has this dignity before she can reason well. A patient retains it when she can no longer reason at all. Dignity is not a performance metric. If it were, the implications would be monstrous not only for the comparison between humans and machines but for the comparison between humans and other humans. A sliding scale of worth indexed to competence leads, by straightforward logic, to conclusions that every functioning legal and moral system has learned to refuse.
So the thought experiment resolves as follows. A world in which machines outperform humans at every cognitive task is a world in which the question of what we owe each other, as beings capable of dignity, becomes more urgent rather than less. The capabilities have changed. The moral structure has not. This is not a comforting conclusion or a reassuring one. It is a structural observation about where questions of value actually live.
The Bus Stop and the Algorithm
There is a further problem, less discussed but equally important.
A planning algorithm can optimize a city's traffic flow, minimize hospital wait times, allocate resources across a supply chain with extraordinary precision. These are real achievements. But notice what the algorithm requires as input: the objective function. Someone must specify what "better" means. Faster? Cheaper? More equitable? Less polluting? These are not technical parameters. They are value commitments, and they often conflict with each other. The algorithm can find the optimum given a definition of optimum. It cannot supply the definition.
Kant's practical philosophy links moral striving with a permissible orientation of hope: the conviction that the highest good is realizable, that moral effort is not futile (Kant, Critique of Practical Reason). Hope, in this sense, is not optimism. It is the commitment to pursuing ends one recognizes as worthy under conditions of genuine uncertainty about whether they will be achieved. A prediction is a calculation about what will happen. Hope is a stance about what ought to be pursued. No increase in predictive power converts one into the other.
This matters practically. The citizens of a European city whose bus routes are being reorganized by an optimization algorithm have a right to ask: optimized for whom? The commuter who values speed and the elderly resident who values a stop near her door are not making a technical disagreement. They are making a value disagreement, and resolving it requires the kind of political deliberation that no algorithm can perform, not because algorithms are stupid, but because deliberation is not optimization. It is the process by which a community decides what it values, and that process is constitutively human: embodied, situated, argued, compromised, revised.
Kant treats the human being not as an abstract intellect but as an embodied, temporally finite agent whose judgments are formed within historical practices. Pragmatic anthropology studies character, temperament, sociability, the arts of living together (Kant, Anthropology). These capacities, judgment under uncertainty, tact in the face of disagreement, the civic virtue required to hold institutions accountable, are learned. They cannot be downloaded from a model. And no amount of external problem-solving power cancels the need to cultivate them.
Who Answers When It Goes Wrong
The practical stakes are already visible, and they are not hypothetical.
When an autonomous vehicle kills a pedestrian in a European city, the legal system must assign responsibility. When a hiring algorithm systematically disadvantages applicants from certain postcodes, someone must answer for the pattern. When a military drone selects a target, the chain of accountability cannot terminate at a neural network. In each case, the institution must trace a line from decision to person, from action to justification, from power to answerability.
If that line cannot be drawn, the decision lacks legitimacy. Not because it produced the wrong outcome (it may have produced the right one) but because no one is answerable for it. Legitimacy requires not just good outcomes but traceable responsibility. Kant's juridical writings ground this in the concept of a Rechtszustand, a rightful condition secured through public law: persons subject to coercive authority must be able to identify who exercises that authority and on what grounds.
This requirement does not disappear when machines enter the decision chain. If anything, it tightens. The more steps between a human decision-maker and the outcome that affects a citizen's life, the more carefully the chain of justification must be maintained. Institutional design must preserve lines of public accountability from human authorities to human addressees. The fourth question, "What is the human being?", is not a philosophical luxury here. It locates the seat of accountability. It identifies who can be held responsible, to whom justification is owed, and under what conditions authority is legitimate.
Constraints That Come From Nowhere Else
Kant's question does not compete with engineering. It disciplines it.
The question "What is the human being?" serves as a regulative frame for the design, governance, and evaluation of human-AI systems. It orients inquiry along four dimensions, and each has practical consequences.
The first is a design dimension. Systems should preserve the autonomy and non-instrumental status of the persons who use them and are affected by them. A system that reduces a human operator to a rubber stamp, confirming outputs she lacks the context to evaluate, violates this principle regardless of how accurate it is. The physician who cannot override, the judge who cannot depart from the score, the soldier who cannot refuse: these are not design successes. They are design failures measured by the wrong metric.
The second is an epistemic dimension. Predictive success is not justificatory adequacy. A model that predicts recidivism is not thereby a reason for detention; the reason must be articulable in terms a defendant can contest and a court can evaluate. If the system's basis for prediction cannot be translated into public reasons, it cannot serve as the ground for coercive state action, however accurate its predictions may be.
The third is an institutional dimension. Responsibility and rights must be anchored in human agents. Constraints imposed through sociotechnical systems require public justifiability: someone must be able to explain, in terms that an affected citizen can engage with, why the system operates as it does and who decided it should.
The fourth is an educational dimension. If hybrid systems demand more of human judgment rather than less, then the cultivation of that judgment is a primary institutional responsibility. Training people to use AI is not enough. Training people to think alongside AI, to know when the system's confidence is warranted and when it is not, to maintain their own capacity for independent assessment: that is the educational task that Kant's pragmatic anthropology anticipates.
The concept of artificial general intelligence is itself a moving target: what counted as general intelligence a decade ago (passing standardized exams, generating fluent prose) is now routine, and the goalpost has shifted to reasoning, creativity, autonomous research. It will shift again. This instability is not a problem to be solved but a feature to be understood. The definition of machine intelligence keeps changing because it is parasitic on our understanding of human intelligence, which is itself contested and incomplete. The timeline matters less than the structure of the question.
No matter how capable artificial systems become, they do not dissolve the question of what the human being is. They sharpen it. That question integrates knowledge, duty, and hope into a single horizon: it asks what kind of beings we are, what we owe each other, and what we are trying to become. It is the question a physician asks when she decides whether to follow the algorithm or her own clinical intuition. It is the question a legislature asks when it determines who bears liability for an autonomous system's failure. It is the question a society asks, whether or not it uses Kant's language, whenever it decides what kind of future it is willing to build.
The question is not whether machines will become intelligent. It is why, when a machine passed a legal examination, nothing changed. Statutes can be amended. Professional bodies can adapt. The slow-moving weight of institutional habit is real, but it is not permanent. When it shifts, something else will have to hold the line, or not.
Suggested references: Kant, Logic; Kant, Anthropology from a Pragmatic Point of View; Kant, Groundwork of the Metaphysics of Morals; Kant, Critique of Practical Reason.