Guest post by Professor Rocky Scopelliti, futurologist and author of The Conscious Code
Scopelliti: Where does assistance end and professional judgement start?
The legal profession’s latest warning about AI arrived in a particularly uncomfortable form.
As Legal Futures recently reported, a registered foreign lawyer was struck off after using misleading AI-generated quotations and citations while defending proceedings before the Solicitors Disciplinary Tribunal. It was the first time the tribunal had dealt with AI hallucinations.
The obvious lesson is familiar: check what AI produces.
But I think there is a much bigger question approaching the legal profession: what happens when lawyers delegate cognitive work to AI but remain personally responsible for judgements they may no longer have independently reasoned through?
From hallucination to judgement
The current regulatory concern is understandable. Generative AI can fabricate authorities, misstate propositions and produce extraordinarily convincing errors. Yet hallucination may ultimately prove to be the easier problem.
AI systems are improving rapidly. They can already help lawyers research authorities, summarise cases, review documents, construct chronologies, analyse contracts and draft correspondence and submissions.
The more reliable these systems become, the more lawyers will use them. And that creates a paradox.
The greatest professional risk may eventually shift from lawyers trusting AI when it is wrong to trusting it because it is usually right.
Imagine an AI system researches the relevant authorities, identifies the strongest argument, analyses contrary precedent and generates the first draft of a submission.
The lawyer checks the citations, reviews the reasoning and approves the document. Who exercised the professional judgment?
That question is considerably harder than asking whether the cases exist.
The professional responsibility gap
I describe the problem emerging here as the professional responsibility gap.
It occurs when AI performs an increasing proportion of the intellectual work underpinning a professional decision while the human remains legally and ethically responsible for its consequences.
Put simply: AI can perform the work. It cannot inherit the professional duty.
A lawyer can be disciplined. A lawyer owes duties to clients and the court. A lawyer can explain a judgement, defend it, reconsider it and ultimately accept responsibility for getting it wrong.
An AI system cannot meaningfully do any of those things. That distinction becomes more important – not less – as AI becomes more capable.
Approval is not judgment
This leads to another problem I call the human rubber-stamp problem.
Human oversight is frequently proposed as the safeguard for consequential AI. But having a human somewhere in the process does not necessarily mean meaningful human judgement has occurred.
Consider a lawyer presented with a sophisticated AI-generated analysis containing dozens of authorities and thousands of words of reasoning.
To genuinely verify it, must the lawyer independently reconstruct the analysis? If so, some of AI’s efficiency advantage disappears. If not, what exactly constitutes adequate verification?
And what happens when the AI’s competence in a specialised domain eventually exceeds that of the person supervising it?
A human signature at the end of an AI-mediated process tells us who approved the work. It does not necessarily tell us who – or what – performed the reasoning. That distinction will matter enormously for professional regulation.
A right to human judgement?
There is a further question for clients. When someone hires a lawyer, what are they actually purchasing? Legal information? Document production? A predicted outcome? Or are they purchasing professional human judgement?
If AI performs most of the analysis behind advice, clients may eventually need greater transparency about where machine reasoning ends and human professional judgement begins.
That does not mean every use of AI should be disclosed. AI can enormously improve access to legal services, reduce costs and release lawyers from repetitive work.
The objective should not be to preserve inefficient human processes merely because they are human.
But neither should efficiency quietly redefine what society means by professional responsibility.
The immediate challenge is consequential AI
These questions connect with a broader argument I explore in my new book, The Conscious Code: From Artificial Intelligence to Artificial Consciousness.
Despite the title, I am not arguing that today’s AI is conscious. There is no established scientific evidence that current AI systems possess subjective consciousness.
Indeed, that distinction is crucial. AI does not need to feel, understand or possess consciousness before humans give it consequential authority.
A machine does not need to understand justice before helping construct a legal argument. It does not need to experience responsibility before influencing a professional decision. And it does not need consciousness before a human professional begins depending upon its judgement.
The immediate challenge, therefore, is not conscious AI – it is consequential AI.
The legal profession may be one of the first places where society is forced to confront what that distinction really means.
The question is no longer simply whether lawyers should use AI. They will. Nor is it merely whether lawyers should check AI-generated work. They must.
The deeper question is where assistance ends and delegated professional judgement begins.
Because, as machines become increasingly capable of performing professional work, one principle should remain unmistakably human: responsibility cannot be automated merely because reasoning can be.
Professor Rocky Scopelliti is an Australian futurologist, author and international keynote speaker whose research explores how emerging technologies are reshaping business, society and humanity.
Leave a Comment