Sargeant: Evaluations not measuring the outcomes that matter most
The performance of AI in the justice system is only being measured in terms of operational efficiencies, and not if it is actually helping people deal with legal issues, research has found.
The review looked at the evidence that exists on the use of AI in administrative, civil, and family justice settings, following the rollout of the government’s AI Action Plan for Justice [1] in July 2025.
It was carried out by Dr Holli Sargeant from Cambridge University on behalf of the Nuffield Foundation as part of its Public right to justice programme [2]
She found plenty of evidence relating to “efficiency and task performance” but very little which showed how AI enabled people to genuinely seek justice.
In her report – AI and Justice – an evidence scoping review [3] – Dr Sargeant said her central finding was that AI evaluations “have not yet measured the outcomes that matter most from the standpoint of the people the justice system serves”.
Legal and human issues such as fairness, comprehension, lived experience and procedural legitimacy – what Dr Sargeant called “the conditions on which the public right to justice depends” – were not being measured.
According to the report, the impact of AI on people going through the justice system has not been measured because the technology was rolled out “faster than its full effects on the public can be independently or transparently evaluated”.
Dr Sargeant acknowledged that greater efficiencies would make the justice system “faster”, which would in turn lead to more access to justice. However, she warned that AI’s impact on people could not be overlooked alongside operational and performance goals.
From the 45 AI tools that Dr Sargeant focused on, only seven had any publicly available evaluation data.
The report noted that many of the AI tools being used across the justice system made performance claims – about reducing processing times, user satisfaction scores and broader claims about improving access or quality – but warned “they are self-reported and unverified claims, presented without published methodologies”.
Dr Sargeant called for independent evidence to be gathered now.
Since the rollout began last year, AI to assist courts and tribunals has been deployed the most, with 12 tools being used for transcription, listing, scheduling, form processing and case-management support.
Civil law has seen the second largest take-up, with eight tools being used in legal aid, negligence and small claims cases, followed by family law where the research team found six AI tools in use.
The report discovered evidence that ordinary people are not benefitting from AI. For example, when looking at how litigants in person used AI for legal assistance, the report discovered a “substantial gap between how users communicate with LLMs and how a legal expert would frame a legal information query”.
Dr Sargeant analysed more than 3,800 queries that had been submitted to a GPT4-powered legal tool by over 1,000 people.
The results showed that 70% of the queries contained no factual context. The litigants in person asked legal questions “without providing the circumstances that would otherwise enable a lawyer, let alone a model, to give a meaningful answer”.
Two-thirds of the queries sought legal information, while a third sought advice on what course of action to take – “a distinction that matters both for professional responsibility and for the kind of reasoning the model must perform”.
More than 70% of the queries were “open-ended” – which gave the AI “discretion over the form and content of its response”. Only 28% set specific requirements for the answers.
Fewer than 4% combined all three – factual grounding, specificity and structural clarity – in the same way “a legal professional would frame a legal query or factual information”.
On the use of AI by litigants-in-person, the review concluded: “The gap between casual lay communication and the precision required for reliable legal reasoning is itself a source of risk for lay users.
“It is also a risk that this could be a source of differential impact. Users with greater legal literacy, education, and digital confidence will receive better outputs than those without, reproducing existing inequalities through a new mechanism.”
In tribunals, the impact of AI on people was more profound, reducing “perceived procedural justice and legitimacy”.
In hearings with no AI, 33% of participants “strongly agreed” that the process allowed the judge to make an informed decision. When AI was used to summarise bundles, the figure fell to 7.5%.
When measuring fairness in hearings with no AI, 58% agreed the tribunal process was fair. With AI, the dropped to 22%.
In calling for more measurement around the use of AI across the justice system, the report stated: “The public have a right to a justice system that is accessible, fair, and effectively meets their legal needs. Whether the system is delivering on that right under conditions of AI deployment is not only a normative question – it is also an empirical one.”