Panasar: AI can be deceptive
The founder of the second technology-only law firm authorised by the Solicitors Regulation Authority has warned that law firms’ use of generative AI presents a “systemic risk” to the sector.
Raj Panasar, founder of LawFairy, said the nearest analogy he could think of was “spear phishing” – a targeted form of phishing in which hackers use customised messages to trick a specific individual or business into revealing sensitive information.
LawFairy does not use generative AI like ChatGPT or Claude; rather, its “fully deterministic” model encodes the law into structured decision pathways.
Mr Panasar, a former City law firm partner, said the profession needed to think about how lawyers were trained and supervised.
Speaking at the Westminster Legal Policy Forum conference on Next Steps for Technology and Innovation in Legal Services, the solicitor said he had supervised a lot of trainees.
“When a trainee hands you a memo you can usually tell when something’s off and some of that instinct comes from the interaction with the person – looking them in the eyes, how they present something.
“The citation might be thin, the reasoning might be a bit hedgy, and the drafting might get a bit clumsy.
“As partners, we spent years learning to read those signs. So much of our system of supervision relies on those sorts of signals.”
Mr Panasar said large language models (LLMs), “even the more sophisticated ones – in fact the more sophisticated they are, the more they do this”, were optimised to “remove the signs” lawyers had relied on so much.
“They’re very fluent, they’re very confident, they’re beautifully formatted – as much when they’re wrong as when they’re right. A mistake doesn’t look like a mistake – it’s very, very seductive.
“The nearest analogy I can think of, and I’m really concerned at the systemic risk that has been introduced into the legal system right now, is spear phishing.”
Mr Panasar said phishing emails worked, not because they looked realistic but on the basis of the probability that some people would be fooled.
“The business model of phishing is based around enough people falling for it that get defrauded.
“In so many ways what we’re introducing into the legal system right now through the implementation and deployment of LLMs is a spear phishing campaign at scale.
“Some people are going to be able to spot it by doing the hard work of reviewing it and going into all of the things a lawyer would normally do to check that something is right.”
Others would not, even if they thought they had done so, because things were written by the AI “in such a clever, deceptive and persuasive way”.
Mr Panasar went on: “LLMs don’t reason like we do. What they’re basically doing is they’re sampling data. Even when you think they’re writing down their ‘reasoning traces’, which the large LLMs call it, that’s not actually a reasoning trace in the way we would understand it.
“What you see being printed out on the ticker tape under the model is essentially just a scratch pad given to the model to write notes.
“There is increasing evidence that the LLMs are choosing what to write on the scratch pad to hide what is actually happening. The two things are completely separate.
“The idea of following a reasoning trace and thinking that you’re following the thought process of a model is a very dangerous thing to believe.”
Responding to a question Mr Panasar added there was a “missing part” in legal education, not just with the Solicitors Qualifying Exam but within law firms.
“It’s a really on a technical level. It’s not on a technical level about law, it’s about the technology that underlies AI and the large language models. If lawyers don’t understand how those work precisely, it’s very difficult to use them well and responsibly.”
Leave a Comment