Bartley:
A City law firm has staged a head-to-head challenge between five leading AI companies to help choose which technology is best for its lawyers – and reported on the results for the benefit of others.
Charles Russell Speechlys (CRS) put together a team of 46 lawyers to take part in the two-week challenge, using real, albeit non-confidential, case work to test the AI “in real practice contexts”.
The lawyers – ranging from trainees to partners – fed the same tasks into all five AI products, so CRS could “compare apples with apples.”
The tasks mirrored the five main ways in which the firm’s lawyers have used AI in recent years: to answer legal queries, summarise documents and email threads, draft clauses and statements, and review contracts and reports.
Users were asked to come up with tasks that aligned with these categories and do the same one across each product.
At the end of each challenge, CRS’s legal technology team took structured feedback from the lawyers – providing 7,000 data points – to help see “where vendor promises matched reality, and where they didn’t”.
Key lessons included that legal competence – how well the product grasped the complexities and nuances of legal work – was more important than efficiency (such as speed of response and how many prompts were required).
“This suggests speed and accuracy have become baseline expectations rather than differentiators. Lawyers assume AI will be fast and reasonably accurate. Yet efficiency and accuracy metrics dominate vendor marketing, assessment criteria and adoption and benchmarking reports.
“Our data shows that what differentiates products, and builds trust, is whether they demonstrate genuine understanding of legal concepts, use terminology appropriately and grasp the complexity of legal work.”
The “clear winner” was Harvey, which was seen as the best AI tool across all of the tasks.
The CRS report on the experience, entitled A practical guide to choosing the right AI tools for your firm [1], sets out out its methodology, data and findings. It was compiled in partnership with TLW Consulting.
It said the data showed how ‘horizontal’ AI products – as opposed to practice area-specific vertical product – could meet general needs across practice areas.
“No single practice type was systematically underserved. Still, there’s substantial room for improvement in capability depth, particularly for disputes-focused reasoning.”
CRS’s disputes lawyers scored products lower overall, “suggesting AI still struggles with the analytical complexity disputes work demands”.
The report revealed the AI challenge emerged from “a potential crisis” at the firm in March 2025.
“The vendor we partnered with on our custom AI solution, Springbok AI, was acquired by another law firm [US law firm Cleary Gottlieb].
“Despite this being exciting news for the industry at large, it crystallised for us one of the risks that all legal tech users fear: your vendor disappearing. Our legal technology team needed to help us find a replacement solution and fast.”
CRS wanted to avoid holding multiple pilots to find a replacement and protect the confidence and trust in AI that existed within the team.
The report noted: “Our legal technology team turned potential crisis into opportunity and decided to do something unprecedented. After a thorough market scan, we designed a two-week comparative assessment that would generate enough structured data to make a confident investment decision.”
Harvey is used by more than 140,000 lawyers around the world, at a cost of around £1,100 per lawyer per month. Using OpenAI and ChatGPT technology, it processes 1.3m legal files each day and handles more than 200,000 legal queries.
After its success in the challenge, CRS has used Harvey to help Swiss Basketball overhaul its national competition regulations; advise on a workplace harassment investigation within a major public healthcare institution, using the AI to analyse the case file and show why a counter complaint was unfounded; and prepare and anticipate questions during a criminal fraud and mismanagement trial.
During the challenge, CRS uncovered a range of new insights into how its lawyers use AI.
Data showed more than a third of all AI tasks at CRS (36%) relate to queries about legal concepts, definitions and procedure, while 27% related to summaries of legal documents and email threads.
Drafting documents represented 7% of all usage, translation 5% and document review 3%.
Tessa Bartley, head of legal technology at CRS, told Legal Futures: “One of the reasons I suspect we may be seeing lower levels of generative AI usage for drafting than some might expect is that, as a firm, we already have a mature drafting offering.
“Over many years, our knowledge development lawyers have developed a gold-standard collection of precedent templates, and a significant proportion of those have been automated, giving our lawyers a reliable starting point and a streamlined drafting experience for the documents they produce most frequently.
“At the other end of the spectrum, the firm’s focus on private capital means much of the work we undertake involves highly bespoke client requirements and complex commercial considerations.
“These matters require nuanced judgement, deep client knowledge and personalised legal analysis. While generative AI can be a valuable supporting tool, our lawyers are often better placed to undertake that drafting themselves.”
The challenge also revealed that partners and trainees wanted different things from AI: partners sought accuracy, issue-spotting and depth of legal reasoning, whereas trainees wanted prompt comprehension, clarity of explanations and usable first drafts.
CRS said: “The legal industry has spent years running pilots that primarily generate anecdotal or behind-closed-doors feedback.
“As AI capabilities and competitive dynamics continue to shift rapidly, it’s time to consider alternative frameworks. Organisations that build better assessment processes will make better decisions faster.”