Test shows ChatGPT still falling short in key areas, says conveyancer


ChatGPT: Real progress

The latest version of ChatGPT provides first-time buyers with better answers than it did two years ago but still falls short in crucial ways, conveyancing solicitors have warned.

In 2023, East Midlands firm Bird & Co put the AI system to the test and found that, while it answered popular queries well, it slipped up on stamp duty and first-time buyer ISAs.

It also failed to ask follow-up questions, gave shallow answers, and often produced dates, locations, and figures that were wrong.

The firm has now asked the same questions of ChatGPT-5 and said the results showed real progress.

It delivered “fuller and more structured explanations, often setting out information in clear step-by-step guides that made complex topics easier to follow”.

It also asked follow-up questions and included country-specific legal advice for different parts of the UK.

But the research found that one of the most serious problems remained, with the figures around stamp duty relief failing to reflect the changes introduced in April 2025.

The model “also continued to make the home-buying process appear more straightforward than it is”, while the role of a solicitor was only mentioned “very briefly”.

The research said: “More information is needed on the importance of independent legal advice when buying your first property.”

The broader finding was that, while ChatGPT-5 provided a good overview and pointed clients in the right direction, it did not give the tailored advice a lawyer would.

Partner Daniel Chard said: “ChatGPT-5 shows just how far this technology has come, and it’s encouraging to see AI tools offering clearer, more structured answers than ever before.”

While there was “real value” in using AI alongside the work of solicitors, proper legal advice remained “essential”.




Leave a Comment

By clicking Submit you consent to Legal Futures storing your personal data and confirm you have read our Privacy Policy and section 5 of our Terms & Conditions which deals with user-generated content. All comments will be moderated before posting.

Required fields are marked *
Email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Blog


What high-performing consumer claims firms get right

Recurring concerns about parts of the volume claims sector show that the gap between well-run firms and those struggling to manage volume effectively is widening.


The SRA’s 2025 AML report: What law firms need to know

The SRA has released its 2024-25 anti-money laundering report and the scale of supervision is striking – it carried out 935 proactive engagements in the year to 5 April 2025.


The managing partner in 2026: skills, security and strategic technology

The legal sector stands at a pivotal moment. The pace of technological change is accelerating, cyber threats are becoming more sophisticated, and client expectations are higher than ever.


Loading animation