Multiple choice exams: Good at assessing high-order thinking skills
The pass-rate for the first part of the Solicitors Qualifying Exam has fallen to an all-time low, new figures from the Solicitors Regulation Authority (SRA) have shown.
Just 41% of candidates in the July sitting of the SQE1 passed, although it was a little higher (46%) for those taking the two-part exam for the first time. Some 5,896 candidates sat both exams.
SQE1 assesses functioning legal knowledge through the single-best-answer multiple-choice exams covering several areas of law. SQE2 tests candidates’ practical legal skills and knowledge.
The first sitting of SQE1 in autumn 2021 saw a pass-rate of 53% and it continued at around that level until July 2024, when it dipped to 44% (46% of first-time candidates). For the sitting in January this year, it jumped up again to 56% (and a record 60% of first-timers).
In publishing the latest figures, the regulator did not seek to explain why the pass-rate has fallen so much, except to note that there were significantly higher levels of resitting candidates both this and last July than in the winter sittings.
A detailed breakdown showed that only 28% of candidates from a Black/Black British background passed SQE1, compared to 39% of Asian/Asian British, 51% of mixed ethnicity and 55% of White candidates.
This reflects longstanding differences in pass-rates that the SRA has been investigating in detail.
Men (49%) and a higher pass rate than women (43%), while 67% of those with first-class degree passed, compared to 46% with a 2:1 and 19% with a 2:2.
The pass-rate for those with qualifications below an undergraduate degree was 32%, while those who had a parent who had gone to university were more likely to pass than those who had not (51% v 41%).
Those who had gone to a fee-paying school performed better than those from state schools, although by far the best achievers were those who went to a fee-paying school but had a bursary covering at least 90% of their fees – their pass-rate was 64%.
Having qualifying work experience made relatively little difference – 47% passed, compared to 44% who had none – and being a qualified lawyer from another part of the UK profession or world appears to be a disadvantage, with only 42% of them passing, compared to 47% who were not.
Over the summer, former Attorney General Suella Braverman KC entered into a war of words [1] with a trainee solicitor over a petition calling for reform of the SQE. The SRA is not showing any signs of doing this anyway, although a five-year review of it should be published by the end of the year.
Meanwhile, the interim chief executive of the Legal Services Board last month called on the SRA [2] to publish data on SQE pass-rates by provider “as soon as possible”.
Separately, the SRA last month issued a detailed rationale [3] for single-best-answer multiple-choice questions (SBA MCQs), which it said were “widely used in professional high stakes exams in the UK and across the world”.
It said: “In SQE1, it is important that candidates are able to demonstrate they can apply their knowledge to a wide range of areas of legal practice.
“SBA MCQs allow candidates to do this in a time efficient and, bearing in mind the large numbers of candidates, cost-effective way. Many alternative methods of assessment only allow for the testing of a limited range of topics and/or are more expensive to administer and/or need to be taken over many days…
“SBA MCQs have been shown to assess high-order thinking skills (such as application, problem solving and evaluation) and differentiate between strong and weak candidates.
“SBA MCQs also allow for the use of a relatively large number of questions which enables sufficient coverage of subject areas and ensures the assessment is reliable and consistent – in other words the assessment can provide assurance that candidates scores are less impacted by chance factors, for example, the right topic coming up on the day.”
The SRA said that, whilst written assessments may be required when assessing legal skills, such as legal writing or drafting – and were used in SQE2 – “it is not a requirement when testing the application of knowledge”.
There was also “no evidence” that this style of assessment “advantage or disadvantage any particular candidates”.