Moorhead: Less sceptical about multiple choice questions than others
A leading academic has strongly criticised the pilot test of the first stage of the Solicitors Qualifying Exam (SQE) due to be introduced by the Solicitors Regulation Authority (SRA) in autumn 2021.
Professor Richard Moorhead, who will start work next month as head of law at Exeter University law school, said the pilot “obscures as much as it reveals” and did not meet “the basic reporting standards of the kind of testing it engages in”.
Referring to the SRA’s goal of creating a “world-class” exam in SQE, the academic said the pilot was “world class in the way that British volleyball is world class”.
He went on: “If the SRA really took a decision based on these reports, I’d be genuinely shocked. There must be a more open, honest, technical report and they should publish it.”
The pilot, run by Kaplan, was of SQE 1, which aims to test students on functioning legal knowledge through multiple choice questions and on basic writing and research skills through written answers.
However, following the pilot, the SRA said last week [1] it was considering dropping the skills element of SQE 1, mainly because it was too hard to assess.
A total of 318 volunteers sat the SQE 1 pilot exams in March this year at 44 test centres in the UK and two more in Singapore and France.
Professor Moorhead he expected to see in the report on the pilot a “technical but reasonably robust and open analysis of the reliability level that the test was set at”, together with examples of questions – some the candidates found easy, others hard.
“The example questions with pass fail rates would also be enormously helpful in generating understanding and perhaps acceptance of the tests.”
The professor said his “modest expectations” were not met.
“We don’t know anything about the representativeness of those who sat the test. We are told that those selected to sit the test are broadly representative but this is cold comfort.
“Those who asked to sit the test may have been broadly representative but who dropped out? The reports rely on telling us about representativeness rather than showing us the data to help us evaluate that claim. It’s a frankly bizarre way to go about dealing with data.”
Professor Moorhead said the testing of reliability suffered from the “same problem”.
The report on the pilot said grouping the multiple choice questions in three exams of 120 questions was less reliable than it should have been but gave no details of the test used or what the data from the test found.
“That would give statistically literate readers and lay readers a chance to assess the strength of the claim that it is ‘slightly less’ reliable than the accepted standard.”
Professor Moorhead criticised the failure to publish “the most basic of information” about the quality of the sample, the tests used and the actual results from the tests.
“Publishing such data is entry level not world class. You have to show us not tell us.”
Professor Moorhead said there were similar problems with the analysis of fairness in the pilot.
“Tantalisingly we are told BAME sitters did worse, but whether those candidates did the GDL [graduate diploma in law] or went to a Russell Group Uni were more significant sources of performance variance.
“The size and robustness of effects, and their interrelationships is crucial information here. It is missing.”
He referred to the “apparent, potential discriminatory effect” revealed by the pilot of the skills tests, where BAME candidates did much worse than their white colleagues. He said the fact that dropping the skills tests “will make the test cheaper” would have “helped the SRA make up its mind” on the future.
Professor Moorhead said the “apparent, potential discriminatory effects” of the multiple choice questions had been “glossed over”.
However, he said he was less sceptical about multiple choice questions than other academics and lawyers and “willing to be persuaded” they could be more sophisticated than generally thought.
“As part of a diet of assessment, they have significant merit. Whether they are sufficiently sophisticated to deal with all the knowledge-based skills a day one lawyer needs, I have reservations about, but as a start, and with supplements, they have value.”