Moorhead savages SQE pilot


Moorhead: Less sceptical about multiple choice questions than others

A leading academic has strongly criticised the pilot test of the first stage of the Solicitors Qualifying Exam (SQE) due to be introduced by the Solicitors Regulation Authority (SRA) in autumn 2021.

Professor Richard Moorhead, who will start work next month as head of law at Exeter University law school, said the pilot “obscures as much as it reveals” and did not meet “the basic reporting standards of the kind of testing it engages in”.

Referring to the SRA’s goal of creating a “world-class” exam in SQE, the academic said the pilot was “world class in the way that British volleyball is world class”.

He went on: “If the SRA really took a decision based on these reports, I’d be genuinely shocked. There must be a more open, honest, technical report and they should publish it.”

The pilot, run by Kaplan, was of SQE 1, which aims to test students on functioning legal knowledge through multiple choice questions and on basic writing and research skills through written answers.

However, following the pilot, the SRA said last week it was considering dropping the skills element of SQE 1, mainly because it was too hard to assess.

A total of 318 volunteers sat the SQE 1 pilot exams in March this year at 44 test centres in the UK and two more in Singapore and France.

Professor Moorhead he expected to see in the report on the pilot a “technical but reasonably robust and open analysis of the reliability level that the test was set at”, together with examples of questions – some the candidates found easy, others hard.

“The example questions with pass fail rates would also be enormously helpful in generating understanding and perhaps acceptance of the tests.”

The professor said his “modest expectations” were not met.

“We don’t know anything about the representativeness of those who sat the test. We are told that those selected to sit the test are broadly representative but this is cold comfort.

“Those who asked to sit the test may have been broadly representative but who dropped out? The reports rely on telling us about representativeness rather than showing us the data to help us evaluate that claim. It’s a frankly bizarre way to go about dealing with data.”

Professor Moorhead said the testing of reliability suffered from the “same problem”.

The report on the pilot said grouping the multiple choice questions in three exams of 120 questions was less reliable than it should have been but gave no details of the test used or what the data from the test found.

“That would give statistically literate readers and lay readers a chance to assess the strength of the claim that it is ‘slightly less’ reliable than the accepted standard.”

Professor Moorhead criticised the failure to publish “the most basic of information” about the quality of the sample, the tests used and the actual results from the tests.

“Publishing such data is entry level not world class. You have to show us not tell us.”

Professor Moorhead said there were similar problems with the analysis of fairness in the pilot.

“Tantalisingly we are told BAME sitters did worse, but whether those candidates did the GDL [graduate diploma in law] or went to a Russell Group Uni were more significant sources of performance variance.

“The size and robustness of effects, and their interrelationships is crucial information here. It is missing.”

He referred to the “apparent, potential discriminatory effect” revealed by the pilot of the skills tests, where BAME candidates did much worse than their white colleagues. He said the fact that dropping the skills tests “will make the test cheaper” would have “helped the SRA make up its mind” on the future.

Professor Moorhead said the “apparent, potential discriminatory effects” of the multiple choice questions had been “glossed over”.

However, he said he was less sceptical about multiple choice questions than other academics and lawyers and “willing to be persuaded” they could be more sophisticated than generally thought.

“As part of a diet of assessment, they have significant merit. Whether they are sufficiently sophisticated to deal with all the knowledge-based skills a day one lawyer needs, I have reservations about, but as a start, and with supplements, they have value.”




Leave a Comment

By clicking Submit you consent to Legal Futures storing your personal data and confirm you have read our Privacy Policy and section 5 of our Terms & Conditions which deals with user-generated content. All comments will be moderated before posting.

Required fields are marked *
Email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Reports

No larger firm can ignore the demands of innovation – that was the clear message from our most recent roundtable: “The law firm of the future”, sponsored by LexisNexis Enterprise Solutions. It comes in many forms, predominantly but not just technology, and is not simply a case of automating process. Expertise and process are not mutually exclusive.

Blog

13 August 2019

Managing internal and external procedures to drive success

Managing a large volume of cases simultaneously with regulation can often prove challenging. But it comes amid a shift towards a more digital approach to case management.

Read More

Loading animation