The Solicitors Qualifying Examination Pilot of the Assessment of Functioning Legal Knowledge: a psychometric and statistical analysis
In 2021, the Solicitors Regulation Authority (SRA) will introduce the Solicitors Qualifying Examination (SQE) as the sole route to qualify as a solicitor of England and Wales.
The SRA envisage that the SQE exam will have two parts. SQE1 will principally focus on application of core legal knowledge – the test of Functioning Legal Knowledge (FLK) which is discussed in this paper. It is undecided if it will also include a test of legal skills and this issue is not addressed further in this paper. The FLK test will utilise single best answer, multiple choice questions already shown to be sufficiently accurate and reliable for assessing applied legal knowledge in the Qualified Lawyers Transfer Scheme.
Three papers, each of 120 single best answer, multiple choice questions were constructed as follows:
- Business Law and Practice, Dispute Resolution, Contract and Tort
- Property Practice, Land Law, Wills and the Administration of Estates and Trusts, Solicitors’ Accounts
- Public and Administrative Law, the Legal System of England and Wales; Criminal Law and Practice.
The potential for two papers of 180 questions each was also evaluated. Reliability (reproducibility) and accuracy of outcomes (precision) are key to a high stakes licensing exam. Three exams of 120 items each, very nearly, but not quite, reached the levels of reliability and precision commonly regarded as the “gold standard” in national licensing exams. Two exams of 180 items each did reach the levels of reliability and precision commonly considered desirable by regulators.
Initial tentative indications are that educational factors (having successfully completed the Graduate Diploma in Law (GDL) and having undertaken a law degree in a Russell Group university in England and Wales) were key predictors of success in the exam. While this provides evidence for the concurrent validity for the exam, it is also a cause for concern, given the confounding of these educational factors with membership of minority groups protected under the Equality Act 2010. Kaplan will continue to work with the SRA to ensure that minority groups protected under the Equality Act 2010 are not unfairly disadvantaged while maintaining the standards of the assessment.
Overall, the SQE1 FLK pilot can be regarded as a successful delivery providing the statistical and psychometric evidence necessary, alongside other factors, for decision making about the SQE assessment design.