First SQE assessment results | How they went and next steps | How did candidates do?
First SQE assessment results
21 January 2022
How they went and next steps
The Solicitors Qualifying Examination (SQE) is a single rigorous assessment for all aspiring solicitors. It consists of SQE1, which tests candidates’ functioning legal knowledge, and SQE2 which tests candidates’ practical legal skills. The first SQE1 assessments ran on 8 and 11 November 2021, with the first SQE2 assessments due in April 2022.
Below we provide information about how the first assessments went, how candidates performed and what we will report on in the future. This builds on the summary report on candidate performance published by Kaplan, the SQE assessment provider. There is also an accompanying report from the SQE Independent Reviewer who provides independent oversight and quality assurance of the SQE assessments.
How did the assessments go?
The SQE1 assessments were delivered successfully across more than 100 test centres in 26 countries.
1,090 candidates, including 27 solicitor apprentices, took part in the November SQE1 assessments, with 1,073 candidates sitting both Functioning Legal Knowledge 1 (FLK1) and Functioning Legal Knowledge 2 (FLK2). This is a transitional period with the majority of people still qualifying through the Legal Practice Course (LPC) or Qualified Lawyer Transfer Scheme (QLTS) routes so we predicted that the size of the candidate cohort would be smaller than in later years.
The SQE Independent Reviewer confirmed that ‘the initial SQE1 exam appears to have successfully delivered valid, fair, reliable and defensible outcomes. Each of the stages of preparation; delivery and processing outcomes for the exam demonstrated significant evidence of good practice. The operational and logistical processes to set up and deliver the exam proved effective.’
We are reviewing feedback from candidates and any learning points to help inform future assessments. We are running an online event in February 2022 where we will feed back on the SQE1 assessments. There will also be an opportunity for training providers to give their experience of preparing candidates for the first assessments.
How did we make sure the assessments were fair?
A key objective of any professional assessment is that it is fair. In other words, a candidate should get the result they deserve.
We have a range of measures in place, both prior to and after the delivery of the assessment, to assure a high quality examination that is fair to candidates. These include:
- pilot exercises, with more than 500 candidates in total across SQE1 and SQE2, to inform the final design and delivery of the SQE assessments
- training of all SQE question writers, including in relation to unconscious bias
- a robust process of editing and review of all questions, including in relation to cultural neutrality
- internal quality assurance by Kaplan and oversight quality assurance by the SRA and independent subject matter experts appointed by the SRA
- interrogation of data and the use of statistical analysis after the assessment.
SQE1 assessments are delivered and marked electronically. After delivery, Kaplan conduct detailed statistical analysis to determine whether the assessments were valid and reliable. The approach to analysis follows international best practice. These methods are well established and used to assess validity and reliability of assessments in a range of sectors including medicine. The review includes a detailed analysis of performance statistics for each question and each assessment. This statistical analysis is reviewed by an independent psychometrician appointed by the SRA.
This psychometrician confirmed that Kaplan employed psychometrically robust procedures in the standard setting and analysis of the assessment data and that the statistics indicate the assessment to be of a high standard. The SQE Independent Reviewer also confirmed that the process for creating the statistical analyses and quality assuring the results data was thorough and comprehensive.
How did we set the pass mark for the assessments?
The pass mark for the assessments was set using a Modified Angoff method. This involves a panel of qualified solicitors, trained for the process, who are familiar with what we have set out as day one competence. They consider each question on the assessment and estimate the proportion of day one qualified solicitors that would answer each question correctly (through reference to our Threshold Standard).
The pass mark for FLK1 was 57% and for FLK2 was 56%.
How did candidates do?
As above, 1,090 candidates, including 27 solicitor apprentices, took part in the November SQE1 assessments, with 1,073 candidates sitting both FLK1 and FLK2.
To pass SQE1, candidates need to pass both FLK1 and FLK2. 53%* of candidates passed both assessments, with 67% of candidates passing FLK1 and 54% passing FLK2. In order to qualify, successful SQE1 candidates will then need to attempt SQE2 (unless they have an SQE2 exemption) as well as showing us that they have a degree (or equivalent), that they meet our character and suitability requirements and have completed two years’ qualifying work experience (QWE).
*Note that for all statistics we have rounded to the nearest whole number
Comparisons with other legal assessments
There continue to be significant differences in successful completion rates on the LPC between providers. Between 2018 and 2020, pass rates ranged from 23% to 100%, with typical overall pass rates of around 58%. For the QLTS multiple choice test between 2018 and 2020, the average pass rate was 57%.
Difference in performance on FLK1 and FLK2
Performance on FLK1 was better than on FLK2. As the Independent Reviewer observes, possible reasons for this difference in performance could include the fact that candidates had less time to prepare for FLK2 as it was taken just three days after FLK1. Another factor could be the fact that candidates tended to do less well on the more transactional subjects such as conveyancing and litigation of which there are more in FLK2 than FLK1.
Statistical analysis and a thorough review and analysis of the questions in FLK1 and FLK2 did not suggest that there was anything in the question design or the standard of the assessments to account for the difference in performance across FLK1 and FLK2. We will continue to monitor and report on performance across the two assessments in future sittings. We will also talk to training providers, at the event in February, about areas where candidates might benefit from more support.
Pass rates across different demographic groups
With more than 1,000 candidates taking the assessment (and, as above, we anticipate larger numbers as we move through what is a transition to SQE), we were able to look at performance by candidates from different demographic groups. However, we must be cautious in drawing firm conclusions at this stage because:
- some of the demographic groups remained small
- 63% of candidates said that they would prefer to not give details of their demographic background in one or more categories
- we will be able to identify more meaningful trends after a number of assessments and when we have larger sample sizes, as is likely to be the case as SQE is embedded
- we will not know how representative this first cohort of candidates is until we have run the assessments for a number of years.
Pass rates amongst males and females were identical when rounded with:
- 54% of females passing overall
- 54% of males passing overall
- 48% of those who preferred not to say passing overall.
Pass rates amongst candidates who declared a disability compared with those who didn’t were also similar, with
- 55% of candidates who declared a disability passing overall
- 53% of those who said they did not have a disability passing overall
- 57% of those who preferred not to say passing overall.
We are keen to monitor whether factors such as socio-economic status affect performance on the SQE. We will look at this over the long term but results from this first assessment suggested that there was no significant difference in results based on socio-economic background.
For example, there was no significant difference between the performance of candidates who declared they went to non-selective state schools (57%) and those who went to a private school without a bursary (54%). And between those who were from a working class background (54%) compared to those with a parent or guardian from a professional background (56%).
Factors such as achieving a top grade at university or prior work experience were indicators of a greater likelihood to pass. We will continue to monitor whether this is the case as candidate numbers increase over future sittings.
Although numbers were small (27 candidates), pass rates by solicitor apprentice candidates were well above average.
As has been seen over many years in legal professional assessments and assessments in other sectors, white candidates generally performed better than candidates from Black, Asian and minority ethnic groups. It should also be noted that candidate numbers were small for some groups so analysis should be treated with caution. Overall, pass rates were:
- 43% for Asian/Asian British candidates
- 39% for Black/Black British candidates
- 58% for candidates from mixed/multiple ethnic groups
- 41% for candidates who declared ‘other’
- 49% for candidates who preferred not to say
- 66% for White candidates
Although candidate numbers are very different, for comparison purposes 2019/2020 completion rates on the LPC were:
- 52% for Asian/Asian British candidates
- 39% for Black/Black British/Caribbean/African
- 57% for mixed/multiple ethnic candidates
- 52% for candidates from other ethnic groups
- 65% for White candidates
- 53% for candidates with unknown ethnicity
And on the QLTS multiple choice test in 2019/20:
- 56% of Black, Asian and minority ethnic candidates passed
- 68% of White candidates passed
As reported above and by the Independent Reviewer, we have a range of measures in place to make sure that the SQE assessments are fair and free from bias and we remain committed to doing whatever we can to understand this difference in performance. The assessments are marked electronically which is the most objective way to mark. We conducted a thorough review and analysis of the questions and the data from the assessment to make sure there was nothing in the design or delivery of the assessment which might contribute to this difference in performance.
The Independent Reviewer has confirmed that the activities we have in place ‘appear appropriate to assure that all demographic groups are treated fairly and equally in the assessment process’.
We have also appointed University of Exeter to conduct research to explore the reasons for differential performance in professional assessments by candidates from Black, Asian and minority ethnic backgrounds.
We will continue to monitor and report on performance by candidates by ethnicity after each assessment. One of the benefits of the SQE is the rich dataset that it will provide over time to help explore this issue. We will provide more detailed analysis across a number of assessments in Kaplan’s annual report, the first of which will be published in 2023.
76 candidates sat the assessments with reasonable adjustments. A range of reasonable adjustments were made available from additional time to complete the assessments to assistive technology. We have collected candidate feedback on how candidates found the reasonable adjustment process. The Independent Reviewer reported that ‘Overall, given the confidentiality, complexity and importance of these processes and the sensitivities involved when making arrangements for candidates, the processes worked at least satisfactorily and were often good.’
We will review what went well and what improvements can be made for the future. We will also continue to engage with groups representing disabled solicitors to inform our work in this area.
The first SQE2 assessments will run in April 2022. Kaplan will publish a summary of SQE2 candidate results when they are released. We will also publish a report on the first SQE2 assessments by the Independent Reviewer.
Kaplan will continue to publish a summary of candidate results after each assessment
On an annual basis, from 2023, we will publish:
- an annual report from Kaplan which will include more detailed information on trends across a number of assessments
- a report on our quality assurance activity
- a report from the Independent Reviewer. They will not continue produce a report after each sitting.
We have also committed to publishing candidate performance data, including pass rates by training provider. This will benefit future candidates as it may be a factor for them to consider when choosing a training provider and it could help education and training providers evaluate the effectiveness of their training.
We will make this data available from late 2023, using the intervening period to work with providers to develop and refine the processes. We will tell those training providers, in confidence, who have sufficient numbers of candidates whether their candidates performed better, in line with, or worse than average in the November 2021 assessments. Providers can use this information to evaluate their training. We plan to share this information with them on a confidential basis after each assessment until we can make more data available.
We need to make sure we get our approach to publishing candidate data absolutely right to avoid any inaccuracies or misleading data that could adversely affect candidates’ ability to make good choices or the operation of what is an emerging training market. That includes making sure any data is appropriately anonymised and contextualised.
We are also mindful that we need a large number of candidates across multiple sittings to make the data we publish as meaningful and useful as possible, and we have been clear that numbers will grow as we move through the transition from the LPC and QLTS routes.
The importance of stakeholder involvement
The process of reaching the first SQE assessment has involved conversations with thousands of stakeholders including training providers, legal businesses, students, solicitors and representative groups. That engagement and feedback has led to us being able to design and develop the SQE assessments through to the launch of the SQE assessment website and the delivery of the first assessments.
For instance, our approach to reasonable adjustments was helpfully informed through feedback from representative and disability groups. We would like to thank all stakeholders for their input in getting us to this point. We will continue to engage with stakeholders to hear their valuable feedback and make sure that we deliver an effective assessment.