The Scottish Qualifications Authority’s 2020 Alternative Certification Model – the deprivation impact

The aim of this blog is not to rehash the controversy surrounding this year’s Scottish Qualifications Authority (SQA) results day. It is to explain the context behind the decision-making, the implications of results in terms of inequalities, and the broader consequences for the Scottish education system.

To do this, Barry Black draws upon Free Schools Meals data gathered as part of his PhD research, alongside recently available secondary SQA data (obtained from Freedom of Information requests). Barry does this to illustrate what the real-world implications of results day were, and might have remained, had these moderated results not been withdrawn.

The context of COVID-19 and the Alternative Certification Model (ACM)

On 20th March 2020, the closure of all schools in Scotland was announced due to the public health restrictions required to tackle the COVID-19 pandemic. Alongside this, the decision was taken to cancel the full SQA exam diet. Instead, the Scottish Qualifications Authority (SQA) was tasked with developing an Alternative Certification Model (ACM) over the following months.

The SQA developed the model to be based on teachers’ estimates, with a system of national moderation that the Authority would apply. Teachers were asked to rank each of their pupils based on a range of holistic evidence from their performance throughout the year. The intention was for teachers to estimate the grades their pupils might have received had an exam diet taken place. Once these estimates had been agreed by the school leadership, the SQA moderated them by applying a system of statistical moderation that derived final grades based on the historical attainment of individual schools from the previous three years.

While the SQA moderation methodology can now be read in full, it was only made publicly available on results day, despite repeated requests from the Scottish Parliament for it to be published sooner.

MSPs, organisations and academics repeatedly warned that if the SQA were to use the historical attainment data of schools to statistically moderate the teacher-estimated grades of individuals, the negative effects of the model would fall more heavily upon pupils and schools with higher levels of deprivation.

And, sadly, this was exactly as it turned out.

On the 4th August, young people in Scotland received their exam results. The SQA and the Scottish Government noted that around 25% of all estimates, about 133,000 grades, had been altered by the moderation. Of these, 93% were adjusted down and 7% were shifted up.

Anger spread that morning, with numerous media reports of pupils whose expected level of results had been ‘downgraded’ by the SQA, but it was not until their Equality Impact Assessment (EQIA) was published later that same day that the predicted injustice became clear.

Table 1 (A13) below, from page sixty-nine of the EQIA, illustrates that the most deprived pupils (based on the Scottish Index of Multiple Deprivation) had their Higher pass rate reduced by 15.2% between teacher estimate and statistical moderation, compared to a reduction of 6.9% for the least deprived – based on SIMD.

Table 1: SQA Table on SIMD Information of Downgrades

Following a week of intense debate surrounding the issue, and similar controversies in England and Wales — who published their results the week after — the decision to use this ACM was reversed. On 11th August, the SQA accepted the teacher estimates in full, except for the results that had been upgraded by the moderation, which were upheld. New exam certificates were issued.

This blog sets out in more detail how this ACM, on average, would have affected the results of schools depending on the deprivation level of their pupils.

It is important to note that the ACM was successful in what it sought to achieve — the replication of the overall distribution of exam result grades achieved in previous years. This was designed to maintain the comparability of statistics between years and stop ‘grade inflation’. Indeed, exam results are always socially patterned, with poorer pupils performing worse than their less deprived peers. Results are a consequence and not cause of the very real poverty-related attainment gap. The unfairness of the ACM, however, stems from the fact the model did not consider any element of individual pupil performance in moderation. So, while in the normal course of events, exam results overall reflect genuine social inequalities, the individual pupil at least has an opportunity to influence their results through coursework or examinations. Instead, in 2020, the approach was to impose system-wide statistical norms on the individual certificates of young people.

Methodology

Three findings are presented below in which outputs from the SQA’s Alternative Certification Model (ACM) are shown in relation to the proportion of pupils at each school who are on Free School Meals (FSM).

The findings include the figures for independent or private (i.e., not state-run) schools who had enrolment in SQA qualifications. Of course, some pupils in these schools may be receiving Free Schools Meals, but their numbers are thought to be very low.

Free School Meals

One of the measures used as a proxy for deprivation in Scottish education is the proportion of pupils at a school who are receiving Free School Meals. This measure, of actual uptake of FSM, not merely entitlement, captures much more roundly the individual financial circumstances of the households from which pupil populations are drawn. The Schools’ FSM data is taken from the Scottish Government Secondary School Data Dashboard, where its represented categorically, as deciles, which sets out — within a ten-percent category — how many pupils at a school are on FSM.

SQA ACM Data

The SQA ACM data were obtained following a Freedom of Information request I submitted to the SQA for:

• Percentages of overall Higher grades adjusted from pass (A-C) to fail (D and lower) by the 2019/20 alternative certification model for each individual [school]
• Percentages of overall grades adjusted downwards by the 2019/20 ACM for each individual [school]
• Percentages of overall grades adjusted upwards by the 2019/20 ACM for each individual [school]

The two datasets — the schools’ Free School Meals data and the SQA results data — were linked, and summary statistics were calculated.

A note on the findings

It is important to note that the findings show the average percentage of moves from ‘pass’ to ‘fail’, and of upgrades and downgrades for the schools in each FSM quintile (and for Independent Schools). For example, Independent Schools considered together could have expected a 3.6% increase in the number of upgraded results as part of the ACM. However, the actual figures for the individual schools, and the raw figures for the sector as a whole, would have varied around that exact average value.

This is an important point, because averages can be skewed by unusually large or small values — outliers — which can increase or decrease these overall percentages. Outliers that have had a large impact on these educational outcomes have been presented clearly in the findings, but were kept in the analysis because they are important reflections of the system. Outliers are particularly likely to occur when, for example, only a small number of pupils from a school sit an exam for a particular subject. This is relevant in itself as small cohorts are a frequent benefit of independent schooling and have positive impacts on pupils’ eventual educational attainment. A more detailed statistical analysis of these results will be published in a forthcoming paper (Black & Mason, In Preparation).

Results

The number of schools included in each category is as follows. Note that two state schools are not included here because of incomplete data. A small number of Independent Schools are located outwith Scotland but enrol in SQA exams, therefore were, of course, included as part of the national statistical moderation.

Higher Qualifications

This graph shows the average percentage of teacher estimates for Higher qualifications that were submitted to the SQA as being an A, B or C grade which were downgraded to a D grade or lower after moderation, by the proportion of pupils at a school who were on Free School Meals.

In other words, Higher grades that were moved from a pass to a fail by the ACM.

We can see that in the most deprived schools — those with more than 40% of their pupils on FSM — the average change from ‘Pass’ to ‘Fail’ at Higher level was just over 20%. It is important to re-state at this point that this downgrade was due to the past attainment of the school and not based on any aspect of pupils’ individual performance.

The data shows that these schools had twice the proportion of downgrades on average compared with the least deprived state schools, and four times as many compared with the independent schools.

Crucially, we can see a very stark social gradient. On average, a particular school will have had fewer downgrades than a more deprived school, and more downgrades than a less deprived school.

Upgrades

This graph shows the average percentage of teacher-estimated grades that were upgraded by the model, by the proportion of pupils at a school who were on Free School Meals.

This is based on information about all grades submitted – for all levels of qualifications (i.e., not just Highers).

*On the left, a graph is presented with outliers included in the ‘Independent Schools’ section. It has been included in the analysis for the reasons outlined in the methodology and is an important reflection of the system-wide impact of the ACM. On the right is the presentation of the data without outliers. As can be seen, the exclusion of outliers moves the figure from 3.6% to 1.9%.

With the exception of the schools with 40+% of pupils on Free School Meals, the percentage of upgraded results broadly dropped as deprivation increased.

While these proportions are inevitably low, they are still relevant. This is the only finding presented in this blog that is still reflected in the grades that students were finally awarded, as results upgraded under the ACM were not returned to their original teacher-estimate. As can be observed this decision formally favours those at the schools with the lowest levels of deprivation.

Downgrades

This graph shows the average percentage of teacher-estimated grades that were downgraded by the model, by the proportion of pupils at a school who were on Free School Meals.

Again, this is based on information about all grades submitted – for all levels of qualifications (i.e., not just Highers).

Once more, note the large disparities linked to deprivation.

The most deprived schools had - on average - around double the proportion of downgrades compared with the private and the least-deprived schools.

Echoing the trend seen with the moderated Highers, there is again a steep social gradient. On average, a particular school would have had fewer downgrades than a more deprived school, and more downgrades than a less deprived school.

The decision was reversed, so why bother with this analysis?

It is reasonable to question whether the findings presented here are of any great relevance, given that the results of the ACM were ultimately scrapped, and the original teacher estimates accepted (except for the statistically modelled upgrades which were upheld).

The answer is two-fold.

Firstly, as is clear from reading the Independent Review of the issue, led by Professor Mark Priestley, the SQA maintain that they have ‘no regret’ regarding their ACM. The review also shows that Government officials and Ministers did not seek a fuller understanding of the data when the inequality within it was revealed. This omission makes it worth fully exploring what the impacts of the model would have been, had it not been withdrawn.

Secondly, as algorithms, big data and statistical models are gaining influence within our public policy and institutions and exerting greater power over our daily lives, it is worth highlighting how grave the outcomes of poorly designed and executed models can be in the real world. The key point here is that such tools are only as good – and as equitable – as the data and decisions that underpin them. This article, therefore, offers a stark example of the inequity that can be caused by such approaches.

Our qualifications system needs new priorities

There are 77 schools in Scotland, predominantly in the West of the country, in which 20+% of their pupils are on Free School Meals. The statistics presented here, and other findings I have published previously on attainment and number of subject choices, show that these schools fare much worse than the national average. This year, the ACM that was developed engrained the inequalities that pupils at these schools experience.

While the scrapping of this model was welcome, understanding how it was developed, used and accredited is important. I argued last week – at a Scotland’s Policy Forum Conference on the Curriculum for Excellence Review - that our qualifications system needs new priorities (PDF 0.11MB). These findings illustrate, in the starkest of terms, the inequality that is baked into our Scottish education system. Far from placement within a strategy seen over the past several years to ‘close the gap’ between the richest and poorest students, decisions this year engrained the gap in a statistical model. This model, before the u-turn, would have guided the next steps and life chances of thousands of young people in Scotland. The goal now must be to learn from this period and re-imagine how we go about removing the educational attainment gap in Scotland - and never again formalise it.

Barry Black

Barry's research aims to investigate the social and structural factors that influence pupils’ school subject choices in the Greater Glasgow region and assess the implications for employer’s engagement in schools.