Best of both worlds: Estelle’s journey in a pioneering role - August 20

A Level grades explained

August 2020

For this month's blog post I wanted to explain how A Level results were initially assigned to students, why teachers were unhappy with the algorithm for calculating exam grades and opportunities for improving grade assessment overall.

As a teacher of Computer Science I have found it amusing listening to people discussing algorithms with little regard to what they actually are - human written computer code. I have heard them discussed as though they were some nefarious computer that's only desire is to cause pain and suffering to the students of the UK. I think sometimes people forget that the fundamental thing with an algorithm is that it has to be written by a human and of course this algorithm is no different. The people who wrote the algorithm didn't intend for it to cause this furore, in fact the reason it was devised to try to make the exam grading process as fair as possible. 

Before I delve into the algorithm itself, I wanted to explain what schools and colleges were asked to supply to the exam boards this year - teacher assessment grades and, crucially, their students in rank order. The reason for both sets of data according to Ofqual was that teachers are better at making comparisons rather than absolute judgements. Which, to be honest I would agree with - it is much easier to rank your students by their ability than it is to assign each one a grade. However, there is a reason why its is particularly difficult to assign students a specific grade (but more on that later).

So, how exactly did the algorithm work. Well, there wasn't a one size fits all approach - there were three different ways that a-level grades were awarded dependant on the size of the cohort.

Small cohorts of five or less students used teacher assessed grades only as the algorithm wouldn't have worked as well on a small data set.

Medium cohorts of between 6 and 15 used a combination of teacher assessment and the algorithm and large cohorts used solely the algorithm. The algorithm is quite complex, you can read Jeni Tennison's, vice president of the Open Data Institute, detailed step by step description of how it worked here.

Tennison explains that there were 9 steps in the algorithm:

1. Look at historic grades in the subject at the school
2. Understand how prior attainment maps to final results across England
3. Predict the achievement of previous students based on this mapping
4. Predict the achievement of current students in the same way
5. Work out the proportion of students that can be matched to their prior attainment
6. Create a target set of grades
7. Assign rough grades to students based on their rank
8. Assign marks to students based on their rough grade
9. Work out national grade boundaries and final grades

And herein lies the problem, the algorithm threw out the teacher assessment all together and used their ranking alongside the modelling of how similar students performed in previous years. That meant that high performing colleges - where students outperformed the typical progress of similar students in previous years, would be downgraded and if a college hadn't achieved any top grades in the last couple of years then no students in the current cohort could. It also meant that a student ranked 7th in the class might have been given a B by teacher assessment but if the 7th ranked students from the two previous years got C's then it would be likely that this student would be awarded a C. 

Interestingly, it appears the model was tested to see if it could accurately predict last years data and it was found that it was only 40% accurate in some subjects.

Earlier, I mused that it's difficult for teachers to assign specific grades to students and said that I would come back to it - you see the process that students grades went though this year, isn't unlike the process student grades go through every year. When exam papers are marked they are not issued grades right away - they are ranked in order by their marks and then allocated their grades based on the standard distribution of grades in previous years. That means if students perform particularly well in a given year then the grade boundaries will shift, making it harder to achieve the higher grades that year. It also means that a '9' in one year isn't necessarily the same as a '9' in any other year. This moving of the goal posts makes it very difficult to accurately predict exactly how students will perform in any particular year and explains why it would be much easier for a teacher to rank their students rather than issue them a specific grade. It also highlights further issues with using historic data in the algorithm as how students have performed in previous years aren't necessarily comparable.  

In conclusion, I am relieved that the Government made a U-turn and I hope that this event presents an opportunity for making the whole process fairer in the future.