“The validity of England’s accountability data”

Having been asked to “Blog my thoughts” following a talk by Prof Rob Coe (Durham University) on

“The validity of England’s accountability data”.

It was a typical Thursday evening in October when I travelled to Downing College, Cambridge with a colleague from another South Leicestershire school. The seminar suggested that it would probe deeply into the form of accountability which has built up in England. I was particularly interested to see how it would be “Critical” of Ofsted’s role and framework, and also how it would explore issues of validity and dependability in ‘data on the system’.

Having just delivered a lesson on Pythagoras the first challenge was to get to Downing College from the Queen Anne’s Terrace car park. Sat Nav suggested a 15 minute walk yet logic dictated a “shortcut” across the grass in front of us. 5 minutes later we are greeted at Downing College by the Porters Lodge and shortly after a welcome from the Network team and also a cup of coffee.

The talk started with Tim Oates introducing @ProfCoe for a discussion of how schools can compare themselves to others. If you haven’t seen the TEd Talk on Big Data here is the link: https://www.ted.com/talks/kenneth_cukier_big_data_is_better_data?language=en There are more and more bits of data collected by more agencies. FFT, (to a wide range of other agencies) they are all hungry for data and ways to use it. The problem I see is that the Education system is part of a complex social situation. There is lots of Data on all aspects of it and how it interacts and how can we measure “it” – Whatever “IT” happens to be at any given moment. The existence of data can often distance people based on the information that they have. For example, a house located in an estate might actually make the ideal home but for the fact of location, location, location.

What data? How much? What do we take from it? Prof. Rob Coe has spent a great deal of time looking at the Validity of England’s Accountability Data and in particular in what way do accountability systems vary. We are all aware that consumer choice drives systemic improvement !!  Accountability = Evaluation + Incentives  

There was a mention to look into Garn & Gibb (2001) for three types of accountability… Bureaucratic / Performance / Market.

Accountability can be used as an improvement mechanism…but what is the mechanism by which we expect to improve ?? Accountability = Evaluation + Incentives

As an audience we were asked “What does accountability look like to you ??” Soon we were all clear as we were all in favour of “Intelligent accountability”… However it was soon pointed out that not many would ever be in favour of “Unintelligent accountability”.

In school we know how important it is to trust your colleagues and that trust is important. You should trust the teachers. But the “Trust” needs to be willing and confidence based. If it is benevolent then we know that it is reliable, competent, honest and open. From a position of trust we know that there are good intentions and that they (whoever they are) would not exploit you !!  

Another question to ponder:

“If what you are doing isn’t good, do you want to:

1) cover it up

2) expose it, share, examine, maximise the learning opportunity.  

Teachers trusting parents and students make the difference. When Parents are well intentioned and in line with what you want then you can work together towards a same set of goals. One of the ways relating to accountability is how you feel about your own inadequacy !! How do/ would you feel ?? Would you want to share or hide ?? The question then becomes, “Do you want to observed during your best or worse lesson ??” – I think most would agree that this form of “performing” for lesson observation and that both sides will learn more with how to make it better.  

Looking into Evidence on impact of accountability …  The Research is not clear as to how far the impact on targeted outcomes can be transferred to other measures. With a lack of evidence towards long term, important outcomes. However it seems that getting rid of accountability widens the gap. A watched pot does not boil but at least if it is watched you can see whether the stove is on.

We need to know more about how evidence of accountability transfers to other studies.  

In 2013, the DfE study showed the limitations of existing research. In terms of accountability there are conflicting claims and evidence. Most evidence is weak at best.   

Here is the link to NUT Exam factories report….   https://www.teachers.org.uk/files/exam-factories.pdf

We need all of the perceptions from teachers, students and parents rather than just what can be independently observed from any one of the groups. We also need to unpick cause and effect.  Pressured accountability leads to students being anxious about exams. However, were they not anxious before accountability, ofsted etc.  Direct incentives do drive people’s behaviour. Accountability has a small positive effect on targeted attainment.  

The strength of an argument can therefore be put forward to make decision based on assessment. Should we use “grade B” in GCSE as filter for A-Level maths study for example. Or if a student does not achieve a threshold in a KS2 exam should they then repeat it again in Year 7.   Exams and test scores ARE used to judge schools and support for schools ie closed down/ taken over (if that can be considered support) !!  

Who should determine whether it is ok to use exams in the way ?? Exam boards, Employers,  OfQual ??

In terms of the accountability measures, do the measures represent valued outcomes? Are there important outcomes not captured by the measures?  For example in a school, is it not important to measures where students are 2 years later. What proportion of students are not in employment, education or training ??  

Could it be possible to improve performance on the measures without any real improvements in valued outcomes ??  Could it be possible to make significant improvements educational outcome without being reflected in any improvement in the measure?  

In the TES Sept 2013 Classroom observation there was “No scientific evidence” to show it led to better learning. And we all know that as teachers we love it when we are judged. This is not evidence !!  

I suggest that you pay a visit to Trevor Burtons Blog Eating Elephants (https://jtbeducation.wordpress.com/author/runningdown/) There is a nice post on “What’s the easiest way to a Secondary Ofsted Outstanding“

But as teachers do we really know a good lesson when we see one ?? With regards to Classroom Observation – it’s harder than you think !! We all know that there are people still in lessons, still grading teaching and learning. But now there is also an emphasis on the analysis of classroom artefacts, lesson plans, assignments, assessments, …  

It is thought that perhaps Inspectors should have to pass an exam. But not only pass an exam but establish the validity of inspection judgements. Show them video (lots of video) of lessons and see how well their judgement aligns with others.  Then give them a set of complicated real data… What would they be able to infer from that? What would you want to know ??  It is not just the skills and ability to make a judgement but the ability to then align that with others and to make sure that the judgement is valid that is important for inspectors. As part of this, perhaps a system to build quality assurance into routines and processes.  Under the best circumstances Prof Rob Coe says the “we can do it, but scale it up is challenging !! But if it is important enough you could build a validity study.”  

Phrase accountability system remains unchanged throughout parliament. It is funny how Parliament see this as a positive. They just love to tweak the accountability measures. They should just choose measures that are aligned with what is valued. They should then look at the distributions and not just the thresholds. It has been suggested that multiple partial inadequate measures build a bigger picture.  Exams and inspection should be improved !!!  

The system needs to develop such that there is accountability for the accountability. Ie to show a general improvement. That in itself is an interesting thought !!  

Progress 8 might not be quite as obvious as 5 A*-C But the question that springs to mind is “What are the loopholes in Progress 8 ??” For example, would you choose the 2nd English qualification when you can but a vocational subject in last bucket. The policy currently says we won’t tell you what to do… and it will take schools a while to adjust to the freedom.  In education it always takes a while to get used to change and there is possibly a transitional phase which schools will need to go through. There are lots of ways to cut the same measures and that is not the same as lots of ways for good measures.  

So I suppose we come back to the bigger question, “Why do we get out of bed in the morning ??” “What do we consider success ??”

What matters is that in terms of accountability we think about the ways we can capture data that we consider important.  It is always going to be impossible to capture things perfectly. Collecting data can be expensive and there are plenty of other limitations. What constitutes success ??  

Competition between schools allows choice. The point is not about winners and losers but that you get a better system wide approach. In education we should be thinking it is important for all to succeed and we don’t want to climb up by treading others down. Is Cooperation and collaboration between schools a better way forward ??  But there is a large part of England and Wales where you either go to this school or you go to that school. An analogy that Education not like buying baked beans… You buy a bad can of beans you can buy another. In Education, you can’t just buy another. The choice of school shouldn’t matter !! If you make bad choices you are going to get bad outcomes.

The solution, “Just make everywhere better !!”  

A final thought,

Thanks for reading my thoughts, and acknowledgements to @ProfCoe and team for a great presentation and for putting on an interesting and thought-provoking session.  


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.