In-School Variation and how schools can respond to it

In the UK, the performance gaps within schools (often called in-school variation or ISV) overwhelm variation between schools as a factor in national outcomes. At KS4, ISV is 14 times greater than between-school variation. If the performance in subjects below the average became average, there would be a 5% improvement in pupil attainment. Reducing in-school variation has been described as “the greatest educational challenge of our time”. ISV was a focus for the National College for School Leadership (NCSL) in 2007. An outcome of this work was a suggested school approach to ISV.  Today, ISV remains a suitable focus for school-improvement initiatives set-up to investigate ISV, to quantify its effects, and design strategies to reduce it.

Should school leaders be concerned by In School Variation?
   Consider this question:
Do Summer-Born pupils in your school perform differently to other pupils?
– Do we know the answer?
– Does it matter if we don’t know the answer?
– If a school leader didn’t know the answer, would this tell us anything about their leadership?

The question of the performance of Summer-born pupils is one which crops up frequently. The differences between Summer born and other pupils are often observable in the early years, but does the effect continue into secondary education?
A school leader might be expected to know what the position was in their school. If they don’t know the answer, it could indicate that they don’t think it is important to find out about this – and perhaps also other school performance issues. Yet we might expect a switched-on school leadership team to know about any under-performing groups in their school and have a strategy in place to support the progress made by such groups.
It makes sense for school leaders, subject leaders and teachers to be specialists in what happens in their schools, and to understand the effect of those factors which are unique to their situation and which influence the success of their pupils.

How easy is it to find the answer to this question?
To investigate the Summer-Born question we could tag all birthdays which fall in June, July or August and compare the performance of this quartile in their GCSE results with that of other pupils.  We could refine this further by looking at just those pupils with August birthdays, where in some cases they could be almost a year younger than their contemporaries. How many Summer-Born pupils in our school overcame the age disadvantage that they may have experienced as early learners, by the time they undertake their GCSE studies?

Because this is so easy to find out about, it would be reasonable that any leadership teams would be curious enough to know the position for their school, if anyone was to ask.
Summer-Born pupils are just one interesting category of pupil that sits outside the usual groups – like Pupil Premium pupils – when we are looking for differences in performance.
There are a great many more interesting groups that could similarly be investigated; and if our assessment systems were reliable enough, we could get an early indication of potential gaps in performance based on current progress, and put in place strategies to reduce any gaps that we found.

What is In-School Variation
In-School Variation is the variation in provision as experienced by different groups of learners. Inherently, ISV is a school effect which can affect the outcomes to learning for pupils. ISV can have a positive or negative influence upon learning effectiveness. Examples would include an under-performing subject or an unusually successful subject, or particular classes that achieve more or less than other classes in a subject. Variable provision can also be found in many characteristics of what the school does well or less well. These can be thought of as ‘local contextual circumstances’. There are also differences caused by the different experiences which pupils have outside of school, eg. home provision.

In-School Variation is often more significant than Between School Variation.
PISA studies first reported that the ratio of within-school variation (ISV) to between-school variation averaged 2:1 across OECD countries, but it was highest in the UK at around 4:1.
A DfES study based on 2003 data reported that in-school variation in England was even greater than the OECD had found.  In value-added terms, in-school variation during Key Stage 2 (KS2) appeared to be five times greater than between-school variation. During KS3 the ratio was 11 times, and for KS4 was a 14 times greater.  Ref: NCSL Closing the Gap.Untitled-2In the secondary school this is a huge effect which increases from key stage 3 to key stage 4. Put starkly, the effect of ISV can swamp any other school improvement initiative in its impact.  It is the case that a majority of school improvement initiatives are based on the notion that pupils need to get better at what they do.  So pupil premium funding in many schools is largely focused on remediation activities, rather than on teacher development, or improvements to the climate of the school, or the ability of parents to support their children’s’ learning.

If we just consider the example of the under-performance of some subjects in a school, the Fischer Family Trust suggested that if subjects below the average in a school became average, nationally, we would see an improvement in attainment at GCSE of around 5%.

Studies into In-School Variation
It was Professor David Reynolds and Professor David Hopkins whose work on ISV in 2003 provided a reference work to how schools could understand and respond to ISV:
http://www.highreliabilityschools.co.uk/_resources/files/downloads/within-school-variation/wsvreport2004.pdf

In-School Variation was also the subject of a two-phased project by the National College for School Leadership (NCSL) in 2007. They found that In-School Variation was:
NCSL findings2
At the heart of a response to ISV is the need for a school to achieve consistent teaching within subjects. The NCSL project built on the work of Professors Reynolds and Hopkins by setting up school projects to investigate the issue and find ways to address it.
The document arising from this work was ‘Schools learning from their best’.
http://dera.ioe.ac.uk/7381/1/download%3Fid%3D17377%26filename%3Dschools-learning-from-their-best.pdf
CPD2.jpg
It noted the difficulties in addressing the issues of tackling ISV.  It outlined the particular problem of quantifying ISV and measuring any changes to it over time. They also noted that Subject Leaders were the key agents of change if departments were to look at achieving teaching that was more consistent and as effective as the best subjects in the school.  The title of the document – ‘Schools learning from their best’ – also indicated an improvement model whereby teachers in a school known to be particularly effective could lead CPD sessions on ‘How I teach my subject’.

How to create the climate for projects to address ISV
In 2009 the Teacher Development Agency Published a document on Reducing in-school variation entitled ‘Making effective practice standard practice’:
http://dera.ioe.ac.uk/1276/1/isv_guide.pdf
This document made practical suggestions about how schools respond to ISV through a positive and optimistic approach based on three tenets: Openness, Trust and Collegiality.
openness_trust3

The NCSL found in its work that a ‘top down’ approach was less effective than a ‘bottom up’ one. In a top-down approach school leaders might hand out tables to show which classes were under performing, and ask for improvements to be made. In a bottom-up approach, teachers could be invited to use data tools to explore how well their subject was performing, to discuss improvement projects, and to report back to SLT on their progress over time.
A clear finding was that schools which encourage an action-research approach to school improvement are more likely to experience success.

The Systemic Influences of ISV
Discussion point: “School senior leaders don’t improve achievement. They set the climate where teachers can be successful”. This question usually gets a lively discussion going, but it also serves to emphasise that it is subject leaders and their colleagues who have the means to manage improvements in their subject area. It is the key role of school leaders to set a climate where teachers can be successful with this task. So senior leaders can, for example, make sure that behaviour management strategies are working and that other procedures are in place to ensure that teachers are empowered to be successful in their work.
climate3Factors which influence learning effectiveness include teaching effectiveness, learner disposition and the context within which learning takes place.  The most important influence on learning effectiveness is the teacher – but the context in which pupils learn, and other factors, influence how they learn.

Making Action Research Effective
The NCSL project pointed out the importance of trying to quantify the effects of ISV.  In the first phase of their project they found that without measures, reporting became descriptive or anecdotal.  In the second phase, spreadsheets were used to try to quantify the effects and track the changes over time. It was at this time that we shared with project schools a technique we were using which calculated the residuals of selected groups of pupils.

Today, ISV can be investigated much more readily because there are a greater range of software tools available to schools.  Many schools’ use of data has rightly come in for criticism of late by contributing to excessive workload. This issue arises where schools have insisted on collecting data too frequently and doing too little with it.  Like many things, it is a question of balance and clarity of purpose, and it would be an unwise school which didn’t explore the data available to it to see what could be learned about how well the school was doing.

teachersplanning2But there is a need for change in the way that schools use data. The 2016 document ‘Eliminating unnecessary workload associated with data management‘ offered good advice and some useful principles to guide schools, and there is still a need for practical guidance about how schools should be using data effectively.

Data is the currency of a diagnostic profession
In medical practice, doctors will take readings in order to gather information which they will then analyse before use their professional judgement to diagnose the condition, and then decide on an appropriate treatment.  There are some similarities here with teaching. A teacher will similarly refer to a range of information before providing a diagnose of how well a pupil is learning and what they need to do to improve.  If we are going to expand the scope of this into also looking into teaching effectiveness, how well the school is doing, and the performance of groups of learners, then a range of data techniques will need to be deployed to help schools to find the answers.

What are suitable questions for us to explore in our school?
I started this article off by posing the question about whether Summer-Born pupils in the secondary school perform differently to other pupils. This is just one of many other questions that we could ask.  What other things like this lie hidden in a school’s data?
In particular, what factors affect learners which may be unique to our school and are having a significant effect?  Are we prepared to sit on our school’s data and not find out what it can reveal?
To answer these questions we need easy-to-use data tools which have been designed to reduce workload rather than add to it.

Wherever there is a group of pupils in a school that has had a different experience to other pupils, there is scope for checking to see if this experience has had a positive or negative affect upon their learning.  We might find there is no effect, and this is evidence in itself.  In our work with schools we have asked for examples of where schools have created a research group and tracked their progress and attainment.  By comparing the residual for the group with other pupils it is possible to see if there are any ‘group effects’ associated with the subject of the research group.

Some examples of research groups reported by schools are as follows:
research3
A healthy situation to arrive at is when bottom-up research is encouraged and teachers have the tools to explore such factors. This is a big step away from data analysis being something that is done by the few and passed down to the many, but it involves all teachers finding out about the impact of their teaching on different groups of learners.

In-School Variation may be “the greatest educational challenge of our time” but a leadership team could start to tackle it by seeking to create a climate of openness, trust and collegiality in their school. They could further encourage CPD run by individuals and departments using the theme of ‘Learning from the Best’.  They could use a bottom-up approach to invite their subject leaders and their colleagues to undertake research-led school improvement using easy-to-use data tools.
The process of finding where variation exists and addressing it more specifically offers a more systemic way to look at school improvement, with the aim of achieving greater consistency of provision, and the possibility of fairer outcomes for pupils.

How can we explore and quantify In-School Variation?
The software tools available to the schools in the NCSL ISV project were simple spreadsheet applications which let teachers compare pupils’ grades in particular subjects with their average grades in all subjects, and look for patterns (residual values) which were of statistical significance. We could do this for groups of pupils with a common experience or intervention (like pupils who attended a revision class) to see if there was a ‘group effect’ of that intervention on the grades attained by pupils.  If a school invests time and money on an intervention it needs to know that it is having a measurable impact – else why do it?   Use of data tools can provide this information.

These software tools have continued to be developed and today are very much more sophisticated as we can see in the picture below.

research3
In the picture we can see for the chosen subject, English Language, the grades of pupils who do this subject compared to their average grades, arranged in rank order from pupils with the highest average grades to those with the lowest average grades. Two pupils are highlighted and we can compare their grade in this subject to their average grade. We can also explore how well groups of pupils perform in this subject, including research groups selected from the right-hand column. The second column shows the automatically-generated commentary for the relative performance of the subject itself.  We can use tools like this to identify effects early in a key stage based on estimated grades, and follow their progress through to their GCSE results.

In an ideal situation, subject leaders and their colleagues would use tools like these to explore the effectiveness of the subject, examine group effects, and design interventions aimed at raising the attainment of any under-performing groups of pupils.  This will help to recognise effects which might otherwise remain unseen, and identify the ‘local contextual circumstances’ which influence learning in a particular school.  An effective use of data tools will support an action-research approach to school improvement, and contribute to school self-evaluation as a more quantitative, and less descriptive process.

Questions for reflection:

How suitable is In-School Variation as a focus for raising school standards?
How important is it to find out what the data we have tells us?
To what extent do your subject leaders exercise a QA role for their subject?
How well-developed are the diagnostic data tools that teachers could be using?
What are the common obstacles to making smarter use of performance data?


mike_small
Mike Bostock is an education consultant/trainer specialising in the use of performance data tools for school improvement. Mike is currently working with the GWE School Effectiveness and Improvement Service in North Wales on In-School Variation and the use of data tools.

mike_wales_vsm3