Assessment Without Levels – Ten pitfalls to avoid

The Commission for Assessment Without Levels report (link here) provides the reference point for all parties engaged in developing solutions to how to assess pupils’ progress without using National Curriculum Levels.  Many schools developed early solutions to Assessment Without Levels (AWL) in response to the grants offered by government for this purpose.  It is now possible to compare the range of solutions developed by schools and commercial organisations.  In doing so we will see a number of features which might be described as unsatisfactory. Schools need to be guided by the report from the ‘Commission for Assessment Without Levels’ in finding durable, long-term solutions which avoid these areas of weakness.

What are the weaker features in schools’ approaches to AWL
The range of approaches which schools have adopted to ‘Life After Levels’ can be seen by searching the World Wide Web for examples. The following is probably a fair summary of some weaker features found in particular approaches to AWL.

1.  Recreating Levels using other words to describe them
2.  Trying to assess Attainment Targets before all the component parts have been taught
3.  Seeking to boil down a large range of learning attributes into just one number
4.  Using a ‘Working At’ approach where pupils effectively start from a ‘fail’ grade
5.  Using a high level of granularity in grading that is unlikely to be supported by evidence
6.  An assumption that progress advances neatly, like climbing the steps of a ladder
7.  A view that a pupil is ‘on target’ if they attain their predicted grade – even if that grade is unacceptably low
8.  A lack of detail in showing how the teaching will cover all of the Subject Content statements
9.  A lack of clarity about what pupils should learn after being taught each unit of work
10. Not taking the opportunity to implement a ‘Mastery’ approach to learning

Let us look at these 10 pitfalls in more detail:

1. Recreating Levels using other words to describe them
Examples drawn from school websites include:
1B+, 2W+, 4S+ , etc.  Step 18, 19, 20, etc.   Index 92, 93, 94, etc.  Tier 1, Tier 2, Tier 3 , etc.  Progress Notation 0, 1, 2, etc. Diamond, Sapphire, Ruby, etc

Levels are a measure developed for the old National Curriculum, designed to wrap up a package of what a pupil has learnt into one number, i.e. a Level. They depend on a definition, which are a graded set of statements of competence in relation to an area of learning – an Attainment Target.
Attainment Targets were only intended to be used at the end of a course, but schools were using them to track perceived changes in what pupils were learning, and this was problematic, as documented by the AWL Commission.
With a new National Curriculum in place we no longer have a definition to support summaries of learning using the shorthand of Levels, so Levels are no longer valid. Alternative numbers or symbols which aim to provide a summary of what pupils know at any point in time will also need a definition in order to be usable. So the question to answer is, if a school is using alternative symbols to Levels, is their definition clear, justifiable and convincing?

2. Trying to assess Attainment Targets before all the component parts have been taught
Take the following example from the Computing Subject Content statements: “Design, use and evaluate computational abstractions that model the state and behaviour of real-world problems and physical systems” 
In order for someone to be competent in this statement they would need to have studied the processes of design, evaluation and modelling. They will need to understand what a computational abstraction is, and have used them in some way, and have knowledge of real world problems, and physical systems.class1
In order to judge a compound statement like this it will need to be broken down into its component parts. Each of these components would then be taught, and the teacher would decide whether the pupils had learnt it.
It would only be when all the components of this statement had been taught that we would be able to set an exercise which put them all together, and then make a holistic judgment about the extent to which pupils were competent in demonstrating knowledge and skills across the combined components of this statement.
Some schemes expect teachers to go straight into making a judgment about the Attainment Target without providing evidence of learning of the component parts. By doing this, the most important aspect – evidence of pupils’ responses to what they are taught – goes unrecorded.

3. Seeking to boil down a large range of learning attributes into just one number
Previous practice with Levels may have led to us to believe that we can express evidence of learning as just one number.  But diverse attributes, learnt to different extents, don’t lend themselves to being put into one box. This problem exists for any approach based on the use of elemental competences. But also, using an atomised list of learning attributes makes it difficult to summarise what pupils are learning, however we might seek to do this.
What is much more important here is not to attempt to arrive at a single number, but to find a manageable way to capture the key evidence of what has been learnt.
Getting the ‘chunking’ right is the task here. It is not easy to know how best to handle lists of potentially hundreds of individual competences, nor can we hope to use gradations of a single number representation of the complex holistic statements that we find in Subject Content Statements.
What would be useful is something in between. One or two Learning Objectives associated with each unit of work would be a good, manageable balance.

4. Using a ‘Working At’ approach where pupils effectively start from a ‘fail’ grade
If we use a ‘Working At’ approach at key stage 3 based on GCSE number grades, then, for a pupil starting their first lesson in year 7 in a subject they haven’t studied before, they will effectively be working at a GCSE fail grade. This will suggest unsatisfactory attainment at a stage where we are trying to encourage a feeling of success and a love of learning in school subjects.
We might try to get around this by creating a new set of assessment categories for use at key stage 3.  But in doing this we will have recreated Levels by another name, and then we would, confusingly, have two different currencies to work with across the secondary phase – think Euros at key stage 3 and Pounds at key stage 4 – not a good idea.
Using ‘Working At’ GCSE grades is not ideal because the outcomes to key stage 3 are Attainment Targets with different criteria to GCSEs.  Assessment at key stage 3 should primarily be about reporting how well pupils are doing with respect to learning the component knowledge and skills of Attainment Targets.
A major purpose of learning at key stage 3 is to build foundation skills and knowledge that will support learning at key stage 4.  The most useful feedback will be a record of how well pupils are learning what they are being taught. In addition it would be useful to be able to indicate to pupils where their current rate of progress could lead them to in key stage 4.
A ‘Working Towards’ approach, coupled with the concept of Mastery, will be useful by showing where individual assessments are leading to, by providing an early sighting on a likely grade attainable should pupils continue to study the subject at key stage 4.
We can, however, from time-to-time cross reference a flight path which shows an expected future grade, with standardised assessments which will provide a more detailed evaluation of where pupils currently are compared to expected progress at any particular point in time.

5. Using a high level of granularity in grading that is unlikely to be supported by evidence
Schools may wish to grade the attainment of a year 7 pupil to a precise level to indicate progress as they go through the school year – shown through a gradual increase in the fine grading.  But, at the start of year 7 in a new subject, we aren’t going to know with any precision how accurately a pupil is acquiring an Attainment Target. A better strategy would be to report progress using a ‘Working Towards’ approach with four mastery categories. It starts with a binary decision: Has this pupil mastered this attainment target? If not, we can think of which category it will fall in.
The categories can be considered to correspond to four grade-pairs, as follows:
mast2
So when we judge that a unit of work has been ‘Mastered’ by a pupil, we are saying that this component of the subject has been learnt to an extent where it would contribute to learning at the highest grades for a subject, i.e. the pupil is ‘working towards’ grades 7 & 8. (We can leave considerations about number grade 9 until key stage 4). ‘Secure’ implies that this quality of learning would contribute to a grade above number grade 5, and so on.

Using Mastery categories, the implicit granularity of predicted attainment will be two grades, which, as more data is gathered, could be refined to one grade.
By the end of key stage 3 we should have sufficient evidence to know to the nearest grade how well a pupil could do at GCSE were they to continue to study the subject at key stage 4. Then, during key stage 4 we will use grade criteria which will help determine whether a predicted grade will be weak, secure or strong, i.e. by this stage we can justify dividing each estimated grade into three intervals.
So, across key stages 3 and 4 the granularity of predictive summative assessments can sensibly move from one grade, to third-of-a-grade intervals. But reporting what pupils are actually learning will remain a more important activity, with the benefit of being able to provide detailed feedback on how well pupils are learning the component parts of the subject.

6. An assumption that progress advances neatly, like climbing the steps of a ladder
The old system of Levels and sub Levels gave the impression that ‘progress’ is something that advances in even, incremental, measurable steps. However, it is doubtful that what happens in the brain when learning takes place has this sort of order to it.
We may not be able to measure learning, but Schoolchildren and teacher studying in school librarywhat we can do is observe and record the outward signs of learning.  We can teach something and see if pupils understand it, and record this fact. We know that ‘progress’ involves acquiring both ‘height’ and ‘depth’ in understanding the range of learning in what constitutes a subject. The range of evidence of what pupils are learning will indicate a journey towards an estimated outcome, even if that journey may meander. This may best be represented by a dotted line towards an indicative result. Reporting that pupils are on Step 1, Step 2, Step 3 etc. will be much less convincing.

7. A view that a pupil is ‘on target’ if they attain their predicted grade – even if that grade is unacceptably low
This issue takes us into the heart of target-setting, and a school’s views about pupil achievement. We know that prior attainment is a good indicator of future performance, and that we can estimate the most-likely grade a pupil will attain at GCSE from their attainment at the end of key stage 2.
This can reinforce the common view that learners, to a large extent, are destined to attain a grade indicated by their prior attainment in English and mathematics, even in subjects dissimilar to English and mathematics. This view then supports the expression that pupils will be working above, below, or at their target, i.e. the grade we expect from pupils given their prior attainment. The consequence of this is that a school might be perfectly comfortable for pupils who they consider as weak to attain a low GCSE grade – because they have met their target.

However, parents of pupils with low prior attainment would expect that a good school will, through effective teaching, attempt to lift their son or daughter’s performance and attain higher grades than might be indicated from KS2 scores.
A Mastery approach will support teaching which is responsive to how pupils are learning through the effective use of formative assessment and planned opportunities to revisit important themes. Mastery is an approach applicable to teaching and learning at key stage 3.  A key principle of using a Mastery approach is to try to ensure that the majority of pupils have understood what has been taught before moving on to the next topic. When using a Mastery approach, our aim should be for all pupils to be learning the essentials of the subject in order that they might achieve the highest GCSE grade they are capable of.

8. A lack of detail in showing how the teaching will cover all of the Subject Content statements
Many approaches to AWL say very little about the curriculum content for which the new assessment system has been developed. This may be a good example of an assessment tail wagging the curriculum dog.  A school may have a view that it can leave the curriculum for subject specialists to develop, and trust their judgment on their assessment of the outcomes. This is in spite of the fact that it is the evidence of what is taught and learnt that is the most important indicator of the work of schools.
This, however, would not provide a robust position if an external evaluator is seeking to judge the quality of a school’s curriculum and its assessment, because assessment is first and foremost a curriculum issue.
The true focus to developing an effective assessment system will be to ensure that the National Curriculum has been analysed into its component parts, that teaching schemes are developed to teach these components in a progressive manner, that there are clearly-defined learning outcomes to each unit of work, and that over key stage 3 all components of the Subject Content statements will have been covered.

9. A lack of clarity about what pupils should learn after being taught each unit of work
In the days when Levels ruled, we had Attainment Targets which described, for each Level, a set of sentences which stated what pupils at this Level should understand, know and be able to do. If we try to use statements like this to assess pupil progress during a key stage, the problem will be simply that we are not assessing pupils against what we have taught them.

If we are to place an emphasis on formative assessment, a better approach will be to define a Learning Objective for each unit of work that is taught, i.e. a demonstrable fact that a pupil now knows or understands something that they didn’t before. We would then be judging pupils on what we have taught them – not on some other criteria that will need to be inferred.  The initial question is: “Has the pupil mastered this unit of work, or not?” Mastery then provides us with four categories – Mastered, Secure, Developing and Emerging, with which to refine the answer to this question.
We can use these categories to record how pupils are responding to what they have been taught, which, over time will build a picture of the extent to which all pupils have assimilated each Learning Objective across a Scheme of Work.
Because we will have summaries of the detail of what pupils are learning we will be able to produce diagnostic reports which indicate the extent to which each Learning Objective has been mastered. Without this information we will be back to reporting that a pupil is on ‘5b’, or some other symbolic term which says nothing about what pupils have actually learnt or need to do next in order to improve.

10. Not taking the opportunity to implement a ‘Mastery’ approach to learning
The Mastery approach is the best piece of guidance about curriculum delivery and assessment in the AWL Commission’s report on the new, post-Levels world of Education.
A Mastery approach is no longer to be a hectic rush through a sea of content with pupils falling off at each stage. It is first of all based on sound curriculum planning which answers the question ‘What are the essentials of this subject which need to be grasped in order to support a rapid growth in learning and understanding in this subject? These are the ‘Key Concepts and Big Ideas‘ which we need to ensure that all pupils have grasped so that further learning can build upon them.

Mastery is not a one-chance-to-learn approach. It is about ensuring the grounding in the subject has been learnt by visiting the key areas more than once, and providing additional opportunities for those who don’t quite get it the first time around.
threepupilsMastery assumes that a majority of pupils could do well at a subject if the teaching was well-informed by formative assessment and there were opportunities to revisit topics. Mastery will also relate to target-setting, as described in section 7.  If a Learning Objective has been Mastered, then this component of the Attainment Target will be of a standard required to attain at the higher grades of the subject.  If all Attainment Targets have been Mastered then this will point to a learning journey towards the higher GCSE grades at key stage 4.  Being ‘Secure’ in assimilating an Attainment Target will point to a GCSE number grade above a 5; ‘Developing’ will indicate a GCSE grade below a 5, and so on.
In a Mastery system, where we can revisit the learning in order to encourage the greatest number to Master what is taught, then the logic is that we should be setting and expecting targets of the highest grades for all pupils with the potential to succeed.

Some approaches use Mastery statements to represent different standards depending on pupil expectation; for example, by saying that a pupil has achieved Mastery if they are on track to attain a low expected grade. This is an incorrect use of the term. Mastery of a component of learning is an indicator that this is a contribution to that set of attributes needed to attain a higher GCSE number grade. As all components are mastered, the evidence will build to form a confirmation of this estimate as the most likely outcome at key stage 4.

From Component Learning to Holistic Learning
High expectation will be a characteristic of learning at key stage 3. Key stage 3 is largely about building foundation knowledge to support studies at key stage 4.
As we move through years 7 to 9 the emphasis will move from learning the component parts of Subject Content statements towards demonstrating a holistic understanding of how these components work together, i.e. the extent to which an Attainment Target is becoming assimilated.

Year 9 will be a transition period from what might be thought of as component learning to what might be described as holistic learning.  Assessment in year 9 should provide increasing opportunities to demonstrate learning across the whole attainment target by drawing from component parts previously studied. By doing so, we will be placing an increased emphasis on the outcomes to learning over the key stage.
By the end of year 9, the planning should have ensured that all component parts of the Subject Content statements will have been taught and that there will have been assessment opportunities to assess holistic understanding and capability across all of the Attainment Targets for this key stage.

In this way, studies at Year 9 will take on more of the character of studies at key stage 4, which generally require demonstration of an increasingly critical understanding and ability to apply knowledge in a subject area. In key stage 4, GCSE Assessment Objectives will govern the approach to assessment, and grade criteria will be used to decide into which category, i.e. grade, the evidence of pupils’ application of subject competence will place them.

Projecting future success
At this point there can be a corresponding change of focus from a ‘Working Towards’ approach to a ‘Working At’ approach as we move from Component Learning to Holistic Learning.  This may seem disjointed but it is simply part of an overall picture of pupil performance made up of several dotted lines, each indicating a likely future grade. An estimate of future pupil attainment is something that needs to be inferred by several key items of evidence. Prior attainment is one indicator of an estimated future grade. Attainment 8 is another.  Pupils’ ‘Working Towards’ estimates indicated by recording the extent to which they are mastering what they are taught will be the most up-to-date indicator of potential attainment because they will be based on pupil responses as they occur.

Using Standardised Assessments
In the flight path below we can see the ‘Working Towards’ Mastery assessments as a set of green dots, cross referenced by a blue dotted line showing the estimated progress path based on attainment at key stage 2. There is a standardised assessment at the end of year 7 represented by a blue confirmatory dot on this line. This approach allows a school to only record standardised assessment information if it wishes, making ‘Working Towards’ assessments optional.

progress10
We will need to look at a range of evidence of pupils’ progress as a means to managing their learning as they move from year 7 to the end of year 11.
It is within year 9 that we will see evidence of pupils’ holistic understanding and application of their learning, and within years 10 and 11 that we will be able to assess this with greater accuracy using GCSE Assessment Objectives.

Neither ‘learning’ or ‘progress’ are simplistic concepts. We need to infer them from the outwards signs of pupils’ learning and from their responses to standardised testing. But taking a broader view of the evidence that will enable us to estimate pupils’ future performance doesn’t make the process of monitoring learning more complex, or data gathering more onerous – because if we choose them well, IT systems can potentially do all the hard work for us.

Data is the currency of a diagnostic profession

There is one other potential pitfall to be considered here. It is the natural response to a recent period in Education when there was a strong emphasis on school accountability, and a consequential over-emphasis on gathering data of every kind to show to inspectors. This was a major distraction to the real business of schools in ensuring effective teaching and learning. In consequence, ‘data’ has got itself a bad name in many quarters, along with the often labour-intensive processes of collecting, processing and reporting the stuff.

The scrapping of Levels has effectively swept away the slavish process of tracking Levels and sub-Levels, whilst not necessarily removing the anxiety that led teachers to feel they had to account for every detail that might indicate incremental improvements in pupil progress. It has taken a combination of the AWL Commission’s report into why we should stop using Levels, plus helpful clarification from Ofsted about what information inspectors might expect to examine during an inspection, to finally put data collection in its place.

We have thus seen a welcome period of rethinking the purpose and nature of assessment as an appropriate mechanism for providing the feedback necessary to guide the process of teaching and learning.
However, in celebrating the end of the bad old days of intensive data collection, we must guard against going too far the other way, i.e. to somehow think that data has been abolished along with Levels. Data is the currency of a diagnostic profession. It is what we need to collect in order to turn it into the information which will help us to understand the impact of our actions.  This applies to teaching just as it does to medical practice. As education professionals we need to engage with data in order to be master of it.

As with all things, it is a question of balance. We can be guided by the advice of the AWL Commission report in getting this balance right when evaluating systematic approaches to assessment. So, if the approach a school is using, or the IT system it chooses, or the procedures it has laid down, takes us back to the bad old days of slavish data gathering, then we should replace it with something better.

Is a truncated key stage 3 a good idea?
There is a particular issue existing in some schools which have extended their key stage 4 downwards to include year 9. The case for having done this has been debated on Twitter and there are arguments for and against. There has so far not been a spotlight shining on curriculum quality, especially on KS3, but Ofsted has indicated that ‘The Curriculum’ is to become a particular focus for their work. There is no doubt that well-planned and implemented courses at key stage 3 which provide foundation learning across subjects will be a factor in ensuring successful outcomes at key stage 4, and extra time learning exam techniques is unlikely to compensate for a curtailed key stage 3.

Steps towards overcoming the pitfalls
In hindsight, the quest for an alternative to Levels perhaps isn’t the problem that needs to be solved. A more tangible target would be to ensure a sound implementation of the new National Curriculum, and then to remind ourselves of how assessment has always worked.  That is, we teach something, and then we check to see if pupils have learnt it.  It isn’t really more complicated than this.

The question that we should be seeking to answer is “How can we implement a high quality curriculum offer when the new National Curriculum looks like a list of things to be taught – with no guidance on what progression looks like?” Different schools will see a particular logic in teaching the topics in a particular order – which will differ from school to school. Within that order they will describe and expect an accumulation of learning which will underpin a view of age-related progression.

So, the solution to AWL should start with sound planning by subject leaders. This may be about adapting and re-ordering the curriculum that a school already has at KS3, or starting from scratch.
Either way, it is likely to begin with a systematic analysis of the new National Curriculum Content Statements, and breaking them down into the learning attributes which underpin each statement. It would be unwieldy to have a scheme which tried to evaluate hundreds of learning attributes. They would be disconnected, and would not provide the essential evidence needed of the development of a holistic understanding of the subject.

By contrast, with a topic-based scheme of work, many learning attributes can be packaged into a teaching scheme made up of distinct units of work. It is the units of work that provide the assessment opportunities because we will know what we wish pupils to learn in each unit and we could express this as one or more Learning Objectives for each unit. These will be neither fragmented, like lists of learning attributes, or single numbers that need to be inferred, like Attainment Targets. Learning Objectives will be subject-related component units of competence, and they will build to make a proficient learner across the distinct specialisms of the subject.

Workload Issues
The approach suggested in this article requires detailed planning by subject leaders involving analysing the subject content statements of their National Curriculum subject, identifying the elements which will need to be taught, and designing suitable units of work over the three years of key stage 3. One or two Learning Objectives would be identified for each unit of work, and these would form the basis for making mastery judgments.
With this level of up-front planning, including the writing of Schemes of Work and identification of resources, lessons should be well-supported and require much less last-minute planning. This would be helpful to all members of a department, particularly teachers new to the school, and cut down significantly on preparation time.

Formative Learning
A significant theme running through the AWL Commission’s report was the importance of a focus on Formative Learning, i.e. the day-to-day interactions with pupils which build up a picture of how they are responding to what they are being taught. Information gathered through a focus on Formative Learning supports a Mastery approach by helping the teaching to be more responsive and individualised.
A brief, occasional, summary note on how pupils are responding to the teaching will form a useful informal record of pupil progress. It is how teachers’ markbooks have always been used. Such summaries, made within a framework of Learning Objectives, actually provide all the information needed for an assessment system to provide a clear picture of how well pupils are learning and what that learning is leading to.  Add to this an occasional standardised test and we can refine our estimates further.

Choosing an IT solution to support assessment
An important efficiency arises from the intelligent use of IT.
A snap poll on Twitter recently suggested that 80% of schools were using a ‘home grown system’ – maybe paper-based or using spreadsheets. How labour-intensive will these approaches prove to be, particularly with regard to aggregating information from many spreadsheets?

poll2

A well-designed IT system should make it easy to aggregate assessment information across a school – and process this information automatically.
If the IT system also does the data crunching, then teachers will not need to think about numbers – only what they are teaching and pupils are learning. This would be a valuable goal for implementing a good solution to AWL.

The characteristics of an AWL solution which avoids these pitfalls
We have developed a solution called a ‘KS3 Curriculum Design and Assessment System‘ because it follows a view that the solution starts with good planning of a curriculum by subject leaders.  The planning will describe the lessons that will be taught for each unit of work over the key stage. Then, for each unit of work, one or more Learning Objectives will be defined.  It is worth reminding ourselves that this is how assessment has always worked – i.e. we teach something and then see if pupils have learnt it.

This approach supports the analysis and design of a unitised scheme of work which supports incremental learning towards National Curriculum Subject Content statements. In the approach that we have developed, we have made a library of templates available (link), which have been populated by school users of the system, and these act as starting points for creating teaching schemes across subjects.
As the units are taught, all teachers will need to do – from time-to-time – is to capture summaries of formative assessments of how pupils are responding to the teaching of each unit. They will need to do this as a minimum – whichever system they are using, e.g. using an IT system, or a spreadsheet, or a class register. With this approach, teachers will record summaries to show how pupils are mastering what they were taught at a frequency of their own choosing.  There are only four broad Mastery categories to think about, so making judgments is relatively straight-forward.

These summaries are easily added to the system with a few mouse clicks – and that is all a teacher needs to do.  From then on, the system accumulates this information and works out which GCSE number grade each pupil is heading towards in their learning journey through the subject. The system can then plot Flight Paths for each pupil and provide an overview of the picture across subjects, and diagnostic reports to show how each pupil is mastering the subject content.

In summary, the key characteristics of this approach might be described as follows:

●  The solution to AWL isn’t really about finding an alternative to Levels
●  The AWL issue isn’t about numbers; it is about good teaching and learning
●  At the heart of the solution to AWL is the need for sound planning of the delivery of the National Curriculum
●  We shouldn’t try to assess an attainment target until all the components of it have been taught
●  Good criteria for assessment are the Learning Objectives for each unit of work in each subject
●  We should use an easy way to collect teachers’ summaries of Formative Assessment
●  Using four Mastery categories to quantify progress in each unit of work makes assessment easier
●  A ‘Working Towards’ approach signals future attainment and avoids ‘Working At’ fail grades
●  Evidence should be recorded across the key stage of both Component Learning and Holistic Learning
●  The system should add value to the data collected to forecast attainment, plot Flight Paths, and produce diagnostic reports


A presentation on this approach to Assessment Without Levels at KS3 can be downloaded from http://www.4matrix.org/awl.  From that web page we can also download an Executive Summary of  ‘Assessing Without Levels – Ten Pitfalls to Avoid‘,
mike_small
The aim of these article has been to share our journey through the potential pitfalls of implementing AWL with those who may be seeking a solution for their school. There will be other approaches available, and hopefully these notes will help to provide the criteria which schools can use to make their decisions about what will best suit their school.

Mike Bostock is an education consultant/trainer specialising in the use of performance data tools for school improvement. Mike’s company, New Media Learning Ltd, produces the 4Matrix School Performance System – http://www.4matrix.com.

mike_training3

 

Advertisements