Evaluating the impact of interventions on exam results – the dangers of flawed logic

Standard

Here is a question from a critical thinking paper:

http://www.ocr.org.uk/qualifications/as-a-level-gce-critical-thinking-h052-h452/

OCR Critical Thinking Unit 2 What’s the problem with the logic used in the argument?

The claim is that the improved exam results have been attributed to the new arrangements. So why is this problematic?

The problem is that whilst the new terms MIGHT have contributed to the results it is not logical to say they were fully responsible for them. In Critical thinking this can be referred to as ‘false cause’ (it may be a contributing factor but there are many more) or ‘post hoc’ (there is no proved link); the logic is flawed.

This is fairly serious when evaluating the impact of interventions in schools; in fact everything we do.

Can we be sure what has had a direct impact on results?  How do we know if specific interventions were successful?

Hopefully schools, subjects and teachers will evaluate interventions in some way. However this process has huge potential issues. If we take one student, in one subject and their one grade, how is it possible to analyse what led to this grade?  We can consider what we know happened in school. We will never really know the extent of external factors on this grade. I’ve blogged before on whose grade it is. If a child has had a private tutor and the school doesn’t know they could be falsely attributing interventions when the external support could have been a large contributor.  We also may focus on just the exam year or the exam course years, but can we evaluate the impact that previous key stages may have on results?

Pitfalls of using flawed logic

  • Some interventions may have no/little impact yet as the results have always been good when they’ve been done it’s almost tempting fate not to keep doing them. (Irrelevant appeal to tradition; we’ve always done it this way.)
  • Copying what other schools do because someone has publicly shared what they think got them good results. (Hasty/sweeping generalisation ; going from one school where it worked, applying it to many where it might not)
  • Spending money/time on the wrong thing
  • We want to believe that we (a teacher & the school) have had the most positive impact on the results; we might not.
  • The assumption that all actions/interventions are measurable.
  • Assuming that there is some sort of panacea out there; other than decent teaching there probably isn’t.
  • I’ve listened to many leaders from schools stand at conferences casually attributing their success to one or two things they’ve done. Their logic is flawed with the ‘cherry picking’ fallacy. They’re only telling you the ‘good’ bits they want you to hear; what did they not tell you about?
  • Judging a teacher/school on the data. Can those that do this ‘prove’ they’re fully responsible for the exam result?

Possible ways to overcome flawed reasoning to make evaluations accurate

  • Ask the students. Whilst this isn’t always reliable, if anonymous it can be useful. How many schools deliberately ask students if they’ve had professional external support out of school?
  • Don’t credit one intervention highly over another.
  • Remembering that excellent teaching through key stages will generally be the best way to achieve good results; maybe with this we don’t need spring term interventions with year 11, it should happen from the moment we see the students.

In case you’re interested, here’s another one:

http://www.ocr.org.uk/qualifications/as-a-level-gce-critical-thinking-h052-h452/

Question:    Answer:

Advertisements

The role of keywords in assessing knowledge & understanding

Standard

As part of my thinking on assessment I’ve been looking at how to assess knowledge & understanding. I’ve been considering stages of knowledge and whether it is possible to use this to ensure students’ thinking is based on solid foundations and then stretched further by more complex concepts.

One area I’ve been struggling to structure is the use of keywords. I added ‘keywords’ into all the stages and thought about how the vocabulary that is used in more complex concepts will differ.

I’m now considering how the use of keywords in themselves can be an indicator of knowledge & understanding.

I will try to explain using an example from my own subject area.

The new GCSE subject content gives the following for knowledge & understanding relating to the crucifixion of Jesus:

  • incarnation, crucifixion, resurrection and ascension
  • salvation, including law, sin, grace and Spirit, the role of Christ in salvation, and the nature of atonement

These are key concepts that students need to understand to truly understand what this event in Christianity means for Christians. 

I wil take the key concept of ‘atonement’. 

If we go from the simplest way to understand this concept, using foundation knowledge & vocabulary then we can see how the complexity of this concept may be broken down, as follows:

  1. Saying sorry for something you’ve done wrong (apology)
  2. Saying sorry and someone forgiving you (forgiveness)
  3. Sinning and asking God to forgive you (repentance)
  4. Sinning and asking God to forgive you, knowing that Jesus dying enables this to be accepted (salvation)
  5. Reestablishing (‘covering’) the pre-sin relationship between man and God ‘At-one-ment’ (Atonement)

Without an understanding of the prior concepts and keywords it would be difficult to understand the final concept of atonement. 

So how can this be used in teaching?

My new plan is to come up with the key vocabulary (alongside threshold concepts – another blog) for each topic or unit of work. I can then use these to check if the students have understood before I move on. This could happen in a lesson or over several lessons depending on the students foundation knowledge & understanding.

This checking can be done at the beginning, at intervals and/or at the end. Ideally as frequently as possible but realistically at least at the beginning and end.

This could be done in several ways:

Questioning – if I ask a student “what will God forgive?” They have to understand the concept of forgives to give a correct answer. Regular, targeted questions should give a good idea of what they do/don’t understand.

Written response – ask students a question, write a statement that means they have to show which concepts they know and understand. For example, “Salvation is possible for everyone” requires them to understand what it is to decide whether it can apply to everyone.

Multiple choice/diagnostic qu – I’m trialling this. A quick way to see what they do/don’t know and understand.

‘Final assessment‘ – however this is completed students will be told they must use the keywords learnt. They can have a list of them. It’s not a test to see if they remember the word, it is checking to see if they fully understand it enough by using it correctly in their work. Setting self differentiating tasks on this means all can achieve and show what they’ve learnt.

These keywords are essential in my new assessment systems as they are a key part of seeing if a student has understood the content.

This has already been highlighted in a student essay on marriage that looked great in terms of structure but had no key terms for the topic in it; it was essentially a sociology essay.

To avoid this from now on all assessed written pieces will have a set of student generated keywords as a part of the essay planning process.

Pre-submission proof reading – student checks

Standard

Having heard on Twitter that some teachers are ‘refusing’ to mark student work until it has been proof read I started to think about my own practice.

Whilst I mark I would say that 70% of the actual pen on paper marking I do is spent on:

  • ‘Write the correct title’
  • Underline title (I have a stamper I got so sick of it)
  • Sp (spelling)
  • Cp (capital letter)
  • // (paragraphs) or ‘where are your paragraphs?!’

So I’ve decided I won’t be doing this much more. There are several ways of doing this but I’ve gone for a simple pre-submission check-list that I put on the board at the end of the lesson.

Checks

As you can see, I’ve decided for the students to have to confirm they’ve checked through the list by writing the final sentence at the end of their work.

With the first group that I did this with most students seemed active in editing their own work pretty quickly which is a great start.

I’ve also started doing a ‘keywords’ section in their essay planning. We’ve done this as a group so all students have the correct spellings of the important keywords in their books. This should mean that any spellings should be general spellings like connectives.

To put this in context I showed two of my groups this:

ks1 criteria

And asked them how old they thought the Government thinks you have to be to do this.

It’s criteria for end of key stage 1 (7 years old). There were gasps. My expectation will be that they always do this and after a while I may not have to put up the check-list on the board at the end of the lesson.

My success criteria for this new approach is the stamper becomes redundant and my time is mostly spent on asking developmental questions for them to complete their improvements on.

So for those of you that do DIRT or similar, maybe consider whether the work that is being submitted can be improved pre-submission. It may save you time and energy and enforces high expectations in written work. Well, at least key stage 1 expectations…

Context is king

Standard

Over the weekend Nicky Morgan made comments on how she proposes to deal with ‘coasting’ schools. She confirms that she means those that ‘just’ get enough to be RI or those where ‘every child doesn’t make progress’.  Whilst we could argue about the semantics of ‘coasting’ I fully agree that schools that seem to consistently be 3/4 need to be sorted.

However, throughout this all there is an implication that a ‘one size fits all’ approach will work. This is where there will be problems.

The schools that Nicky is referring to have huge challenges;not just getting better results. We’re not talking about the Outstanding school that drops to Inadequate overnight due to their lack of British values. We’re talking about schools that are usually in socially deprived areas or unique social situations. Recruitment of teachers is hugely problematic ( more than other schools) and the poor reputation of the school usually comes before any of the good.

But these contexts are not new. They’ve been this way for years and years. One of them used to be on the largest council estate in Europe. The crime rates and deprivation are nothing new. Some are in areas where they are guaranteed work in local industries regardless of grades. Some have low aspirations.

The problem is that these are rarely taken into account. I don’t mean make excuses. I mean take into account the context of the school and then deal with it head on. Telling these schools you will become an Academy means nothing unless the people that ‘take over’ understand and can deal with the intricacies of the context. One academy chain that seems to be working well in one area of the country may have absolutely no idea how to deal with this next school. And we’ve seen this happen locally. Some are now into their second sponsor. The first having no idea how to deal with the social issues the schools presented. Coming along with great pedagogical ideas and visions that just don’t work in this context.

I’m passionate about this. I’ve worked in these schools. I’ve seen students achieve great things in some subjects but not in others. They’re not impossible to teach. They need the right teaching for the right context. We need teachers and leaders that know how to get the best from these students. Not leaders who once got a job there 10/20 years ago and are sitting pretty, doing very little on 60K and retiring on it. Or a leader from the other side of the country who’s only worked in a completely different context.

So when Nicky sends these new Headteachers and NLEs in to rescue these failing schools what are the chances they will have experienced these contexts before? NLEs have to be from good/outstanding schools (or those from a category up to these). How many Headteachers does this actually apply to? And are they the ones willing to move to help out another school? With great respect to Headteachers that have always lead Outstanding schools I’m almost certain there’s only a handful of these that would know how to deal with these school contexts. Skills transfer but context doesn’t.

I’m sick of seeing the same local schools bounce between 3 and 4. Even worse when the Secretary of State for Education comes to praise it one minute and the next it goes into special measures. Let’s get real here. These schools need targeted, specialised help, in some cases some of the current staff may be able to do this with leaders who know what they’re dealing with. In others, most of the leadership needs replacing.

I can’t stand any more local headlines where new Heads say they are ‘turning around’ the school or ‘vow to make it outstanding’. Let’s get on with the real job; recognising the context & putting together a strong team to manage it.

Either way a ‘one size fits all’ approach does not work. Becoming an academy doesn’t work. They already are.

Context is king.

It’s not the only way – the lies and myths that are fed to teachers & what can be done about it

Standard

People that follow me on Twitter & elsewhere can tell the past two days have been interesting.

I had an interesting discussion with @llewelyn20 on Twitter a few days ago. I posted a matrix that someone had shared of what they’ve been told Outstanding teaching looks like. I genuinely wondered how many people were still using these.

Whilst I understand the sentiment (I’m sick of it too) I don’t agree with not highlighting the realities of what is being told is good practice in schools. It’s wrong and if we know it is wrong then those of us that know it should share it. Whilst we may be fed up of hearing and seeing the bad practice & myths, I know that Twitter has enabled me to find out all sorts of truths about education that I would have been at the mercy of my SLT colleagues if not.

However I also hear anecdotes of what is going on in schools across the country (interestingly not on Twitter – are people on Twitter some how different? In the know?) running themselves for Ofsted. Implementing unfair, illogical and in some cases damaging systems. Examples include, telling the staff lessons won’t be graded and then secretly keeping a spreadsheet of grades that have been deduced from paperwork, telling good/outstanding labelled teachers when they’ll be observed but  teachers labelled RI won’t be told, using a check-list to decide if your lesson was outstanding; the list goes on.

Why do some leaders perpetuate these myths?

Sadly, I’m being told by people that it is consultants and ‘mocksted’ teams telling SLTs that this is what Ofsted want and they believe it 100%. These middle leaders and teachers are told it and they trust their SLTS. Why wouldn’t you?

It really concerns me that some people in senior leadership roles have to rely on external advisers to tell them how to run their school. Surely, you’ve been promoted because you can see what ‘works’ without someone telling you?

There is also an element of ‘here is the panacea to make a good school’; use this grid, do learning walks, do mocksteds and unannounced lesson observations. Leaders are scared. Some will take anything they can. Irrespective on the effect it has on teachers themselves. What they fail to see is there is one thing that IS the panacea to a school’s success, excellent teaching ( I mean teaching in a wide sense). However these seem to only ‘weigh the pig’ ( I HATE this phrase as it was used by someone who only ever did it). If these schools spend a disproportionate timing monitoring & observing compared to supporting and developing their teachers then it just won’t work.

Sadly some schools that are outstanding are using these methods and use causation rather than correlation to justify them; “we use them and we are outstanding so they must be good systems. Change might mean a change of grading and therefore we must use these systems.”

Some genuinely don’t know how to make things work. Their strategic leadership & managerial skills aren’t developed enough. They’ve been over promoted. Instead of realising this and working on it, they flounder and take any sort of advice and use it to grab on to, to look like they’re doing the right things.

Concerns for the future

My big concern throughout this is that if teachers and middle leaders believe that these ways are the best way to raise achievement and school performance that they then go on to be like this themselves. Relying on external advisers, believing what they’re told and in some cases implementing crazy systems that have little logic and questionable evidence of success.

Teachers that have only ever worked on one school, then as a middle and then senior leader have no idea what really works. They’ve only ever seen one context. They don’t’ know that what they’ve been fed over the years isn’t actually reality. They perpetuate the myths.

I am so lucky to be in a school that doesn’t do these things but I have been in schools that do. My lifeline has been Twitter & great leaders in the RE world like Mary Myatt, and people in the world of education, for example Mike Cladingbowl & now Sean Harford, who have kept me going & and have told me the reality. So, whilst it may be painful for those in the ‘know’ to watch, I will continue to post about these myths and dated systems that Ofsted do not demand. As said above Twitter seems to be full of those who know better and get frustrated when it’s pointed out that not all leaders are like them. I’m not leadership bashing; I desperately want decent leadership in all schools for all teachers and all students. I know and have worked for/work for some amazing leaders.

So if one person following me is helped by my tweets/posts dispelling myths then it is worth it.

Critique in RE

Standard

These are my opinions of what I call critique in RE. I haven’t read any pedagogical stuff on this. It’s just from my experience and own ideas, much of which has come from teaching Critical Thinking A Level.

Critique vs criticism

There is a subtle difference between these. Criticism is about directly commenting on a religion and it’s beliefs and teachings and saying if it is ‘good’ or ‘bad’; whether you agree with the actual teaching.

For example, Christianity isn’t true because it’s impossible for any human to be resurrected after 3 days.

Critique doesn’t require a student to evaluate if a a religion is ‘true’ or ‘right’ but to consider key issues in religions and the wide range of opinions there are on them.

If you want to see criticism of religion follow Richard Dawkins on Twitter.

Critique however is about looking at the beliefs and teachings and considering what questions need to be asked in order to understand them and if the response to these are ‘valid’.

Critique

The questions that inform critique come under several areas:

  • Reliability
  • Credibility
  • Logic of reasoning

these can be broken down into further detail to help students understand and try out simple critique.

Examples

  • Reliability – Is the source reliable? why?
  • Credibility – Is the source credible? Are they biased? Why? Do they have expertise in this area? Do they have a vested interest? It is affiliated to a religion? Does any of this impact the reliability?
  • Logic of reasoning – Does it logically make sense? Does it make assumptions? Does it attack the arguer instead of the argument itself? (Ad Hominem), Is the argument flawed?

A teacher can simplify these when questioning students in class, for example “Why might he/she say that?”. The more a teacher gets student developing their critical thinking skills through questioning and answers the more they should be able to get this embedded into their written work.

More questioning

Developing students’ questioning skills is and important aspect of developing good critique. I’m thinking of using and maybe developing this questioning grid to ensure students can ask a wide range of questions.

John has also developed one here that allows students to track which they can use well.

Pre-published critique

This is also where students can engage with what has already been said about the particular topic they are thinking about. This may be of philosophers, theologians, historians, religious leaders etc The quantity and complexity of the views will depend on the student; I differentiate these.

How I also encourage a use of anything published, usually online. A particularly useful resource for this is news articles which have a comment facility at the bottom. There are generally a wide rang of opinions and views and there are several activities that students can do to utilise them in their work.

I’m also trying to develop a culture of research with the students. It’s early days but putting the responsibility on them to find differing views is great study skills particularly needed at A level. I’m happy to share how I’m doing this as well.

So critique can be three-fold in students’ development; being able to ask the right questions, being able to respond to the questions with the correct use of language, being able to research, understand, use and critique pre-published critique.

The language of critique

It is essential for the language of critique to be taught; it’s the language that can make it criticism instead of critique.

Giving students a bank of phrases and vocabulary to use when critiquing helps them to focus on the correct aspect.

Who can critique?

I teach secondary but believe that critique an easily be taught to primary students. The earlier we start the better.

Putting it all together

I am getting students to pull this all together in written arguments. The plan is to get at least 3 of these in the year, with in between assessment focussing on a specific aspect, which could be critique.

I’ve decided to pull this down into key stage 3 as I think these are essential skills for GCSE and A Level (see red below where I think critique comes into play)

GCSE draft assessment objectives

AO1

  • Demonstrate knowledge and understanding of religion, including:
  • similarities and differences between and within religions and beliefs;
  • the nature of religious beliefs and teachings and their impact on individuals, communities and societies.

AO2

  • Analyse and evaluate questions and issues related to religious beliefs, values and teachings:
  • using and applying knowledge and understanding of religions;
  • constructing well-informed and balanced arguments.
A level draft assessment objectives

AO1

  • Demonstrate knowledge and understanding of religion, including:
  • religious belief, thought and relevant issues;
  • explaining the nature of religious beliefs and teachings and their impact on individuals, communities and societies;
  • making connections across different aspects of the study of religion and belief.

AO2

  • Analyse and critically evaluate questions and issues related to religious beliefs, values and teachings:

  • applying knowledge and understanding of religion;

  • using evidence and reasoning;

  • constructing well informed and balanced arguments

And of course, this highlights the importance of knowledge and understanding. You cannot critique effectively without initially understanding the issue. I really encourage students to prove they fully know and understand an issue before they offer critique otherwise it becomes ignorant critique which can be dangerous. Critique without knowledge of religions is simply critical thinking or maybe sociology/history. It is essential that the religious knowledge is equally important. Knowledge and understanding are also key aspects of the assessment system I’m developing.

If you’d like any specific examples of how I’ve used critique then let me know and I’m happy to share.

Looking for learning progress in books – sorcery or a skilled art?

Standard

I really struggle with the concept of seeing learning progress in books; especially those that aren’t your subject or from another school.

I think I would be able to pick out some sort of change in writing in student books that I don’t teach within my subject area but progress of learning? Without discussing what the work was with the student or teacher I’m not so sure.

There are some things that I think you can see in books:

  • If school/dept marking policies are being followed
  • If incorrect SPAG is being picked up on
  • If students are acting on feedback (Is this progress of learning?)
  • The different levels they have achieved (going ‘up’ doesn’t always mean progress?)
  • If students are taking care with presentation
  • If students are making the same errors or not (simple errors such as capital letters)
  • If their written style is improving ( possibly progress for literacy/English but not directly for other subjects)

But I struggle to see that this shows progress in their subject learning.

If we are teaching them different topics in succession that require different knowledge it is very difficult to see progress over time with knowledge; the knowledge needed for Buddhism is different needed for Islam.

The only way that I can try to display progress in their learning is if there are explicit points made to show it off. For example, a test taken at the start and then again at the end of the learning period; with more marks on the 2nd test. Or asking them to explain what  they’ve learnt in the topic. Are these truly representative of what they’ve learnt?

If we’re looking at writing some may say ‘their writing is more extended’, but there are times I say to my top set they can only write one page ( or they’ll write 10). Then it doesn’t look more extended.

Drafting and redrafting seems a sensible way to show development in their writing, but is there then pressure on a teacher to triple mark? Maybe we should spend several lessons on one piece, crafting it to excellence than several that may not show much progression?

Part of a puzzle of evidence?

This phrase has been used to try to explain to me how books help form part of a picture of learning. My concern is that if an Ofsted team spend 20 mins doing a book scrutiny how could they possibly have enough time to investigate the ‘whole’ picture of one child let alone a pile of them?

Isn’t there a large probability that a misinterpretation of one book linked with a misinterpretation of a lesson observation lead to a completely incorrect judgement of learning?;the wrong puzzle being pieced together?

Books don’t tell enough of a story. They don’t tell you where that student was in their learning at the start of the year, last year or the year before. They don’t tell you that for that child, that messy paragraph without capital letters was the best work they’ve ever done for you. They don’t record the conversations ( some teachers are made to do this) that you have with students, that a 30 minute observation by an inspector may not see. There are too many possibilities for misinterpretation unless discussed with the teacher.

Any sorts of tracking sticker or graph on their book may well easily be misinterpreted without discussion and we all know that learning isn’t linear anyway. How do you know that the levels recorded weren’t for different skills, some being more difficult? Is it true that only subject specialists could see this?

A whole new world

My final concern is the new assessment systems being used by different schools and different subjects around the country. Unless I explain my assessment system to someone looking at the books, how will they know how I’m assessing and why I may get students to write in a particular way? If an inspection team come and look at books that are used in a specific way without any explanation is there a danger of the misinterpretation above?

Playing the game?

When I look at books within my department, it is a two-way process. I ask my colleague questions. I want to find out what is going on with a student and their work. But many people who look at the books won’t have this luxury.

So the question is, do we create systems that jump the hoops? Create stickers saying “I have learnt…”? using stampers to say “Miss Cox said….” for oral feedback? make a sticker saying “books checked by Miss Cox” to show I’m doing my middle leader job?

Or just carry on and hope the outsiders looking in are better at the skilled art rather than practising some unknown sorcery?