Posts Tagged assessment

Assignment marking: Do the math

Something I’ve been meaning to blog about for a while is the unseen labour that goes into marking student work.

It’s semester 2 marking time in Australian universities, and I’ve just finished a stack of mine. ‘Stack’ in the figurative sense, because these were a combination of learning logs and and video blogging, all submitted and marked digitally, so there were no actual stacks of anything.

Being the audience and assessor for these students’ work was a privilege, and I don’t think any teacher should forget that having the authority to do this work is always a privilege. Sometimes it is also a joy. And it is always something that we do, knowing the important positive impact that quality feedback has on student learning.

It is a labour of love, but it is a labour to be sure.

Generally a student assignment takes 30 minutes to mark. So they say. Once I get my hand in, I can usually get through an exam response in 20 minutes (they don’t tend to require any feedback), and an essay in 30 minutes, but a set of professional plans (e.g. annotated lesson plans, units of work) takes about 45 minutes and you just can’t rush it.

A typical formula for university marking in my field is that formal assessment feedback and grading for each student should get an hour of your time each semester. That’s 30 minutes for each assignment if you only set two assignments. If you want to set more assignments, it’s on you to mark them quicker. If you’re in the edu-biz, you’ll know that this is where group presentations and short response exam papers start looking attractive.

In a typical semester I have 120 students. That’s maybe 90 students in one big unit, and 30 students in a smaller unit. As a high school teacher this was also roughly the number of students I had – roughly five classes of 25 (some with 30 students, some with closer to 20, e.g. senior classes).

Using me as an example: I set two assignments each semester. And we know that on average I plan to spend 30 minutes on each.

My semester runs for 9 weeks (because it’s followed by prac.), but they can also run for 13 weeks. You can’t really set an assignment in weeks 1 or 2. If the assignment is big, worth 40% or more, you can’t really set it in weeks 3 or 4 either.

Let’s say assignment 1 is submitted in week 5. We’re expected to get work back to students in 2-3 weeks. So they say. Which puts me giving their results back to them in week 8 (a very important deadline if the next assignment is due in week 9).

120 students

x

30 mins each

=

60 hours extra work

/over 3 weeks

=

20 hours of extra work each week.

Add about 3 hours for each of the following:

  • getting your head around the task and long times spent on first few tasks marked
  • moderation with a colleague
  • administration of grades, uploading feedback to LMS etc.

Rinse and repeat just one week later if you have set assignment 2 to be due in week 9.

So if you’ve got about 120 students on average, and managed to keep yourself limited to your 30 minutes per assignment in all units in both semesters, then you will have worked about 23 hours of overtime for 12 weeks out of the year.

I say overtime, because the whole time you’ve been doing this, life, and other work, goes on…

Classes still need teaching. Emails still need answering. You still have to front up to important meetings. Research papers still need writing, grant proposals still need submitting, you may be collecting research data and attending conferences too. If you’re a school teacher, it’s classes, emails, meetings, lesson prep, school dance supervision, year 8 camp, sport coaching, bus duty…the list goes on.

As depressing as this exercise is, I think we should all do the math on this for our own teaching context.

People need to know the reality of what teachers mean when someone asks them how they’re going and the only response they can muster is a stoney-eyed “I’ve been marking”.

Spouses and family members need to be acknowledged for how they support the teachers in their lives during marking seasons.

Teachers need to grasp the reality of their workloads so they aren’t taken by surprise each time the overtime cycle hits, and help each other learn how to manage the physical, mental and emotional toll it takes (or collectively rise up and change this system maybe, hey?).

And beginning teachers need to be aware of what they’re in for.

I am so grateful to my boss for giving me lighter teaching load this semester (just 35 students!) so I can focus on my research publications, but next semester I’ll have 120 students again. I can’t wait to meet them, but I sure do wish their assignments would mark themselves!

, , ,

Leave a comment

If students designed their own schools…

Chatting in the mid-year break with Bianca and some other PBL-peeps, this video was recommended to me. It’s only 15 minutes long, and now I’m recommending it to you too:

The video shows what can be done in a school where teachers and leaders are prepared to really let students design their own learning. Like, really let them do it.

The students in this alternative academic program design their own Independent Learning Projects (that they report on weekly to other students), as well as their own Individual Endeavours (ambitious term-long projects, e.g. learning to play the piano and putting on a recital).

Something that interested me was, about 1 minute in, one of the students explained that in the course they look at “the four main bodies of learning”:

  • English
  • Math
  • Social Sciences
  • Natural Sciences.

Make no mistake – I was totally inspired by this video and even showed it to my students this semester. So inspired, that I changed our first assignment to be based on completion of an Independent Learning Project! But when those four areas are offered up as the “main bodies of learning”, I can already see points of tension for making this kind of program work across the board. What of the other learning areas? What of health and physical education? What of the arts? Foreign languages?

Without engaging with conversations about what is ‘essential’, ‘core’, or ‘fundamental’ in education – and working out some kind of common goal or philosophy to anchor us – I suspect alternative programs like the one featured here will (continue to) struggle to gain traction.

Although these programs aren’t (yet) the silver bullet we need to shed our teacher-centred shackles, I believe bringing these approaches into our teaching is vital.

Personal take-away thoughts:

  • Students have passions and interests that they are entitled to pursue.
  • Students are capable of designing their own learning, if we give them some parameters.
  • Students are more motivated to learn when they have some control in devising the questions for investigation.
  • Independent learning approaches seem an immediate good fit for students like this (this is is a class of nine Honours students, who self-selected into the program), but would disengaged or recalcitrant students need more scaffolding?
  • Doing my own Independent Learning Project in high school was a transformative experience for me. It was called a ‘mini thesis’ by my teacher, and I chose to study the French Revolution. I did this for just one term in just one subject – surely this is achievable across the board without rethinking our whole approach to schooling?

, , , ,

Leave a comment

Assessing soft skills in PBL

This week in class we explored the Essential Fluencies as an alternative set of ‘soft skills’ to the General Capabilities in the Australian Curriculum.

One of my students followed up this investigation with the following juicy question:

Essential fluencies seem to structure skills within select criterion, however I am curious as to whether PBL uses these as guides (depending on the student’s PBL objective) or whether students are meant to meet all of these at different stages of their PBL (to achieve a final product)?

If this is a flexible criteria, would using a feedback grid be the most effective way of communicating the development of an idea (as it focusses less on curriculum goals, more on constructive advice)?

I decided to post my answer to part of this question here on the blog:

You’ve asked a good question about skills and standards. My understanding of PBL (and other inquiry-based models) is that assessing skills is just as important as assessing content knowledge.

There are two (opposing) axioms that relate to this:

  1. ‘What gets measured gets done’.
  2. ‘Not everything that matters can be measured; not everything that can be measured matters’.

At the moment I’m inclined to agree with the PBL movers and shakers – that developing ‘soft skills’ should be seen as a vital curriculum goal, just as important as the acquisition of discipline knowledge and technical skills. The argument here is that if we don’t find a way of measuring/assessing soft skills then teachers will continue to sideline them. Because ‘what gets measured gets done’.

The BIE crowd have developed a range of assessment rubrics for the four skills that they identify as most important to PBL specifically: creativity and innovation, presentation/communication, collaboration, and critical thinking. You can find them here:

http://www.bie.org/objects/cat/rubrics

Of course, the opposing view is that such assessment rubrics lead people to forget the second axiom ‘not everything that matters can be measured’. I know sometimes I’ve watched presentations for example that are awesome, but their awesomeness can’t be explained using the BIE assessment rubric. It’s like all rubrics actually need a criteria labelled “X factor!” for when a piece of work or project does something amazing that we didn’t plan to (or cannot) measure. And sometimes by focussing students so explicitly on assessment rubrics, they can get obsessed with how to ‘game’ the criteria to reach the highest standard, rather than taking risks in their learning to work toward a big-picture goal.

Opposing axioms.

Opposing axioms.

As there is no ‘Ultimate God of PBL’, we are free to use whatever framework we want to think about “soft skills”. We can take up the Essential Fluencies, we can take up the skills foregrounded by BIE, we can use the 4Cs proposed by p21.org, or we can use the General Capabilities from the Australian Curriculum.

But ultimately I’d argue that yes, whatever framework you choose, you should find a way of explaining to students the standards you are looking for on a range of criteria, for the particular project they’re working on. Assessment rubric sheets should be designed to make the criteria and expected standards transparent to the learner, and to aid the feed-forward process throughout a project as well as the feed-back process at the end of a project.

*****

I know I haven’t answered all of the parts of this student’s juicy question, and we’ll be talking more about it in class. It may generate another blog post. In the meantime…

  • How would you answer this student’s question?
  • Do you agree that providing assessment rubrics for soft skills is useful for learning in PBL (or otherwise)?

, , , , ,

6 Comments

Slaves to the Grade

It’s that time of year. Teachers of Year 12 around Australia are scrambling to varying degrees to prepare students for final assessments and exams, which inevitably involves a whole lotta marking.

Of course, all teachers have to grade student work. And they are engaged in doing this all year. But nothing beats the pedal-to-the-metal feeling of marking Year 12 practice tasks in a last ditch effort to refine their examination responses.

In particular, nothing beats the hellish pressure that exists in states like NSW and Victoria where the HSC and VCE exams respectively loom over teachers and students alike. And out of all these teachers and students, I argue that subjects that are writing-intensive (e.g. English and History) have it the toughest; if you have a class of 25 for Year 12 and it’s coming up to an assessment, teachers in these subjects are spending their nights and weekends correcting pages and pages and pages of long form expositions.

Which can leave your eyes (and soul) feeling kinda like this:

bill crying blood

I was prompted to write this blog post after watching my friends Justin and Alex tweet about their marking yesterday:

twitter convo 24.08.2013 HSC marking

I’ve taught for the HSC three times and this slavish marking routine is the only part I do not miss…having said that, the jolly task I have now of marking as a university lecturer has involved marking binges that certainly rival the pain of HSC workload.

The question is – what can we do about it?

Is there anything we can do about it?

Some ideas that I threw out into the twittersphere yesterday seem promising, but without a class to try them on I’m at a loss, not sure if they would work. The ideas I bounced around with Justin and Alex were:

  1. Focussing on writing just the introduction, or a body paragraph. This would make the task smaller and more focussed for students, and more manageable to mark 25-30 of them.
  2. Setting a paragraph writing challenge. To address Justin’s problem of the student that only writes about ‘tone’, each week set a different language feature/form for students to write a paragraph on. By the end of the term they will have a bank of paragraphs on different elements.
  3. Gamify the writing process. This could be done by putting students in groups, getting every student to write a paragraph (or essay), then each group submits it’s best one (as judged by the students in the group) for marking. This means you only have to mark one essay/paragraph per group, not per student. Keep a chart of which group wins each week and award them a prize at the end of the unit. Change the groups around for each new unit.
  4. Peer assessment. This can only be used in a limited way, as students don’t have the capacity to grade work to a Year 12 standard. However you could use the ‘medals (feedback) and missions (feedforward)’ framework that Bianca draws on to give students a direction. I think the main benefit is that they read each other’s work and discuss their strengths, not that they actually give each other a ‘grade’.
  5. Find an authentic audience. Partnering up with another teacher/class would provide an avenue for students to share their work with another class on a platform such as a wiki. This would give students someone to perform for besides their own teacher, which could prove motivating. The teachers could also arrange to do a marking-swap, and grade each other’s student essays…this may get you writing less comments, marking more objectively (?) and just plain old provide a change of pace as you get to read a different set of handwriting!

I really hope these ideas are useful to someone out there.

If you have any other good ideas for getting feedback to students without going through so much of the eye-bleedingly painful million-essay marking process, I would LOVE to hear them!

Thanks to Justin and Alex for inspiring this post and helping me brainstorm ideas🙂

 

Images: Cropped screen still from True Blood, Season 5; Screen shot of conversation on Twitter.com

Postscript: If you liked this post, you may also like the post Matt Esterman wrote today, ‘The home stretch for Year 12’. Looks like we all have Year 12 on the brain this weekend!

, ,

8 Comments

A ‘rhizomatic’ take on Semester one so far

Last week I had the good fortune to hear Professor Diana Masny speak about her Deleuzian approach to researching multiliteracies theory (which she referred to as ‘MLT’). Masny is from Ottowa, Canada, and is an adjunct prof at QUT.

In this presentation I was returned to the idea of ‘the rhizome’, something that had interested me when I encountered the work of Deleuze and Guattari. The idea behind looking at things rhizomatically is that we can stop focussing on binary oppositions, or organising concepts into ordered taxonomies and such. Instead, rhizomatic analysis involves looking at things and ideas spread/propagate…and at where new possibilities ‘shoot off’ out of of what already exists.

A rhizome in plant form

A rhizome in plant form

This talk by Masny was interesting for a number of reasons to do with designing research methodology, as well as considering MLT from new angles. One thing that inspired me was the way that her presentation was organised around ‘entry points’ to her own topic as a rhizomatic collection of findings. This is in contrast to a presentation that tries to summarise ‘key findings’ or ‘ways forward’. Seeing as I most often use my blog to reflect on ‘findings’ and ‘planning’, I thought it might make a nice change to adopt Masny’s (after Deleuze’s) approach of exploring the ‘entry points’ into my practice so far this semester…

ENTRY POINT: Attendance

At QUT we have a policy that attendance is not to be counted in any way toward assessment, and that students choosing to catch up on their study from home are to be supported in that choice. I have heard some lecturers complain about this – they think students would learn better if they turned up to all the classes, and wish the university would enforce this. Most of us, however, respect the purpose of this arrangement, which is to provide flexible study options for the grown-up human beings that are our ‘students’, and cater for a range of learning styles. Personally I find it very motivating, as it forces me to think about HOW I can make my lessons “worth coming to”!

I’m really happy with the attendance rate in my classes at the moment. Out of the 110 students I have studying on campus, almost 100% turned up in Week one, and the students that were away mostly emailed in their apologies. In Week 2, attendance in tutorials and the lecture was down to about 85%, which is to be expected. What I am eager to see is that 85% attendance rate maintained for the rest of the 9-week semester, rather than drop of over time to 20-50%, as other lecturers often report. I’m pleased to say that in the past few years here, I haven’t noticed the same kind of drop of, and I like to think this reflects the usefulness of my classes.

ENTRY POINT: Engagement

As always it has been a slow start on Twitter…but as always, there are several students ‘coming around’ to the tool already and engaging with informal peer tutoring as well. Once again, I am glad I chose to persevere with introducing students to an unfamiliar (and for many of them, unloved) social media tool.

I had a really great out-of-context engagement moment as well last week, on Pinterest. I use Pinterest among other things to collect useful resources for English teachers, and one day I saw a collage about English teaching and ‘re-pinned it’ to my board. I thought (and commented) ‘wow…this is just like an activity I do in class!’. Then I realised that I was following one of my students already, and that it was her! Funniest bit was though, she had been following me too without realising who I was, or making any connection to out uni lives. Good times!

There has been a growth in socia media profiles and ‘chats’ that I can now connect my students to, and the most important of these is the #ozengchat that takes place on Twitter on Tuesday nights. Feeling like they are engaging with ‘real teachers’ seems to be helping with motivation in the class, but at the moment that’s just my anecdotal take on the situation.

ENTRY POINT: Assessment

In my class students undertake THREE assessment tasks:

  1. Personal essay on teaching philosophy and resource analysis (individual, 30%)
  2. Lessons plans for a junior English class (in pairs, 40%)
  3. Portfolio of completed learning ‘challenge tasks’ (individual, 30%)

What I like about what I have achieved with this set of assessments is that there is a balance of individual and group work, that there is a variety of tasks, and that no task is worth more that 40%.

At this point I’ll put myself out there to say I am disappointed to see how many uni coordinators choose to use just TWO assessment piece in their own classes. This is not good practice imo! Having less assignments does mean a smaller marking load for the lecturer, and less due dates for the student, but at what cost?

I really do believe that students in uni should not have assessments that are worth 50% or over, as this is too high-stakes to promote good learning. To do this, you must have more than two assessments for a unit in a semester.

FINAL WORDS: The CLB018 ‘assemblage’

In the theory of Deleuze and Guattari, the context of my CLB018 class provides an assemblage of bodies and things that can produce any number of effects. I hoe to keep reporting throughout the semester on the effects (and affects) of our assemblage!

In the meantime, any comments on these POINTS OF ENTRY are most welcome.

, , , , , ,

3 Comments

Evaluating the HSC: An interview

Last year I was interviewed by Melissa Wilson, a Journalism student from the University of Newcastle, about my views on the Higher School Certificate (HSC).

Melissa was in contact again recently, and this prompted me to ask if I could reproduce the interview here on my blog. She kindly obliged, and so here it is!

I was careful to read back over my answers, to make sure I still felt the same way about these issues. I do. I wish I could say that things were vastly different up here in Queensland. They’re not. When a Queenslander tells you there is no external exam for Year 12 in their state, they’re misleading you at best. Here is a bit of information from the QSA website about the Queensland Core Skills (QCS) test:

Preparing for the test

The Common Curriculum Elements are generic skills that students work with across their subjects; therefore the real preparation for the test goes on all the time and in every subject. The QSA also makes available a variety of test preparation resources, including Retrospectives and past testpapers (see QCS Test publications and Retrospectives and MC response sheets). Most schools provide some focused preparation for the test.

Hmm, sounds familiar.

But, I digress…

Here are my responses to the interview by Melissa Wilson:

Interview answers for Melissa Wilson (University of Newcastle, 2011)
Interviewee: Kelli McGraw (Lecturer, QUT)

  • How do you feel about how the HSC is structured in 2011?

When I think about the HSC structure in 2011, the main things that leap to mind are the fact that studying English is mandatory, that half of the student assessment is based on a timetabled external examinations, and that a no more than of 30% of your school assessment is supposed to be ‘exam-type’. I think it’s really important for English to remain compulsory right up to the end of school, but I’d like to see more room for students to choose electives within the course, not just different levels i.e. Standard or Advanced English. At the moment I think the HSC is still structured in a way that is too rigid for students to feel like they have a lot of choice over their learning.

  • Many people say that the HSC is focused on teaching students a whole lot of information that isn’t exactly relevant to them later in life – but instead they just regurgitate it in an exam and then discard it – how do you feel about that statement?

Personally, I can think of countless things that I learned in my HSC year (1998). In those days the emphasis on exams was just as great, but I am often surprised by the things I remember from senior high school and have found a lot of what I learned to be very relevant in life. Having to finish ‘major works’ for Visual Art and Drama also taught me valuable lessons about project management and self-directed learning, which I didn’t get from participating in written exams, so in that sense I guess I was lucky to be an ‘art-sy’ student.
I think the real problem with exams is not that students have to cram ‘irrelevant’ information – I think that all learning can be made relevant, depending on what you choose to do in life. The problem I find is that the examination system has too much of an effect on what happens inside the classroom. The constant pressure to cover content is a strain on students and teachers, and even though school-based assessment is supposed to involve deep learning and reflection, many schools I know of set far more than 30% of their assessments in an exam style in order to condition students in preparation for the external exam. So I think there is a ‘hidden curriculum’ in the HSC, which can dilute learning experiences based on the official subject syllabuses.

  • And from this, what would you personally change about the HSC? 

I think that the only assessment that students should have to do under exam conditions is the Trial. If more student work was assessed through project work, or using collaborative group tasks, or using portfolios, I think that students would feel more connected to the learning, and be motivated to achieve. Even though the HSC now uses criteria-based assessment, students are acutely aware that the HSC places them in competition with one another as in-class assessment ranks still play a role in determining a student’s final subject results, and the year culminates for most students in receiving a ranked national placement through the UAI. With only about 30% of students moving from school to university after Year 12, it seems like we compromise a lot of educational values for the sake of a privileged minority.

  • How do you feel about the pressure and emotional stress that students endure throughout their HSC?

When I think about the stories that students have told me over the years – about how they feel inadequate, or like a failure in the face of HSC assessment tasks – it makes me really upset. I have seen a lot of students in Year 12 lose a lot of weight, with girls in particular showing signs of early and advanced eating disorders. Senior school is also a time when increased numbers of students pick up casual and part-time employment, in many cases out of a necessity to contribute to household finances. I think the HSC creates an environment where students are given too many adult responsibilities without being given the corresponding rights.

While schools play a vital role in developing students’ resilience and capacity for work, the emotional stress endured during the HSC year is too much, in my opinion. I read a study awhile back where Year 11 and 12 students reported symptoms of anxiety, depression and stress that fell outside the ‘normal’ range. We know that when this happens, students stop focussing on ‘mastering’ the material, instead focussing on performance; they stop believing in themselves, stop seeing the learning as a worthwhile goal, and switch to performance-oriented goals. Some of my own school friends took years to recover from the emotional damage of the HSC year, especially those whose final results didn’t meet expectations.

Good riddance to my English notes.

Good riddance to my English notes. Flickr image by Jean-Rene Vauzelle CC-BY-2.0

, , , , ,

3 Comments

Coming up for air

Hi folks – it’s been a bit quiet here on the blog, I know.
All I can say is … omg MARKING!

I have always had an interest in assessment, but this semester has made really clear to me how dire the situation is with our current practices.

I don’t want to ‘buy out’ my marking (i.e.pay someone else to do it for me) but I feel like I am wasting so much of my time at the grindstone, like a machine, writing the same lines over and over in delightful pink pen in the margins of my students’ work.

“Check the APA style guide for rules about how to format this”
“Formal essays require shorter paragraphs than this”
“Avoid rhetorical questions – make strong statements instead”
“Use your introduction to tell me what your main points will actually be, not to explain the structure of your work”
“Don’t use a quote as a sentence on it’s own – introduce it i.e. ‘Sawyer (year) explains that…'”
“You have not included reference to any unit readings in this rationale”

I worry about RSI. I worry about carpal tunnel! Marking more tasks electronically next semester will hopefully fix the hand ache, but what about the mind ache??

I’m not alone – every teacher reading this knows what I mean.

What are we going to do about it?

Portland Oregon & VA exam 46 – CC-BY-2.0 Flickr image by Parker Knight

,

1 Comment