The challenge of meaningfully assessing discussion participation

Since my last MOOC course update (E-learning & Digital Cultures), I’ve completed four more courses, including: Introduction to Communication Science (University of Amsterdam); Social and Economic Networks (Stanford, Coursera); Infographics and Visualization (Knight Center for Journalism in the Americas); and ‘Art and Inquiry: Museum Teaching Strategies For Your Classroom’ (New York Museum of Modern Art, Coursera). I chose to study some because of relevance to my work, and others just out of interest and curiosity. All were enjoyable and interesting courses overall. I’m not going to go into detail about the first two courses as they both followed the now fairly standard model of video lectures and multiple choice questions, but going to focus on the second two as these are the first MOOCs I’ve taken which require forum participation in the way the courses are assessed.

The Infographics and Visualization MOOC started in mid January and lasted for six weeks. It was my first non-Coursera MOOC (the Communication Science one was also independent of a MOOC platform, but this started later), and used Moodle as its virtual learning environment. The following criteria needed to be met to gain a certificate (in addition to paying $30 – it was also the first and only certificate I have shelled out for):

  • Week 1: Participated in the discussion by posting at least twice in the following forum: Critique a graphic forums. Submitted Quiz 1 with at least an 80% correct.
  • Week 2: Participated in the discussion by posting at least twice: Critique a graphic forums. Submitted Quiz 2 with at least an 80% correct.
  • (Week 3: Participation in forums was not mandatory this week.)
  • Week 4: Participated in the discussion by giving feedback to at least three of your classmates’ projects. Submit your project link about tenure (assigned in week 3) in the designated forums.
  • Week 5: Participated in the discussion by giving feedback to at least three of your classmates’ final projects. Submit your project link about personal taxes (assigned in week 4) in the designated forums.
  • Week 6: Participated in the discussion by giving feedback to at least three of your classmates’ final projects.

After the first couple of weeks, students were required to post links to projects they had undertaken according to various briefs, however these weren’t formally peer graded – as long as you posted something, and posted comments about the work of at least three others, this met the assessment criteria. It became quite difficult quite quickly to keep up with the discussion forums, and my inbox was deluged with forum post notifications. I soon gave up any hope of being able to keep up, and adopted a strategy of being a bit of a forum ninja, sneaking in and posting the required number of comments and giving up on the possibility of returning to the discussions. I suspect that other students were also using the same approach, as I didn’t think that the comments posted about my work were as useful or considered as peer reviews I’ve received via Coursera generally have been.

The Art & Inquiry course used a similar mix of assessments; to gain a certificate, students were required to achieved over 70% across automated quizes (two quizzes, combined weighting of 25%), forum posts (four topics, combined weighting 25%), and a peer assessed project (50%). It wasn’t made clear exactly how the forum participation was assessed, although the general impression was that as long as you posted the required number of times, you would satisfy that part of the assessment criteria. From the outset, the course set out with an ethos of rewarding active engagement; “your success in this course is based on your level of engagement and participation, and not on memorizing facts or grades.” (course site overview text).

I think that this is an admirable goal, but not sure that simply counting whether students have made a required number of posts is really assessing this. Assessing collaboration and interaction with students is still a very open challenge for MOOCs. Is it simply not achievable at this scale, and not a realistic part of ‘MOOC pedagogy’? Or does it require thinking and development of solutions beyond the e-learning tools (such as forums) which have served smaller distance learning courses well, but struggle to scale up? The jury is out for me at the moment – I’d love to hear your thoughts and ideas about alternative ways of assessing this in MOOCs :-)

Something else which the Infographics and MoMA courses have in common is the fact that the classes are not archived – both course sites and materials becoming inaccessible shortly after the courses ‘ended’ (the MoMA course site is still accessible at the moment, but says it will close after a month). This seems to be happening more frequently with Coursera – for example, I was enrolled on a course on Systems Biology, which I’d not been ‘actively’ engaged with, but wanted to dip into the materials at my own pace later in the summer in preparation for a course on Network Analysis in Systems Biology. However, the course site promptly closed at the end, and this hasn’t been possible. This is a bit at odds with the ‘Open’ in MOOC, and not very helpful for students auditing the materials.

Edit 2013-09-21: Recently I’ve read two other blog posts asking important questions about MOOC discussion fora. See also Alastair Creelmans’ post ‘The silent majority – why are MOOC forums counterproductive?’, and ‘MOOC Discussion Forums: barrier to engagement?’ from Phil Hill.

About these ads

1 Comment

Filed under Uncategorized

One response to “The challenge of meaningfully assessing discussion participation

  1. Pingback: The challenge of meaningfully assessing discuss...

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s