Synthesising MOOC completion rates

Via a combination of thinking about ‘what makes a successful MOOC?’, and looking for a topic for my final project on the Infographics MOOC, I decided to try to pull together the various statistics floating around online about MOOC completion rates. I’m trying to see if any differences emerge on the basis of platform or the assessment methods used.

My draft graph synthesising everything I’ve found so far can be found here: http://www.katyjordan.com/MOOCproject.html Clicking on any of the data points will pull up a bubble with more information about that course, and a link back to the data source.

(note: the interactive version of the chart uses javascript. It has problems with some versions of Internet Explorer; I’ve found it works more consistently with Firefox. If you are having problems, click the pictures below to view screen grabs, although these may not be as up-to-date as the interactive chart).

AssessmentType_20140225

CourseLength_20140225

This is off to quite an interesting start, but I need help sourcing more data and categorising courses according to their assessments.

Courses which I have completion rates for, but need more information about how they were assessed, include:

If you studied on any of these courses, please do post a comment here outlining how it was assessed (just MCQs? Peer graded projects? Or something else?). If you know about any other sources of data about MOOC completion rates (how many registered and how many completed) in addition to the ones already in my chart, please do post a comment too (& a link to the data source), and I’ll add them to the chart. Thanks & looking forward to seeing the picture which emerges!

Advertisements

138 Comments

Filed under Uncategorized

Fundamentals of Online Education: Dude, where’s my MOOC?

After a party on Saturday night, on Sunday I awoke to find not my car missing, but that a MOOC I’d been taking had apparently disappeared. I’d completed the first week of the Georgia Tech course on ‘Fundamentals of Online Education: Planning and Application‘ (FOE), but at the weekend the course was ‘suspended’ after less than a week of its six-week run.

This was a bit of a surprise. Critical accounts of the course and its demise have been circulating (see for example this post outlining in detail the problems with getting into groups, and a critique of the video material here).

Now, I realise I may be a bit of a dissenting voice, but – I didn’t think it was that bad. Not great, but not bad enough to stop.

I agree with the criticism that the groups were chaos; lesson number 1 was that a Google Spreadsheet can’t cut it when you have 40,000 students. However, none of the other Coursera courses I’ve taken before have attempted to get all students interacting, beyond anonymous peer grading exercises. Although I’d not managed to take part in a discussion before the course was cancelled, I was interested to see where it was going and thought kudos for trying. It’s worth remembering that not only had this course not run before, but MOOCs as a whole are still relatively new and there aren’t established ways of doing many things at this scale. This kind of group discussion does appear to be happening successfully in another MOOC I am currently enrolled on, the Infographics and visualisation MOOC at the Knight Center. However, this is at a cost; the course was limited to 5,000 participants at sign-up, and my impression is that only a fraction of these are still actively participating in the course.

The quality of the video lectures was not great, but apart from not being HD or having a talking head as the lecturer goes through the slides, narrated Powerpoints have been common in the other Coursera MOOCs I’ve used. My impression was that the week 1 lectures were particularly dry in order to just get through a wide range of basic concepts, and that it would probably get better as the course went on; I was prepared to give it the benefit of the doubt, but sadly it doesn’t look like I will find out.

Of course, there is a huge irony in this case as it was a course about developing online courses. I think that there may have been a couple of other factors at play in its demise. It launched on the same day as the University of Edinburgh course E-learning and Digital Cultures (EDCMOOC), which has an overlap in focus, and this invited comparisons from students enrolled on both courses. The EDCMOOC also seeks to get learners actively participating in group discussions, but does so via social media channels. This has been more successful, but has also attracted some criticism from students who would rather not use these channels or find the volume of discussion across several platforms hard to manage – see this post, for example. Maybe finding a way for the platform to better support discussions is something that Coursera needs to work on more generally.

Also, it’s fairly unlikely that in a class of 40,000 everybody will be happy, and there will always be dissenting voices. The biggest disappointment about the course for me was that in the end, after having invested my time in the first week, that the course shut down. I would have much rather seen it regroup and maybe just stick to the videos and multiple choice questions – it was my impression that it was only a beginner-level course anyway. In the discussion forums which remain on the course site, there are many students who would like to see it return and are still keen to learn in spite of the problems. At the time of writing, the course still appears in my list at Coursera, so it remains to be seen whether this is the end.

It also remains to be seen what the extent of the repercussions of a ‘failed course’ will be for Coursera and perceptions of MOOCs more generally. As I’ve got one day left of my Topsy trial, I’ll finish off with a chart of sentiment analysis on Twitter for the keyword ‘coursera’ in the past month, which took a downward dive in the last three days (data collected 2012-02-04, click picture to enlarge):

Screen Shot 2013-02-04 at 19.40.12

So in conclusion, I’m not saying that folks should put up with shoddy materials ‘cos it’s free, but rather this example highlights that mainstream MOOCs are still in a kind of ‘beta’ phase – and everyone is learning from it, one way or another, even those who are seasoned e-learning professionals. The next iteration of the course would probably have benefited from the course running to completion, so that more systematic feedback could have been elicited about its whole structure. I really hope that we do see a second run and this experience hasn’t put the instructors (or the other students) off completely.

1 Comment

Filed under Uncategorized

MOOC buzz in the Twittersphere

In the last couple of weeks, in conversations about MOOCs, I’ve found myself making the point that a year ago we didn’t really use the word MOOC, and while the MOOC platforms were getting organised at that point, no Coursera/Udacity courses had even run. So, I thought it might be interesting to take a look at some of the Twitter data to see roughly the shape of the body of tweets mentioning MOOCs in the past 365 days.

I’ve not played with Twitter data before, but through some quick Googling, I found a few potential tools to do all the hard work for me – Topsy being the one which I found best to work with, and generated the following brief analysis using it.

Number of mentions, per day, in the last 365 days (data collected 22nd January 2013 – click on picture to enlarge):
MOOC_by_day

There has been a definite amplification in the conversation around MOOCs. If we plot the data as monthly rather than daily mentions, a steady linear increase is seen during 2012:

Number of mentions, per month, in t he last 365 days (data collected 22nd January 2013):
MOOC_by_month

The sharp drop off at the end of the graph is an artefact of data collection (i.e. January 2013 isn’t finished yet). If we ignore the last data point (January 2013), there was a slight dip in December, but this may have been a result of the holiday season. So while 2012 may well have been the ‘year of the MOOC’, I don’t think the bubble has burst yet!

Topsy also has sentiment analysis functionality built in, so I couldn’t resist trying it out, although this part comes with a caveat in that I have no idea exactly how the website defines sentiment, so the tool is a bit of a ‘black box’ to me.

Sentiment analysis (data collected 22nd January 2013 – click on picture to enlarge):
MOOC_sentiment

General sentiment seems to be positive; although negative spikes are rare, they are more pronounced. To create this chart, Topsy processed 83,158 tweets, 10,407 it deemed to be ‘positive’ and 3,538 ‘negative’ in sentiment. It could be interesting to do a deeper analysis like this to explore trends in the discussion topics surrounding MOOCs.

I also couldn’t resist having a look to see how the Twitter buzz varies around the three main MOOC platforms.

Monthly tweets mentioning Coursera, EdX and Udacity (data collected 22nd January 2013):
MOOC_platforms

Looking at the tweets from January 2012, the Udacity and Coursera tweets are still at a very early stage; the tweets are focused upon news stories about the platforms being founded. Note that EdX was not even on the horizon at this point – rather, January 2012 tweets appear to relate instead to a DJ called EdX.

Anyway – while this has been quite a quick-and-dirty kind of analysis, it sort of captures a bit of the whirlwind and buzz that has surrounded MOOCs in the past year, and the gain in momentum. I wonder where it’ll be this time next year!

5 Comments

Filed under Uncategorized

Networked Life, Social Network Analysis, & a new appreciation for feedback

Two key Coursera courses which did not fall by the wayside for me this Autumn were ‘Networked Life‘ (Michael Kearns, University of Pennsylvania) and ‘Social Network Analysis‘ (Lada Adamic, University of Michigan).

Both courses held a particular significance for me. Whereas the MOOCs I had taken up until this point had been ‘just for fun’, these courses were directly related to my PhD, which I started this Autumn. The courses were incredibly helpful as introductory social network analysis courses, giving me a head start methodologically, and saving me probably a couple of hundred pounds in face-to-face training courses. The courses were also the first MOOCs which I’ve found to be challenging; graph theory gave my brain the best mathematical workout since I did A-level maths (and I really do mean that as a compliment!).

Although the courses were ostensibly very similar, in terms of the subject matter and scope, they each provided very different MOOC experiences, and I ended up with very different marks – 98.8% for Networked Life, compared to 80.3% for Social Network Analysis (just scraping a certificate)! Here I’m going to try to explore the reasons behind this.

‘Networked Life’ (NL) had a simpler course structure than ‘Social Network Analysis’ (SNA). NL was slightly short, at 6 weeks, compared to SNA’s 8 weeks. In NL, the material was taught via video lectures (including simulations), and assessed by multiple-choice questions. Assessments were quite strict in that you could only attempt quizzes a maximum of twice. In contrast to the other Coursera courses I’ve taken in the past, all of the NL lectures and quizzes were available right from the start of the course, i.e. there was no weekly release of material (which was nice – I enjoyed being able to get further ahead when I had spare time, but still having weekly deadlines for the quizzes kept me from falling behind when time was tighter). The quizzes were tough, but the feedback on the first attempt was helpful and constructive, which really helped in learning from mistakes.

The main mode of material delivery in SNA was also video lectures including simulations. There were however two different ‘modes’ to the course; a basic certificate could be earned by gaining a mark of over 80% via the weekly multiple choice quizzes and final exam, while a certificate with distinction could be earned by completing this plus additional programming assignments (usually multiple choice questions relating to interrogating a network data set in a social network analysis package, plus a peer-assessed project). The multiple choice questions in SNA were the hardest I have encountered in a MOOC yet, mainly because it did not give any feedback after each attempt – not even an indication of which questions were right or wrong! This was very frustrating, and has given me a whole new level of appreciation of feedback! Because of this, I could generally do quite well on the assignments, but remember that the pass mark was set at 80% right from the start of the course; I found it really hard to get really good marks on the assignments due to the lack of feedback, and kept bumping along right on the pass mark.

I think it helped that I had chosen to do the optional programming assignments, as I had been doing a bit better on these (getting more like 90-100%). It was a bit of a surprise when I realised that I need to complete a project though – I’d thought that the optional programming assignments were a sort of ‘extra credit’ affair, and didn’t realise ’til a week before the project was due that doing the programming assignments kind of put me on the programming ‘track’ and I would fail if I did not submit a project. If you are interested, here is the project that I submitted – if you refer to it anywhere, please use the suggested citation at the top of the PDF file. I got a bit carried away and marked 25 peer assessments! It was compelling to see what other students had chosen for their project. I think this was the advantage of the SNA course over NL; NL was better for covering the basic concepts, but SNA offered the benefit of the opportunity to think about and apply what you had learned, which was invaluable for me.

It was a bit of a mystery to me as to whether I would actually get a certificate or not (my calculations put me right on the grade boundary), and was genuinely pretty surprised when I did. The following statistics were released at the end of the course:

“Some participation stats: 61,285 students registered, 25,151 watched at least one video, 15,391 tried at least one in-video quiz, 6,919 submitted at least one assignment, 2,417 took the final exam. 1303 earned the regular certificate. Of the 145 students submitting a final project, 107 earned the programming (i.e. ‘with distinction’) version of the certificate.”

The most surprising statistic here for me though is that out of the 2,417 students who took the final exam, I’m assuming having completed all the weekly assignments and made it through the course to the end, only 1,410 actually got a certificate. I think that there could not have been a bit more leniency in terms of the pass mark – 80% seemed quite arbitrary and was fixed at the start of the course, which had not run on Coursera before. Also, it turns out that with only 145 students submitting a project, I had actually personally marked 17% of the projects submitted! Which made me think … is this really massive?

3 Comments

Filed under Uncategorized

The MOOCs that got away: A brief roundup & reflection on my drop outs this Autumn

In the past couple of months, I’ve been spoiled for choice at Coursera, as many more universities have joined the platform. After finishing the Internet History, Technology and Security course, there were several which caught my eye . I initially thought ‘hey, why not, let’s push this and see how many I can do simultaneously’, but there was a another factor involved – needing to move house and starting my PhD! So compromises had to be made. Here is a whirlwind summary:

Networks: Friends, Money, and Bytes: I had signed up to this course quite a while in advance as it seemed very interesting, and if I hadn’t had two ‘essential’ courses at the same time (Networked Life and Social Network Analysis – blog post on these to follow soon), I imagine I would have stuck with it. Additional factors which led me to drop the course were: (i) it seemed very long, at 14 weeks; (ii) despite seeming like quite a lot of work, it did not offer a certificate (I’m slightly ashamed to say that this did contribute to putting me off – I know getting a certificate is not really the point!); and (iii), the course content had recently been published as a book, so I could potentially read this at some point in the future instead.

Web Intelligence and Big Data: I completed the first week of this course, before dropping out in the second. Again, had it been running at a time when I did not have higher priority MOOCs underway, I would probably have persevered. The course included programming assignments, which I don’t think I would have been able to do without further study (other students on the forum helpfully suggested that the Udacity CS101 course is a good introduction to Python, so I will aim to take this course in preparation for the next offering of Web Intelligence and Big Data).

Securing Digital Democracy: A bit of a detour from my main interests (I set out with MOOCs to learn about Computer Science), I took this course because I thought it may fit with my broader interest in Web Science. It was relatively short (5 months) – small, but perfectly formed. The professor, J. Alex Halderman, had an excellent presentation style and the lectures were nicely finished. The course was assessed by multiple choice question sets, and a final peer reviewed essay (although due to time pressures, the course topic not being a high priority for me at the moment, and safe in the knowledge that I had scored well enough on the quizzes to gain a certificate, I did not submit an essay).

Learn to Program: The Fundamentals: Completed the first weeks’ material, but then it slipped and missed deadlines. Think it would have been manageable otherwise – and useful as my lack of programming knowledge is proving a stumbling block with some other courses (such as Big Data). It was a victim of purely bad timing for me. Hoping it will prove popular and run again soon!

Reflecting on these hits and misses, I think it raises an interesting question about how you join together multiple MOOCs into a broader learning pathway. I’ve gone on a bit of a detour from my original aim of studying computer science, which is not a bad thing – the detours have been enjoyable and I don’t really have any kind of deadline I’m aiming for – but as a beginner, it is a bit tricky to tell where I ought to go next. There is also an issue about starting to need prerequisites for courses, such as programming, which creates a progression between courses, but this kind of progression is not explicit. On a related note, Coursera have recently introduced profile pages for students, which show which courses students are taking; it would be interesting to map the network of co-studied MOOCs to see if students’ choices cluster into traditional disciplines or emergent interdisciplinary areas. As for my next MOOC, I am planning on going back to Computer Science and taking either CS101 at Udacity, or EdXs’ CS50 – watch this space!

1 Comment

Filed under Uncategorized

MOOC thoughts from Glasgow Social Media Week

Found this whilst tidying up my desktop and posting here in case it is of interest: last summer I was invited to create a one minute video about my current thoughts on MOOCs for Glasgow Social Media week, which took place in September 2012. Click here to view the video.

thumbnailVideo

Leave a comment

Filed under Uncategorized

IHT&S addendum: Some answers to ‘who studies a MOOC?’

In my first post, I remarked upon some of the demographic trends I had observed when taking the CS101 course at Coursera, and expressed my hope that the MOOC providers would share some of their data to help the wider world understand the impact of MOOCs.

Dr. Chuck, the instructor on the Internet History, Technology & Security (IHT&S), took the initiative to circulate a demographic survey – and to share the results with the course participants. He encouraged us to reflect on the data and blog about it, so here we are: some answers to my first question, of ‘who studies a MOOC?’. Graphs have been drawn and interpreted by me, data gratefully courtesy of Dr. Chuck. The data comes with the following ‘health warning’ from Dr. Chuck: “Of course the caveat is that it is not scientific, it is partial, incomplete, your results may vary, void where prohibited, etc etc etc. It is anecdotal at best but certainly interesting.” We can’t tell how representative the sample is of the course as a whole, or assume that IHT&S can be generalised to other courses, but it does provide an interesting insight and raises some interesting questions.

Student demographics

Note: ‘Associate degree’ denotes 2 years undergraduate-level study; ‘Bachelors degree’ was described as 4 years of udnergraduate level study, based on the American model

  • Male students outnumber female by 2:1. Why? How much does gender of students depend on the course topic?
  • The course was fairly popular across the whole range of ages. Not sure why an under 18 category was not included. Modal category is ’25 to 34 years old’; interesting that this is the category following the one which university-level study would typically fall into. Does this indicate the importance of MOOCs as a next-stage in lifelong learning for the recent graduate? Is this in response to career pressures – MOOCs as a way to get ahead in the workplace?
  • Most respondents have a degree already – either undergraduate or masters, relatively few doctorates.

Students’ previous experiences of online learning

  • While the course is the first MOOC that most students have taken, more than half of the respondents have taken online courses before. Are MOOCs particularly attractive to students who have previously studied online? Do they have different expectations of the online MOOC environment to students who have not studied online before?

Reasons for taking the course

Note: Respondents could select multiple answers about their motivation for taking the course.

  • Givent that respondents could select multiple responses to their motivations for taking the course, it is more meaningful in a sense to focus on the categories which people did not select. In this case, it is notable that the lowest response categories – ‘Supplement other college/university classes courses’ and ‘Decide if I want to take college/university classes on the topic’ – are the ones which relate study to formal higher education structures.
  • Non-students outnumber those in formal education by approx 5:2.

Reuse and OER

  • While this pair of questions could suggest that most or all teachers taking the course would consider reusing the course materials in their own teaching, these responses should be treated with some caution, as more positive responses were gained to the reuse question (510) than respondents who indicated that they are actually teachers (451).

So: I’m intrigued by the gender differences, and the indication that MOOCs may be playing an important role in initiating lifelong learning in the years after formally leaving the academy. Of course, this is quite speculative as the data here is quite limited and only form one course. I’d be very interested to hear others’ take on the data – please do feel free to leave a comment here.

4 Comments

Filed under Uncategorized