Communications of the ACM
Results From the First-Year Course MOOCs: Not There Yet
Georgia Institute of Technology professor Mark Guzdial
MOOCs in the Coursera, Udacity, and edX form are tightly tied to CS. The leaders of the xMOOC movement came out of computer science, and most of the first generation of xMOOCs focused on teaching computer science. Many of the MOOC evaluations so far have been expert reviews. Our Learning Sciences and Technologies seminar at Georgia Tech's College of Computing just read Moti Ben-Ari's travelogue on his experiences in Coursera's and Udacity's introductory CS MOOC. The empirical results of the first rounds of MOOCs on intro courses are now in, so it's worth considering how they're doing.
Karen Head has finished her series of posts in The Chronicle on the freshman-composition MOOC that she taught with Gates Foundation funding. The stats were disappointing — only 238 of the approximately 15K students who did the first homework finished the course. That’s even less than the ~10% we saw completing other MOOCs.
Karen Head writes:
No, the course was not a success. Of course, the data are problematic: Many people have observed that MOOCs often have terrible retention rates, but is retention an accurate measure of success? We had 21,934 students enrolled, 14,771 of whom were active in the course. Our 26 lecture videos were viewed 95,631 times. Students submitted work for evaluation 2,942 times and completed 19,571 peer assessments (the means by which their writing was evaluated). However, only 238 students received a completion certificate—meaning that they completed all assignments and received satisfactory scores.
Our team is now investigating why so few students completed the course, but we have some hypotheses. For one thing, students who did not complete all three major assignments could not pass the course. Many struggled with technology, especially in the final assignment, in which they were asked to create a video presentation based on a personal philosophy or belief. Some students, for privacy and cultural reasons, chose not to complete that assignment, even when we changed the guidelines to require only an audio presentation with visual elements. There were other students who joined the course after the second week; we cautioned them that they would not be able to pass it because there was no mechanism for doing peer review after an assignment’s due date had passed.
via Lessons Learned From a Freshman-Composition MOOC – Wired Campus – The Chronicle of Higher Education.
Georgia Tech also received funding from the Gates Foundation to develop a MOOC approach for a first year college physics course. I met with Mike Schatz, the lead teacher on that effort. It's a remarkable course, including a "laboratory" where students take videos of moving objects, then construct computational simulations in Python to match the real-world observations. The completion results were pretty similar to Karen's: 20K students signed up, 3K students completed the first assignment, and only 170 finished.
In terms of empirical studies, Mike had an advantage that Karen didn’t — there are standardized tests for measuring the physics knowledge he was testing, and he used those tests before and after the course. Mike said the completers fell into three categories: those who came in with a lot of physics knowledge and who ended with relatively little gain, those who came in with very little knowledge and made almost no progress, and a group of students who really did learn a lot. They don’t yet know the relative percentages of the three categories. However, it's clear that being a completer doesn't mean that anything was learned.
I also met with Jason Freeman who just finished his Survey of Music Technology MOOC for Coursera. His results were a bit better: 24K signed up, 13K ever visited, and 900 completed. It seems that the more advanced the course, the better the completion rate.
The report from the San Jose State University MOOC experiment with a remedial mathematics course was published with the finding:
The researchers also say, perhaps unsurprisingly, that what mattered most was how hard students worked. "Measures of student effort trump all other variables tested for their relationships to student success," they write, "including demographic descriptions of the students, course subject matter, and student use of support services."
It’s not surprising, but it is relevant. Students need to make an effort to learn. New college students, especially first generation college students (i.e., whose parents have never gone to college), may not know how much effort is needed. Who will be most effective at communicating the message about effort and motivating that effort — a video of a professor, or an in-person professor who might even learn your name?
MOOC companies have set a goal of democratizing education. (See Daphne Koller's TED video.) They aim to make education available to people who would not otherwise get access to education. MOOCs are not yet succeeding at that goal. The empirical findings (e.g., the detailed study from Duke, Armando Fox's results at Berkeley, and Tucker Balch's demographics) suggest that MOOCs draw mostly men (especially in the CS MOOCs) who already hold degrees and are overwhelmingly from the US and the developed world. Right now, xMOOCs seem most successful for professional and continuing education.
Gary May, our Dean of Engineering, recently wrote in an op-ed essay published in Inside Higher Ed, "The prospect of MOOCs replacing the physical college campus for undergraduates is dubious at best. Other target audiences are likely better-suited for MOOCs." That summarizes the current state pretty well.
No entries found