Monthly Archives: January 2017

Engagement vs Learning – Assessing the research in online video production

At last years 8th International Conference on Multimodality (8ICOM) in Cape Town, video producer at CILT, Don Leitch presented on “What does an effective e-learning video look like”. As part of the MOOC team he shares his thoughts with us. 

As the world of MOOCs (Massive Open Online Courses) and e-learning in general continues to expand, educators and researchers have unprecedented access to data and are finding new ways to interpret it. When it comes to video in e-learning however, there is surprisingly little research available, and some fundamental questions of how to interpret what little data there is, remain unanswered.

Much of the difficulty, both in theory and in practice, comes from the confluence of filmmaking with education. The two fields of knowledge can be boiled down, in terms of their comparison with one another, to two fundamentally different concerns. Filmmaking and videography in general is all about audience engagement, while pedagogy is about learning.

It is often pointed out that engagement is a necessary but not sufficient aspect of learning, meaning that one cannot learn anything without being engaged (i.e. caring) but one can easily be engaged and entertained from watching a video without learning a thing.

This has practical implications on how educational videos are produced. I work as a video producer on MOOCs at the University of Cape Town. Like almost all video producers my practical and theoretical training are centred around film as an art form and as a medium of entertainment. Many of the techniques and theory focused on audience emotion, engagement, and immersion. This presents a challenge when attempting to balance my idea of the best way to make a video with that of the learning designer or instructor who is relying on an entirely different practical and theoretical background.

This divergence of engagement and learning can also be found in the research done on videos in e-learning. I have noted five broad categories of studies assessing e-learning videos; surveys, taxonomies, video interaction data, AB testing, and controlled experiments.

Surveys are useful for knowing audience preference and opinion, but often don’t correlate well with actual learning. There are some interesting data points however, such as the fact that people are apparently more forgiving of longer educational videos than entertainment videos.

Taxonomies are broad studies that attempt to categorise videos and measure how many of what type are used where. I believe such research will be most useful when used in combination with other data to create a more complete picture of the e-learning arena.

Studies making use of video interaction data interpret information generated by viewers pausing and moving to different parts of videos. These studies have the advantage of very large sample sizes (making use of MOOCs with thousands of participants) and have interesting implications about how e-learning videos should be edited. Due to the nature of MOOCs, however, it is very difficult to marry this information to measurements of student learning and the studies are limited to measuring engagement alone.

AB testing is widely used by MOOC platforms to assess the best ways to retain students and have great potential for setting up controlled experiments of enormous sample sizes. For example, one could test two different styles of video production to see which is more effective. Unfortunately, MOOC platforms tend not to publicly publish their results. Furthermore, the same problem of linking engagement data to learning data exists.

Researches are able to get around the problem of measuring both learning and engagement by setting up smaller scale controlled experiments. These do have smaller sample sizes but have yielded some interesting results about the link between engagement and learning. 

Ultimately as producers of e-learning videos we largely rely on our cross-disciplinary interaction, some basic findings from research, as well as intuition and opinion. There is certainly much more that can be learned from the large amounts of data being generated, but we will need to think a lot more about how to make use of the findings  for improving production.

Written by Don Leitch

(For the full slideshow please click here)

References:

  • Chen, Z., Chudzicki, C., Palumbo, D., Alexandron, G., Choi, Y.J., Zhou, Q. and Pritchard, D.E., 2016. Researching for better instructional methods using AB experiments in MOOCs: results and challenges. Research and Practice in Technology Enhanced Learning, 11(1), pp.1-20.
  • Guo, P.J., Kim, J. and Rubin, R., 2014, March. How video production affects student engagement: An empirical study of mooc videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 41-50). ACM.
  • Hansch, A., Hillers, L., McConachie, K., Newman, C., Schildhauer, T. and Schmidt, P., 2015. Video and online learning: Critical reflections and findings from the field
  • Kim, J., Guo, P.J., Seaton, D.T., Mitros, P., Gajos, K.Z. and Miller, R.C., 2014, March. Understanding in-video dropouts and interaction peaks in online lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 31-40). ACM.
  • Kizilcec, R.F., Papadopoulos, K. and Sritanyaratana, L., 2014, April. Showing face in video instruction: effects on information retention, visual attention, and affect. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (pp. 2095-2102). ACM.
  • Pierce, M., 2015. Learning and Development: What Makes Videos Effective? Iconlogic. http://iconlogic.blogs.com/weblog/2015/02/learning-and-development-what-makes-videos-effective.html
  • Reutemann, J., 2016. Differences and Commonalities–A comparative report of video styles and course descriptions on edX, Coursera, Futurelearn and Iversity. Proceedings of the European Stakeholder Summit on experiences and best practices in and around MOOCs (EMOOCS 2016), p.383.
  • Simonite, T., 2013. As data floods in, massive open online courses evolve. Technology Review.
  • Wulff, B., Rupp, L., Fecke, A. and Hamborg, K.C., 2013, December. The lecture sight system in production scenarios and its impact on learning from video recorded lectures. In Multimedia (ISM), 2013 IEEE International Symposium on (pp. 474-479). IEEE.

UCT MOOCs ranked highly on Class Central

In 2016 the University of Cape Town (UCT) was ranked as the second best institution creating MOOCs according to a report published by MOOC aggregator website Class Central. The UCT MOOC team would like to thank all our participants for their constructive reviews.

10-Best-Online-Courses-of-2016

In terms of the methodology for the ranking it appears that by using the course ratings of a course to represent its university, Class Central took the Bayesian average of each to compile its list. Universities with fewer than five courses or fewer than 50 ratings were excluded.

Not only was UCT ranked as the second best institution, but our course Becoming a changemaker: Introduction to Social Innovation was ranked in Class Central’s top 10 best online courses of 2016. Again, over 8000 reviews were analysed and the Bayesian average of their ratings were used to compile the top-rated free online courses for 2016.

Becoming a changemaker is designed to encourage people to begin acting as social innovators and changemakers, debunking common assumptions around what resources are needed to get started.

“I have just finished the course and cannot express enough how energizing and motivating it was to be taken on a journey of out the box thinking and new vision towards social problems with examples of down to earth achievable solutions.”

“For anyone who is concerned about social problems and wants to make a difference, this course will give you a foundation and spark your thinking. I wish everyone could be exposed to this thinking because I know we can solve these problems, however wicked they may be..”

The course is offered in conjunction with the UCT GSB Bertha Centre for Social Innovation and Entrepreneurship and RLabs – RLabs empowers youth through innovative and disruptive technology by teaching them vital skills and providing much needed support and a sense of community.

Becoming a changemaker is one of seven free online courses currently available from UCT, and several of them have been highly rated by participating students. Earlier in 2016, UCT’s first two MOOCs, What is a Mind, led by Professor Mark Solms and Medicine and the Arts: Humanising healthcare, hosted by Professors Steve Reid and Susan Levine were ranked in Class Central’s top 50 online courses of all time.

With over 80 reviews on Class Central, Prof Solms’ What is a Mind  received the highest praise from learners:

“A very thought provoking and insightful course. Mark Solms is hugely engaging and paces his lectures well. His explanations are careful, very lucid but never boring. The best MOOC I have taken.”

The interdisciplinary, Medicine and the Arts: Humanising healthcare featured 17 academics and healthcare practitioners sharing their perspectives:

“This is an absolutely brilliant course! Each week you look at a topic within the healthcare system through a different perspective related to the arts, i.e. literature, sociology, anthropology, artistic performance. This broad overview really allows you to generate different perspectives and viewpoints. You will be challenged, you will change your views when you do this course.” – Medicine and the Arts

Our other courses, Education for All: Disability, Diversity and Inclusion, Julia Scientific Programming, and Understanding Clinical Research have all garnered positive reviews as well:

“Very enjoyable and thought provoking course. Challenges your views and provides the practical and solution orientated strategies you need to improve this sector in education as a parent / educator/ community member / policy maker / volunteer” – Educational for All

“Finished this course from Coursera and I really enjoyed this one. They provided an interesting analytical programming language that I believed most people haven’t heard nor used it. I really recommended everyone to give it a try, because Julia provided a unique environment for analyze data. And this course provides us a fundamental knowledge of Julia. – Julia Scientific Programming

“This is an excellent course for anyone looking to develop an understanding of the “why” behind statistical analysis! The instructor does a wonderful job of explaining tricky concepts. I now feel better able to understand – and not be afraid of – methods and results sections in medical journal articles. Thanks, Dr. Klopper!” – Understanding Clinical Research

More courses are in production for 2017 – including the soon-to-be launched Extinctions: Past and Present, with renowned palaeobiologist, Professor Anusuya Chinsamy-Turan. We hope the new courses will be as well received as our current courses.

To learn more about our free online courses and to sign up visit: www.cilt.uct.ac.za/cilt/moocs-uct