Research Article: Visualizing learner engagement, performance, and trajectories to evaluate and optimize online course design

Date Published: May 6, 2019

Publisher: Public Library of Science

Author(s): Michael Ginda, Michael C. Richey, Mark Cousino, Katy Börner, Andrew R. Dalby.


Learning analytics and visualizations make it possible to examine and communicate learners’ engagement, performance, and trajectories in online courses to evaluate and optimize course design for learners. This is particularly valuable for workforce training involving employees who need to acquire new knowledge in the most effective manner. This paper introduces a set of metrics and visualizations that aim to capture key dynamical aspects of learner engagement, performance, and course trajectories. The metrics are applied to identify prototypical behavior and learning pathways through and interactions with course content, activities, and assessments. The approach is exemplified and empirically validated using more than 30 million separate logged events that capture activities of 1,608 Boeing engineers taking the MITxPro Course, “Architecture of Complex Systems,” delivered in Fall 2016. Visualization results show course structure and patterns of learner interactions with course material, activities, and assessments. Tree visualizations are used to represent course hierarchical structures and explicit sequence of content modules. Learner trajectory networks represent pathways and interactions of individual learners through course modules, revealing patterns of learner engagement, content access strategies, and performance. Results provide evidence for instructors and course designers for evaluating the usage and effectiveness of course materials and intervention strategies.

Partial Text

In the information age, skills and knowledge required to perform professional jobs are changing rapidly. Proactive up-skilling and retraining of people are critical. Companies are spending billions of dollars each year to develop courses, train their existing workforce, and onboard new hires. Many companies resort to mass training; the majority being inefficient web based and costly instructor lead training with some companies innovating in teaching and learning analytics to increase return on investment (ROI) for the many diverse learning and training interventions [1].

Learning analytics (LA) refers to the measurement, modelling, and communication of learner data to understand and optimize teaching and learning and the socio-technical environment in which it occurs [14, 15]. With the rise of MOOCs and technical learning platforms (e.g., edX, Udacity, Pluralsight, Udemy, Lynda), applications in workforce training have increased, driving investments in LA applied to data generated by millions of learners taking online classes [16, 17]. LA uses diverse temporal, geospatial, topical, and network analyses and visualizations to answer when, where, what, and with whom questions [9]. LA provides predictions (e.g., of at risk learners, procrastination, and concept mastery), supports authentic learning (e.g., learning modules and paths that emerge based on authentic individual interest and engagement); increases our understanding of how people learn; and quantify return on evidence based instructional design that enables learners to explore and individual tailored and authentic learning paths.

The studies described above use a variety of measures and metrics to analyze learner engagement and transitions in online courses. Some common themes emerge in the metrics and variables used in these studies that can be broken into module use (node) and transition (edge) measures. Module measures include: total number of events [22] and dwell times for using modules [22, 25], total number of learners active or participating in a module [22–26]. Transition measures include the proportion of learners transitioning between modules [22–24, 27] and the time between transitions [25–27] and designed course paths and state transition probabilities [27]. Likewise, these studies have classified data for different types of engagement, including assignment submissions [23], grade performance after first two weeks [24], and certificate status for courses [27].

Study results presented here showcase the value of visualizations for examining and communicating learners’ engagement, performance, and trajectories in online courses. The data analysis and visualization methods were exemplified using an extensive corporate dataset of 1,608 Boeing engineers. Visualization results show course structures and patterns of learner interactions with course material, activities, and assessments. They can be used to optimize the sequence and content of online course materials, to inform learners and instructors about learning progress. Novel learner trajectory visualizations make it possible to understand and communicate how individual learners access course content modules, revealing patterns of learner engagement, content access strategies, social interactions, and performance. They aim to empower instructors and corporate partners to optimize current courses and to develop more efficient future courses.




Leave a Reply

Your email address will not be published.