Handbook of Learning Analytics
Chapter 6
Handbook of Learning Analytics
First Edition
Going Beyond Better Data Prediction to
Create Explanatory Models of Educational Data
Ran Liu & Kenneth R. Koedinger
Abstract
In the statistical modelling of educational data, approaches vary depending on whether the goal is to build a predictive or an explanatory model. Predictive models aim to find a combination of features that best predict outcomes; they are typically assessed by their accuracy in predicting held-out data. Explanatory models seek to identify interpretable causal relationships between constructs that can be either observed or inferred from the data. The vast majority of educational data mining research has focused on achieving predictive accuracy, but we argue that the field could benefit from more focus on developing explanatory models. We review examples of educational data mining efforts that have produced explanatory models and led to improvements to learning outcomes and/or learning theory. We also summarize some of the common characteristics of explanatory models, such as having parameters that map to interpretable constructs, having fewer parameters overall, and involving human input early in the model development process.
Export Citation: Plain Text (APA) BIBTeX RIS
No Supplementary Material Available
Barnes, T. (2005). The Q-matrix method: Mining student response data for knowledge. Proceedings of AAAI 2005: Educational Data Mining Workshop (pp. 39–46). Technical Report WS-05-02. Menlo Park, CA: AAAI Press. http://www.aaai.org/Library/Workshops/ws05-02.php
Cen, H., Koedinger, K. R., & Junker, B. (2006). Learning factors analysis: A general method for cognitive model evaluation and improvement. In M. Ikeda, K. Ashlay, T.-W. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS 2006), 26–30 June 2006, Jhongli, Taiwan (pp. 164–175). Berlin: Springer-Verlag.
Clark, R. E., Feldon, D., van Merriënboer, J., Yates, K., & Early, S. (2008). Cognitive task analysis. In J. M. Spector, M. D. Merrill, J. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling & User-Adapted Interaction, 4, 253–278.
D’Mello, S., Blanchard, N., Baker, R., Ocumpaugh, J., & Brawner, K. (2014). I feel your pain: A selective review of affect sensitive instructional strategies. In R. Sottilare, A. Graesser, X. Hu, & B. Goldberg (Eds.), Design recommendations for adaptive intelligent tutoring systems: Adaptive instructional strategies (Vol. 2). Orlando, FL: US Army Research Laboratory.
D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., & Graesser, A. (2010). A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (ITS 2010), 14–18 June 2010, Pittsburgh, PA, USA (pp. 245–254). Springer.
González-Brenes, J. P., & Mostow, J. (2012). Dynamic cognitive tracing: Towards unified discovery of student and cognitive models. In K. Yacef, O. Zaïane, A. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (EDM2012), 19–21 June, 2012, Chania, Greece (pp. 49–56). International Educational Data Mining Society.
Koedinger, K. R., Baker, R. S. J. d., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. d. Baker (Eds.), Handbook of educational data mining. Boca Raton, FL: CRC Press.
Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798.
Koedinger, K. R., & McLaughlin, E. A. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (CogSci 2010), 11–14 August 2010, Portland, OR, USA (pp. 471–476). Austin, TX: Cognitive Science Society.
Koedinger, K. R., McLaughlin, E. A., & Stamper, J. C. (2012). Automated cognitive model improvement. In K. Yacef, O. Zaïane, A. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (EDM2012), 19–21 June, 2012, Chania, Greece (pp. 17–24). International Educational Data Mining Society.
Koedinger, K. R., & Nathan, M. J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences, 13(2), 129–164.
Koedinger, K. R., Stamper, J. C., McLaughlin, E. A., & Nixon, T. (2013). Using data-driven discovery of better cognitive models to improve student learning. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Proceedings of the 16th International Conference on Artificial Intelligence in Education (AIED ʼ13), 9–13 July 2013, Memphis, TN, USA (pp. 421–430). Springer.
Lan, A. S., Studer, C., Waters, A. E., & Baraniuk, R. G. (2013). Tag-aware ordinal sparse factor analysis for learning and content analytics. In S. K. DʼMello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (EDM2013), 6–9 July, Memphis, TN, USA (pp. 90–97). International Educational Data Mining Society/Springer.
Lan, A. S., Studer, C., Waters, A. E., & Baraniuk, R. G. (2014). Sparse factor analysis for learning and content analytics. Journal of Machine Learning Research, 15, 1959–2008.
Li, N., Cohen, W., Koedinger, K. R., & Matsuda, N. (2011). A machine learning approach for automatic student model discovery. In M. Pechenizkiy et al. (Eds.), Proceedings of the 4th International Conference on Education Data Mining (EDM2011), 6–8 July 11, Eindhoven, Netherlands (pp. 31–40). International Educational Data Mining Society.
Li, N., Matsuda, N., Cohen, W. W., & Koedinger, K. R. (2015). Integrating representation learning and skill learning in a human-like intelligent agent. Artificial Intelligence, 219, 67–91.
Lindsey, R. V., Khajah, M., & Mozer, M. C. (2014). Automatic discovery of cognitive skills to improve the prediction of student learning. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, & K. Q. Weinberge (Eds.), Advances in Neural Information Processing Systems, 27, 1386–1394. La Jolla, CA: Curran Associates Inc.
Liu, R., & Koedinger, K. R. (submitted). Closing the loop: Automated data-driven skill model discoveries lead to improved instruction and learning gains. Journal of Educational Data Mining.
Liu, R., & Koedinger, K. R. (2015). Variations in learning rate: Student classification based on systematic residual error patterns across practice opportunities. In O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, & M. Desmarais (Eds.), Proceedings of the 8th International Conference on Education Data Mining (EDM2015), 26–29 June 2015, Madrid, Spain (pp. 420–423). International Educational Data Mining Society.
Liu, R., Koedinger, K. R., & McLaughlin, E. A. (2014). Interpreting model discovery and testing generalization to a new dataset. In J. Stamper, Z. Pardos, M. Mavrikis, & B. M. McLaren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM2014), 4–7 July, London, UK (pp. 107–113). International Educational Data Mining Society.
MacLellan, C. J., Harpstead, E., Patel, R., & Koedinger, K. R. (2016). The apprentice learner architecture: Closing the loop between learning theory and educational data. In T. Barnes, M. Chi, & M. Feng (Eds.), Proceedings of the 9th International Conference on Educational Data Mining (EDM2016), 29 June–2 July 2016, Raleigh, NC, USA (pp. 151–158). International Educational Data Mining Society.
Nathan, M. J., Koedinger, K. R., & Alibali, M. W. (2001). Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In L. Chen et al. (Eds.), Proceedings of the 3rd International Conference on Cognitive Science (pp. 644–648). Beijing, China: USTC Press. http://pact.cs.cmu.edu/pubs/2001_NathanEtAl_ICCS_EBS.pdf
Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
Pardos, Z. A., Trivedi, S., Heffernan, N. T., & Sárközy, G. N. (2012). Clustered knowledge tracing. In S. A. Cerri, W. J. Clancey, G. Papadourakis, K.-K. Panourgia (Eds.), Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS 2012), 14–18 June 2012, Chania, Greece (pp. 405–410). Springer.
Rosé, C. P., & Tovares, A. (in press). What sociolinguistics and machine learning have to say to one another about interaction analysis. In L. Resnick, C. Asterhan, & S. Clarke (Eds.), Socializing intelligence through academic talk and dialogue. Washington, DC: American Educational Research Association.
Rosé, C. P, & VanLehn, K. (2005). An evaluation of a hybrid language understanding approach for robust selection of tutoring goals. International Journal of Artificial Intelligence in Education, 15, 325–355.
San Pedro, M., Baker, R. S., Bowers, A. J., & Heffernan, N. T. (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In S. K. DʼMello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (EDM2013), 6–9 July, Memphis, TN, USA (pp. 177–184). International Educational Data Mining Society/Springer.
Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289–310. doi:10.1214/10-STS330
Stamper, J., & Koedinger, K. R. (2011). Human-machine student model discovery and improvement using data. Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED ʼ11), 28 June–2 July, Auckland, New Zealand (pp. 353–360). Springer.
Trivedi, S., Pardos, Z. A., & Heffernan, N. T. (2011). Clustering students to generate an ensemble to improve standard test score predictions. Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED ʼ11), 28 June–2 July, Auckland, New Zealand (pp. 377–384). Springer.
VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16, 227–265.
Winne, P., & Baker, R. (2013). The potentials of educational data mining for researching metacognition, motivation and self-regulated learning. Journal of Educational Data Mining, 5, 1–8.
Title
Going Beyond Better Data Prediction to Create Explanatory Models of Educational Data
Book Title
Handbook of Learning Analytics
Pages
pp. 69-76
Copyright
2017
DOI
10.18608/hla17.006
ISBN
978-0-9952408-0-3
Publisher
Society for Learning Analytics Research
Authors
Ran Liu
Kenneth R. Koedinger
Author Affiliations
School of Computer Science, Carnegie Mellon University, USA
Editors
Charles Lang1
George Siemens2
Alyssa Wise3
Dragan Gašević4
Editor Affiliations
1. Teachers College, Columbia University, USA
2. LINK Research Lab, University of Texas at Arlington, USA
3. Learning Analytics Research Network, New York University, USA
4. Schools of Education and Informatics, University of Edinburgh, UK