Handbook of Learning Analytics
Chapter 18
Handbook of Learning Analytics
First Edition
Diverse Big Data and Randomized
Field Experiments in MOOCs
René F. Kizilcec & Christopher Brooks
Abstract
A new mechanism for delivering educational content at large scale, massive open online courses (MOOCs), has attracted millions of learners worldwide. Following a concise account of the recent history of MOOCs, this chapter focuses on their potential as a research instrument. We identify two critical affordances of this environment for advancing research on learning analytics and the science of learning more broadly. The first affordance is the availability of diverse big data in education. Research with heterogeneous samples of learners can advance a more inclusive science of learning, one that better accounts for people from traditionally underrepresented demographic and sociocultural groups in more narrowly obtained educational datasets. The second affordance is the ability to conduct large-scale field experiments at minimal cost. Researchers can quickly evaluate multiple theory-based interventions and draw causal conclusions about their efficacy in an authentic learning environment. Together, diverse big data and experimentation provide evidence on “what works for whom” that can extend theories to account for individual differences and support efforts to effectively target materials and support structures in online learning environments.
Export Citation: Plain Text (APA) BIBTeX RIS
No Supplementary Material Available
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. Jossey-Bass.
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014). Engaging with massive online courses. Proceedings of the 23rd International Conference on World Wide Web (WWW ’14), 7–11 April 2014, Seoul, Republic of Korea (pp. 687–698). New York: ACM.
Baker, R., Dee, T., Evans, B., & John, J. (2015). Bias in online classes: Evidence from a field experiment. Paper presented at the SREE Spring 2015 Conference, Learning Curves: Creating and Sustaining Gains from Early Childhood through Adulthood, 5–7 March 2015, Washington, DC, USA.
Bakshy, E., Eckles, D., & Bernstein, M. S. (2014). Designing and deploying online field experiments. Proceedings of the 23rd International Conference on World Wide Web (WWW ’14), 7–11 April 2014, Seoul, Republic of Korea (pp. 283–292). New York: ACM.
Bather, J. A., & Gittins, J. C. (1990). Multi-armed bandit allocation indices. Journal of the Royal Statistical Society: Series A, 153(2), 257.
Blanchard, E. G. (2012). On the WEIRD nature of ITS/AIED conferences. In S. A. Cerri et al. (Eds.), Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS 2012), 14–18 June 2012, Chania, Greece (pp. 280–285). Lecture Notes in Computer Science. Springer Berlin Heidelberg.
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Studying learning in the worldwide classroom research into edX’s first MOOC. Research & Practice in Assessment, 8. http://www.rpajournal.com/dev/wp-content/uploads/2013/05/SF2.pdf.
Brooks, C., Thompson, C., & Teasley, S. (2015a). A time series interaction analysis method for building predictive models of learners using log data. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 126–135). New York: ACM.
Brooks, C., Thompson, C., & Teasley, S. (2015b). Who you are or what you do: Comparing the predictive power of demographics vs. activity patterns in massive open online courses (MOOCs). Proceedings of the 2nd ACM Conference on Learning @ Scale (L@S 2015), 14–18 March 2015, Vancouver, BC, Canada (pp. 245–248). New York: ACM.
Clow, D. (2013). MOOCs and the funnel of participation. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium. New York: ACM. doi:10.1145/2460296.2460332
Cochran, M. J., & Heron, P. R. L. (2006). Development and assessment of research-based tutorials on heat engines and the second law of thermodynamics. American Journal of Physics, 74(8), 734.
Coetzee, D., Fox, A., Hearst, M. A., & Hartmann, B. (2014). Should your MOOC forum use a reputation system? Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’14), 15–19 February 2014, Baltimore, Maryland, USA (pp. 1176–1187). New York: ACM.
Davis, D., Chen, G., van der Zee, T., Hauff, C., & Houben, G.-J. (2016). Retrieval practice and study planning in MOOCs: Exploring classroom-based self-regulated learning strategies at scale. Proceedings of the 11th European Conference on Technology Enhanced Learning (EC-TEL 2016), 13–16 September 2016, Lyon, France (pp. 57–71). Lecture Notes in Computer Science. Springer International Publishing.
DeBoer, J., Stump, G. S., Seaton, D., & Breslow, L. (2013). Diversity in MOOC students’ backgrounds and behaviors in relationship to performance in 6.002x. Proceedings of the 6th Conference of MIT’s Learning International Networks Consortium (LINC 2013), 16–19 June 2013, Cambridge, Massachusetts, USA. https://tll.mit.edu/sites/default/files/library/LINC%20’13.pdf
Dillahunt, T. R., Ng, S., Fiesta, M., & Wang, Z. (2016). Do massive open online course platforms support employability? Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’16), 27 February–2 March 2016, San Francisco, CA, USA (pp. 233–244). New York: ACM.
Emanuel, E. J. (2013). Online education: MOOCs taken by educated few. Nature, 503(7476), 342.
Festinger, L. (1954). A theory of social comparison processes. Human Relations: Studies towards the Integration of the Social Sciences, 7(2), 117–140.
Gašević, D., Dawson, S., Rogers, T., & Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28(January), 68–84.
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf.
Guo, P. J., & Reinecke, K. (2014). Demographic differences in how students navigate through MOOCs. Proceedings of the 1st ACM Conference on Learning @ Scale (L@S 2014), 4–5 March 2014, Atlanta, Georgia, USA (pp. 21–30). New York: ACM.
Hansen, J. D., & Reich, J. (2015). Democratizing education? Examining access and usage patterns in massive open online courses. Science, 350(6265), 1245–1248.
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? The Behavioral and Brain Sciences, 33(2–3), 61–83; discussion 83–135.
Jonassen, D. H., & Grabowski, B. L. H. (1993). Handbook of individual differences, learning, and instruction. Routledge.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.
Kizilcec, R. F. (2016). How much information? Effects of transparency on trust in an algorithmic interface. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ16), 7–12 May 2016, San Jose, CA, USA (pp. 2390–2395). New York: ACM.
Kizilcec, R. F., Bailenson, J. N., & Gomez, C. J. (2015). The instructor’s face in video instruction: Evidence from two large-scale field studies. Journal of Educational Psychology, 107(3), 724–739.
Kizilcec, R. F., & Halawa, S. (2015). Attrition and achievement gaps in online learning. Proceedings of the 2nd ACM Conference on Learning @ Scale (L@S 2015), 14–18 March 2015, Vancouver, BC, Canada (pp. 57–66). New York: ACM.
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2016). Recommending self-regulated learning strategies does not improve performance in a MOOC. Proceedings of the 3rd ACM Conference on Learning @ Scale (L@S 2016), 25–28 April 2016, Edinburgh, Scotland (pp. 101–104). New York: ACM.
Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium. New York: ACM.
Kizilcec, R. F., Saltarelli, A. J., Reich, J., & Cohen, G. L. (2017). Closing global achievement gaps in MOOCs. Science, 355(6322), 251-252.
Kizilcec, R. F., & Schneider, E. (2015). Motivation as a lens to understand online learners: Toward data-driven design with the OLEI scale. ACM Transactions on Computer–Human Interaction, 22(2), 1–24.
Kizilcec, R. F., Schneider, E., Cohen, G. L., & McFarland, D. A. (2014). Encouraging forum participation in online courses with collectivist, individualist and neutral motivational framings. eLearning Papers, 37, 13–22.
Koedinger, K. R., Booth, J. L., & Klahr, D. (2013). Instructional complexity and the science to constrain it. Science, 342(6161), 935–937.
Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603.
Kulkarni, C., Cambre, J., Kotturi, Y., Bernstein, M. S., & Klemmer, S. R. (2015). Talkabout: Making distance matter with small groups in massive classes. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15), 14–18 March 2015, Vancouver, BC, Canada (pp. 1116–1128). New York: ACM.
Lamb, A., Smilack, J., Ho, A., & Reich, J. (2015). Addressing common analytic challenges to randomized experiments in MOOCs: Attrition and zero-inflation. Proceedings of the 2nd ACM Conference on Learning @ Scale (L@S 2015), 14–18 March 2015, Vancouver, BC, Canada (pp. 21–30). New York: ACM.
Laney, D. (2001). 3D data management: Controlling data volume, velocity and variety. META Group Research Note, 6, 70.
Li, J. (2012). Cultural foundations of learning: East and west. Cambridge University Press.
Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304–316.
Martinez, I. (2014). MOOCs as a massive research laboratory. University of Virginia.
Mayer, R. E. (2001). Multimedia learning. Cambridge University Press.
Mitchell, S. D. (2009). Unsimple truths: Science, complexity, and policy. University of Chicago Press.
Mullaney, T., & Reich, J. (2015). Staggered versus all-at-once content release in massive open online courses: Evaluating a natural experiment. Proceedings of the 2nd ACM Conference on Learning @ Scale (L@S 2015), 14–18 March 2015, Vancouver, BC, Canada (pp. 185–194). New York: ACM.
Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., & Heffernan, C. (2014). Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology, 45(3), 487–501.
Pappano, L. (2012, November 2). The year of the MOOC. The New York Times. http://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html
Reich, J. (2014, December 8). MOOC completion and retention in the context of student intent. EDUCAUSE Review Online. http://er.educause.edu/articles/2014/12/mooc-completion-and-retention-in-the-context-of-student-intent
Reich, J. (2015). Rebooting MOOC research. Science, 347(6217), 34–35.
Renz, J., Hoffmann, D., Staubitz, T., & Meinel, C. (2016). Using A/B testing in MOOC environments. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 304–313). New York: ACM.
Rienties, B., & Tempelaar, D. (2013). The role of cultural dimensions of international and Dutch students on academic and social integration and academic performance in the Netherlands. International Journal of Intercultural Relations, 37(2), 188–201.
Rogers, T., & Feller, A. (2016). Discouraged by peer excellence: Exposure to exemplary peer performance causes quitting. Psychological Science, 27(3), 365–374.
Rose, T. (2016). The end of average: How we succeed in a world that values sameness. HarperCollins.
Shah, D. (2015, December 28). MOOCs in 2015: Breaking down the numbers. EdSurge. https://www.edsurge.com/news/2015-12-28-moocs-in-2015-breaking-down-the-numbers
Siemens, G. (2013). Massive open online courses: Innovation in education? In R. McGreal, W. Kinuthia, & S. Marshall (Eds.), Open educational resources: Innovation, research and practice (pp. 5–16). Edmonton, AB: Athabasca University Press.
Simonson, M., Smaldino, S. E., Albright, M., & Zvacek, S. (2011). Teaching and learning at a distance: Foundations of distance education. Pearson Higher Ed.
Steele, C. M., Spencer, S. J., & Joshua, A. (2002). Contending with group image: The psychology of stereotype and social identity threat. Advances in Experimental Social Psychology, 34, 379–440.
Thaler, R. H., & Sunstein, C. R. (2009). Nudge: Improving decisions about health, wealth, and happiness. Penguin.
Thille, C., Schneider, E., Kizilcec, R. F., Piech, C., Halawa, S. A., & Greene, D. K. (2014). The future of data-enriched assessment. Research & Practice in Assessment, 9, 5–16.
Tomkin, J. H., & Charlevoix, D. (2014). Do professors matter? Using an A/B test to evaluate the impact of instructor involvement on MOOC student outcomes. Proceedings of the 1st ACM Conference on Learning @ Scale (L@S 2014), 4–5 March 2014, Atlanta, Georgia, USA (pp. 71–78). New York: ACM.
Waldrop, M. M. (2013). Online learning: Campus 2.0. Nature, 495(7440), 160–163.
Walton, G. M., & Cohen, G. L. (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92(1), 82–96.
Williams, J. J., Li, N., Kim, J., Whitehill, J., Maldonado, S., Pechenizkiy, M., Chu, L., & Heffernan, N. (2014). The MOOClet framework: Improving online education through experimentation and personalization of modules. Working Paper No. 2523265, Social Science Research Network. doi:10.2139/ssrn.2523265
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686–688.
Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. J. (2015, September 22). Who’s benefiting from MOOCs, and why. Harvard Business Review. https://hbr.org/2015/09/whos-benefiting-from-moocs-and-why
Zhou, Y., Jindal-Snape, D., Topping, K., & Todman, J. (2008). Theoretical models of culture shock and adaptation in international students in higher education. Studies in Higher Education, 33(1), 63–75.
Title
Diverse Big Data and Randomized Field Experiments in MOOCs
Book Title
Handbook of Learning Analytics
Pages
pp. 211-222
Copyright
2017
DOI
10.18608/hla17.018
ISBN
978-0-9952408-0-3
Publisher
Society for Learning Analytics Research
Authors
René F. Kizilcec1
Christopher Brooks2
Author Affiliations
1. Department of Communication, Stanford University, USA
2. School of Information, University of Michigan, USA
Editors
Charles Lang3
George Siemens4
Alyssa Wise5
Dragan Gašević6
Editor Affiliations
3. Teachers College, Columbia University, USA
4. LINK Research Lab, University of Texas at Arlington, USA
5. Learning Analytics Research Network, New York University, USA
6. Schools of Education and Informatics, University of Edinburgh, UK