Catharsis Consulting
Evidence science for organizational impact
Measuring children's language learning with a large technology company using log platform data, causal inference, and curriculum design
Challenge: Our partner was exploring a new product for language learning with young children, and had collected both survey data, evaluation and assessment data, and enormous time series data. They needed a research approach which could leverage machine learning techniques to mine and process the data for patterns, coupled with a learning science perspective that would understand their curricula design and previous research, provide best practice recommendations for measuring learning, and create rigorous statistical models to evaluate the efficacy of the product on learning.
Project & Insights: We provided long-term support for this project, creating iterative models based on data techniques from learning analytics, answering multiple questions about users' behavior, and helping to develop strategy for further measurements between the engineering teams and the curriculum design teams. One surprising insight: that we could learn not only from data within the educational games, but that how children moved between games provided important design recommendations. In true Catharsis Consulting spirit, we focused not only on measures of learning but also on measures of long-term engagement, enjoyment, and motivation, setting our partners up for product design that would have long-term benefits for users.
Bonus deliverable: We took this project beyond our planned internal deliverables and product recommendations to a joint external presentation with a large academic audience. In another bonus deliverable, we led statistical literacy and mentorship sessions with the curricula team, empowering them to not only benefit from our methods, but apply them to new projects.