From bae57eeafb842b1de1444e41ade6637258579416 Mon Sep 17 00:00:00 2001 From: Charlotte Van Petegem Date: Fri, 16 Feb 2024 13:48:39 +0100 Subject: [PATCH] Finish research opportunities --- book.org | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/book.org b/book.org index db30500..5b1b454 100644 --- a/book.org +++ b/book.org @@ -2854,9 +2854,26 @@ Chapter\nbsp{}[[#chap:feedback]] also suggests a number of improvements that cou It gives us a framework for suggesting the feedback a teacher probably wants to give when selecting a line, but we could also try to come up with a confidence score and use that to suggest feedback before the teacher has even done that. Another interesting (more educational) line of research that this work suggests is building the method into an actual assessment platform, and looking at its effects on feedback consistency and quality, time saved by teachers,\nbsp{}... -- Exercise recommendation -- Skill estimation -- Using generative AI to prepare possible answers to student questions +A new idea for research using Dodona's data would be skill estimation. +There are a few ways we could try to infer what skills are being tested by exercises: we could try to use the model solution, or the labels assigned to the exercise in Dodona. +Using those skills, we could try to estimate a student's mastery of those skills, using their submissions. + +This leads right into another possibility for future research: exercise recommendation. +Right now, learning paths in Dodona are static, determined by the author of the course the student is following. +Dodona has a rich library of extra exercises, which is linked to in some courses, but it is not always easy for students to know what exercises would be good for them. +The research from Chapter\nbsp{}[[#chap:passfail]] could also be used to help solve this problem. +If we know a student has a higher chance of failing the course, we might want to recommend some easier exercises. +The other way around, if a student has a higher chance of passing, we could suggest harder exercises, so they can keep up their good progress in their course. + +The final possibility we will present here is to prepare suggestions for answers to student questions on Dodona. +At first glance, LLMs should be quite good at this. +If we use the LLM output as a suggestion for what the teacher could answer, this should be a big time-saver. +However, there are some issues around data quality. +Questions are sometimes asked on a specific line, but the question doesn't necessarily have anything to do with that line. +Sometimes the question also needs context that is hard to pass on to the LLM. +For example, if the question is just "I don't know what's wrong.", a human might look at the failed test cases and be able to answer the "question" in that way. +Passing on the failed test cases to the LLM is a harder problem to solve. +The actual assignment also needs to be passed on, but depending on its size this might also present a problem given token limitations of some models. ** Challenges for the future :PROPERTIES: @@ -2865,9 +2882,9 @@ Another interesting (more educational) line of research that this work suggests - Sustainability of the project - Fairness/integrity of evaluations in general + - Improvements to submission process when in evaluation - Generative AI - Integration of similarity checking (Dolos) -- ... #+LATEX: \appendix * Pass/fail prediction feature types