phd-thesis/book.org

#+TITLE: Dodona
#+SUBTITLE: Improving programming education through automated assessment, learning analytics, and educational data mining
#+AUTHOR: Charlotte Van Petegem
#+LANGUAGE: en-gb
#+DATE: 2024-06-19
#+LATEX_CLASS: book
#+LATEX_CLASS_OPTIONS: [paper=240mm:170mm,parskip=half-,numbers=noendperiod,BCOR=10mm,DIV=10]
#+LATEX_COMPILER: lualatex
#+LATEX_HEADER: \usepackage[inline]{enumitem}
#+LATEX_HEADER: \usepackage{luacode}
#+LATEX_HEADER: \begin{luacode*}
#+LATEX_HEADER:   function parseargv()
#+LATEX_HEADER:     local rep = {}
#+LATEX_HEADER:     for k, x in pairs(arg) do
#+LATEX_HEADER:         local kw, vw = string.match(x, "([^=]+)=?([^=]*)")
#+LATEX_HEADER:         rep[kw] = vw
#+LATEX_HEADER:     end
#+LATEX_HEADER:     return rep
#+LATEX_HEADER:   end
#+LATEX_HEADER:   local arguments = parseargv()
#+LATEX_HEADER:   local outputdir = arguments["-output-directory"]
#+LATEX_HEADER:   if outputdir ~= nil then
#+LATEX_HEADER:     tex.print([[\PassOptionsToPackage{outputdir={]]..outputdir..[[}}{minted}]])
#+LATEX_HEADER:     tex.print([[\PassOptionsToPackage{inkscapepath={]]..outputdir..[[}}{svg}]])
#+LATEX_HEADER:   end
#+LATEX_HEADER: \end{luacode*}
#+LATEX_HEADER: \usepackage[newfloat]{minted}
#+LATEX_HEADER: \usepackage{color}
#+LATEX_HEADER: \usepackage{url}
#+LATEX_HEADER: \usepackage{svg}
#+LATEX_HEADER: \usepackage[type=report]{ugent2016-title}
#+LATEX_HEADER: \usepackage[final]{microtype}
#+LATEX_HEADER: \usepackage[defaultlines=2,all]{nowidow}
#+LATEX_HEADER: \usepackage[dutch,AUTO]{polyglossia}
#+LATEX_HEADER: \usepackage{ragged2e}
#+LATEX_HEADER: \newenvironment{RIGHT}{\begin{FlushRight}}{\end{FlushRight}}
#+HTML_HEAD: <style>.RIGHT {text-align: right;}</style>
#+LATEX_HEADER: \academicyear{2023–2024}
#+LATEX_HEADER: \titletext{A dissertation submitted to Ghent University in partial fulfilment of\\ the requirements for the degree of Doctor of Computer Science.}
#+LATEX_HEADER: \promotors{%
#+LATEX_HEADER: Supervisors:\\
#+LATEX_HEADER: Prof.\ Dr.\ Peter Dawyndt\\
#+LATEX_HEADER: Prof.\ Dr.\ Ir.\ Bart Mesuere\\
#+LATEX_HEADER: Prof.\ Dr.\ Bram De Wever
#+LATEX_HEADER: }
#+LATEX_HEADER: \addtokomafont{caption}{\small}
#+LATEX_HEADER: \setuptoc{toc}{numbered}
#+LATEX_HEADER: \addto\captionsenglish{\renewcommand{\contentsname}{Table of Contents}}
#+OPTIONS: ':t
#+OPTIONS: H:4
#+OPTIONS: toc:nil
#+OPTIONS: broken-links:mark
#+MACRO: num_commits 16 thousand
#+MACRO: num_prs 3\thinsp{}800
#+MACRO: num_contributors 26
#+MACRO: num_exercises 16 thousand
#+MACRO: num_releases 340
#+MACRO: num_schools 1\thinsp{}700
#+MACRO: num_submissions 17 million
#+MACRO: num_users 66\thinsp{}500
#+cite_export: csl citation-style.csl
#+bibliography: bibliography.bib

#+LATEX: \frontmatter
#+TOC: headlines 2

* Dankwoord :noexport:
:PROPERTIES:
:CREATED: [2023-10-23 Mon 09:25]
:CUSTOM_ID: chap:ack
:UNNUMBERED: t
:END:

#+LATEX: \begin{dutch}

Familie:
Mama.
Papa.
Hannelore, Tomas, Seppe.
Robbe, Esther.
Kero Kero.

Promotoren en jury.

Werk:
Rien.
Simon.
Niko.
Alexis.
Asmus.
Carol.
Dieter.
Steven.
Louise.
Robbert.
Tom.
Jonathan.
Heidi.
Felix.
Toon.
Pieter.
Tibo.
Mustapha.
Nico & Joyce.
Benjamin.
Oliver.
Roy.

Mede-lesgevers:
Annick.
Henri.
Adnan.
Niko.
Felix.
Louise.
Toon.
Lotte.
Yentl.
Felipe.
Silvija.
Antoine.
Oliver.
Dieter.
Ellen.
Tibo.

Zeus:
Jasper.
Klimcrew (Tom "gewoon doorstappen" Naessens, Felix, Ruben, Arthur, Titouan).

Rode Kruis:
Luc.
Wim.
Sarah.
Henk.
Pascal.
Jonas.
Rien.
Jietse.

Anje De Baets van de resto.

D&D:
Bart.
Kenneth.
Maxiem.
Arne.

Muziek:
Jan Swerts.
Eliza McLamb.
Katy Kirby.
Marika Hackman.
Boygenius: Phoebe Bridgers, Lucy Dacus, Julien Baker.
Pinegrove.
Charlotte Cardin.
Tate McRae.
Spinvis.
SOPHIE.
ANOHNI.

#+BEGIN_RIGHT
{{{author}}}

{{{date}}}
#+END_RIGHT

#+LATEX: \end{dutch}

* Summary in English
:PROPERTIES:
:CREATED: [2023-10-23 Mon 17:54]
:CUSTOM_ID: chap:summen
:UNNUMBERED: t
:END:

Ever since programming has been taught, its teachers have sought to automate and optimize their teaching.
Due to the ever-increasing digitalization of society, programming is also being taught to ever more and ever larger groups, and these groups often include students for whom programming is not necessarily their main subject.
This has led to the development of myriad automated assessment tools\nbsp{}[cite:@paivaAutomatedAssessmentComputer2022; @ihantolaReviewRecentSystems2010; @douceAutomaticTestbasedAssessment2005; @ala-mutkaSurveyAutomatedAssessment2005].
One of those platforms is Dodona[fn:: https://dodona.be], which is the platform this dissertation is centred around.

Chapters\nbsp{}[[#chap:what]],\nbsp{}[[#chap:use]]\nbsp{},\nbsp{}and\nbsp{}[[#chap:technical]] focus on Dodona itself.
In Chapter\nbsp{}[[#chap:what]] we give an overview of the user-facing features of Dodona, from user management to how feedback is represented.
Chapter\nbsp{}[[#chap:use]] then focuses on how Dodona is used in practice, by presenting some facts and figures of its use, students' opinions of the platform, and an extensive case study on how Dodona's features are used to optimize teaching.
Chapter\nbsp{}[[#chap:technical]] focuses on the technical aspects of developing Dodona and its related ecosystem of software tools.
This includes a discussion of the technical challenges related to developing a platform like Dodona, and how the Dodona team adheres to modern standards of software development.

Chapters\nbsp{}[[#chap:passfail]]\nbsp{}and\nbsp{}[[#chap:feedback]] are a bit different.
These chapters each detail a learning analytics/educational mining study we did, using the data that Dodona collects about the learning process.
Learning analytics and educational data mining stand at the intersection of computer science, data analytics, and the social sciences, and focus on understanding and improving learning.
They are made possible by the increased availability of data about students who are learning, due to the increasing move of education to digital platforms\nbsp{}[cite:@romeroDataMiningCourse2008].
They can also serve different actors in the educational landscape: they can help learners directly, help teachers to evaluate their own teaching, allow developers of education platforms to know what to focus on, allow educational institutions to guide their decisions, and even allow governments to take on data-driven policies\nbsp{}[cite:@fergusonLearningAnalyticsDrivers2012].

Chapter\nbsp{}[[#chap:passfail]] discusses a study where we tried to predict whether students would pass or fail a course at the end of the semester based solely on their submission history in Dodona.
It also briefly details a study we collaborated on with researchers from Jyväskylä University in Finland, where we replicated our study in their educational context, with data from their educational platform.

In Chapter\nbsp{}[[#chap:feedback]], we first give an overview of how Dodona changed manual assessment in our own educational context.
We then finish the chapter with some recent work on a machine learning method we developed to predict what feedback teachers will give when manually assessing student submissions.

Finally, Chapter\nbsp{}[[#chap:discussion]] concludes the dissertation with some discussion on the current status of Dodona, the research related to it, and Dodona's challenges for the future.

* Nederlandstalige samenvatting
:PROPERTIES:
:CREATED: [2023-10-23 Mon 17:54]
:CUSTOM_ID: chap:summnl
:UNNUMBERED: t
:END:

#+LATEX: \begin{dutch}

Sinds het begin van het programmeeronderwijs, proberen de lesgevers die anderen het programmeren aanleren hun onderwijs te automatiseren en optimaliseren.
Onze samenleving wordt ook steeds digitaler, waardoor programmeren wordt aangeleerd aan steeds grotere groepen.
Deze groepen bevatten ook vaker studenten voor wie programmeren niet het hoofdonderwerp van hun studies is.
Dit heeft geleid tot de ontwikkeling van zeer veel platformen voor de geautomatiseerde beoordeling van programmeeropdrachten\nbsp{}[cite:@paivaAutomatedAssessmentComputer2022; @ihantolaReviewRecentSystems2010; @douceAutomaticTestbasedAssessment2005; @ala-mutkaSurveyAutomatedAssessment2005].
Eén van deze platformen is Dodona[fn:: https://dodona.be], wat het platform is waar dit proefschrift over handelt.

Hoofdstukken\nbsp{}[[#chap:what]],\nbsp{}[[#chap:use]]\nbsp{}en\nbsp{}[[#chap:technical]] focussen op Dodona zelf.
In Hoofdstuk\nbsp{}[[#chap:what]] geven we een overzicht van de gebruikersgerichte features van Dodona, van gebruikersbeheer tot hoe feedback getoond wordt.
Hoofdstuk\nbsp{}[[#chap:use]] focust zich dan op hoe Dodona in de praktijk gebruikt wordt, door wat statistieken over het gebruiken te presenteren, de meningen van studenten over het platform te presenteren en door in een uitgebreide case study aan bod te laten komen hoe de features van Dodona gebruikt kunnen worden om het lesgeven te optimaliseren.
Hoofdstuk\nbsp{}[[#chap:technical]] focust op het technische aspect van het ontwikkelen van Dodona en het gerelateerde ecosysteem van software.
Dit bevat onder meer een bespreking van de technische uitdagingen gerelatteerd aand het ontwikkelen van een platform zoals Dodona en hoe het Dodona-team zich houdt aan de moderne standaarden van softwareontwikkeling.

Hoofdstukken\nbsp{}[[#chap:passfail]]\nbsp{}en\nbsp{}[[#chap:feedback]] zijn anders.
Deze hoofdstukken bespreken ieders een /learning analytics/educational data mining/ studie die we uitgevoerd hebben met de data die Dodona genereert.
/Learning analytics/ en /educational data mining/ bevinden zich op het kruispunt tussen informatica, datawetenschap en de sociale wetenschappen, en focussen zich op het begrijpen en verbeteren van leren.
Ze worden mogelijk gemaakt door de toegenomen beschikbaarheid van data over lerende studenten, op zijn beurt door de toegenomen beweging van onderwijs naar digitale platformen\nbsp{}[cite:@romeroDataMiningCourse2008].
Ze kunnen ook dienen voor verschillende actoren in het onderwijsveld: ze kunnen de lerenden direct helpen, lesgevers helpen om hun eigen onderwijs te evalueren, ontwikkelaars van onderwijsplatformen laten weten waar ze zich om moeten focussen, de beslissingen van onderwijsinstellingen helpen gidsen, en zelfs overheden toelaten om op data gebaseerd beleid te voeren\nbsp{}[cite:@fergusonLearningAnalyticsDrivers2012].

Hoofdstuk\nbsp{}[[#chap:passfail]] bespreekt een studie waarin we geprobeerd hebben te voorspellen of studenten al dan niet zouden slagen voor een vak op het einde van het semester, enkel en alleen gebaseerd op hun indiengedrag op Dodona.
Daarnaast wordt er kort een samenwerking besproken met onderzoekers van de universiteit van Jyväskylä in Finland, waar we onze studie herhaald hebben in hun educationele context, met data van hun platform.

In Hoofdstuk\nbsp{}[[#chap:feedback]] geven we eerst een overzich van hoe Dodona het manueel verbeteren in onze eigen educationele context veranderd heeft.
We sluiten dan het hoofdstuk af met een recent door ons ontwikkelde /machine-learning/-methode om te voorspellen welke feedback lesgevers zullen geven tijden het manueel verbeteren van indieningen van studenten.

We sluiten af in Hoofdstuk\nbsp{}[[#chap:discussion]] met een bespreking van de huidige status van Dodona, het onderzoek eraan gerelateerd en de uitdagingen waar Dodona in de toekomst voor staat.

#+LATEX: \end{dutch}

#+LATEX: \mainmatter

* Introduction
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:47]
:CUSTOM_ID: chap:intro
:END:

Ever since programming has been taught, its teachers have sought to automate and optimize their teaching.
Due to the ever-increasing digitalization of society, programming is also being taught to ever more and ever larger groups, and these groups often include students for whom programming is not necessarily their main subject.
This has led to the development of myriad automated assessment tools\nbsp{}[cite:@paivaAutomatedAssessmentComputer2022; @ihantolaReviewRecentSystems2010; @douceAutomaticTestbasedAssessment2005; @ala-mutkaSurveyAutomatedAssessment2005], of which we give a historical overview in this introduction.
We also discuss learning analytics and educational data mining, and how these techniques can help us to cope with the growing class sizes.
Finally, we give a brief overview of the remaining chapters of this dissertation.

** Automated assessment in programming education
:PROPERTIES:
:CREATED:  [2024-02-01 Thu 10:46]
:CUSTOM_ID: sec:introhistory
:END:

Increasing interactivity in learning has long been considered important, and also something that can be achieved through the addition of (web-based) IT components to a course\nbsp{}[cite:@vanpetegemPowerfulLearningInteractive2004].
This isn't any different when learning to program: learning how to solve problems with computer programs requires practice, and programming assignments are the main way in which such practice is generated\nbsp{}[cite:@gibbsConditionsWhichAssessment2005].
[cite/t:@cheangAutomatedGradingProgramming2003] identified the labor-intensive nature of assessing programming assignments as the main reason why students are given few such assignments when in an ideal world they should be given many more.
Automated assessment allows students to receive immediate and personalized feedback on each submitted solution without the need for human intervention.
Because of its potential to provide feedback loops that are scalable and responsive enough for an active learning environment, automated source code assessment has become a driving force in programming courses.

*** Humble beginnings
:PROPERTIES:
:CREATED:  [2024-02-06 Tue 15:30]
:END:

Automated assessment was introduced into programming education in the late 1950s\nbsp{}[cite:@hollingsworthAutomaticGradersProgramming1960].
In this first system, programs were submitted in assembly on punch cards.
For the reader who is not familiar with punch cards, an example of one can be seen in Figure\nbsp{}[[fig:introductionpunchard]].
The assessment was then performed by combining the student's punch cards with the autograder's punch cards.
In the early days of computing, the time of tutors was not the only valuable resource that needed to be shared between students; the actual compute time was also a shared and limited resource.
Their system made more efficient use of both.
[cite/t:@hollingsworthAutomaticGradersProgramming1960] already notes that the class sizes were a main motivator to introduce their auto-grader.
At the time of publication, they had tested about 3\thinsp{}000 student submissions which, given a grading run took about 30 to 60 seconds, represents about a day and a half of computation time.

They also immediately identified some limitations, which are common problems that modern graders still need to consider.
These limitations include handling faults in the student code, making sure students can't modify the grader, and having to define an interface through which the student code is run.

#+CAPTION: Example of a punch card.
#+CAPTION: Picture by Arnold Reinhold, released under the CC BY-SA 4.0 licence via WikiMedia Commons.
#+NAME: fig:introductionpunchard
[[./images/introductionpunchcard.jpg]]

In the next ten years, significant advances were already made.
Students could submit code written in a text-based programming language instead of assembly, and the actual testing was done by running their code using modified compilers and operating systems.

[cite/t:@naurAutomaticGradingStudents1964] was the first to explicitly note the difference between the formal correctness, and the efficiency and completeness of the programs being tested.
The distinction between formal correctness and completeness that he makes can be somewhat confusing from a modern standpoint: we would only consider a program or algorithm formally correct if it is complete (i.e. gives the correct response in all cases).
In more modern terminology, Naur's "formally correct" would be called "free of syntax errors".

[cite/t:@forsytheAutomaticGradingPrograms1965] note another issue when using automatic graders: students could use the feedback they get to hard-code the expected response in their programs.
This is again an issue that modern assessment platforms (or the teachers creating exercises) still need to consider.
Forsythe & Wirth solve this issue by randomizing the inputs to the student's program.
While not explicitly explained by them, we can assume that to check the correctness of a student's answer, they calculate the expected answer themselves as well.
Note that in this system, they were still writing a grading program for each individual exercise.

[cite/t:@hextAutomaticGradingScheme1969] introduce a new innovation: their system could be used for exercises in multiple different programming languages.
They are also the first to implement a history of student's attempts in the assessment tool itself, and mention explicitly that enough data should be recorded in this history so that it can be used to calculate a mark for a student.

Other grader programs were in use at the time, but these did not necessarily bring any new innovations or ideas to the table\nbsp{}[cite:@braden1965introductory; @berryGraderPrograms1966; @temperlyGradingProcedurePL1968].

The systems described above share an important limitation, which is inherent to the time at which they were built.
Computers were big and heavy, and had operators who did not necessarily know whose program they were running or what those programs were.
The Mother of All Demos by\nbsp{}[cite/t:@engelbart1968research], widely considered the birth of the /idea/ of the personal computer, only happened after these systems were already running.
So, it should not come as a surprise that the feedback these systems gave was slow to return to the students.

*** Tool- and script-based assessment
:PROPERTIES:
:CREATED:  [2024-02-06 Tue 17:29]
:END:

We now take a leap forward in time.
The way people use computers has changed significantly, and the way assessment systems are implemented changed accordingly.
Note that while the previous section was complete (as far as we could find in published literature), this section is decidedly not so.
At this point, the explosion of automated assessment systems/automated grading systems for programming education had already set in.
To describe all platforms would take a full dissertation in and of itself.
So from now on, we will pick and choose systems that brought new and interesting ideas that stood the test of time.[fn::
The ideas, not the platforms.
As far as we know none of the platforms described in this section are still in use.
]

ACSES, by\nbsp{}[cite/t:@nievergeltACSESAutomatedComputer1976], was envisioned as a full course for learning computer programming.
They even designed it as a full replacement for a course: it was the first system that integrated both instructional texts and exercises.
Students following this course would not need personal instruction.
In the modern day, this would probably be considered a MOOC.[fn::
Except that it obviously wasn't an online course; TCP/IP wouldn't be standardized until 1982.
]

Another good example of this generation of grading systems is the system by\nbsp{}[cite/t:@isaacson1989automating].
They describe the functioning of a UNIX shell script that automatically e-mails students if their code did not compile, or if they had incorrect outputs.
It also had a configurable output file size limit and time limit.
Student programs would be stopped if they exceeded these limits.
Like all assessment systems up to this point, they only focus on whether the output of the student's program is correct, and not on the code style.

[cite/t:@reekTRYSystemHow1989] takes a different approach.
He identifies several issues with gathering students' source files, and then compiling and executing them in the teacher's environment.
Students could write destructive code that destroys the teacher's files, or even write a clever program that alters their grades (and covers its tracks while doing so).
Note that this is not a new issue: as we discussed before, this was already mentioned as a possibility by\nbsp{}[cite/t:@hollingsworthAutomaticGradersProgramming1960].
This was, however, the first system that tried to solve this problem.
His TRY system therefore has avoiding that teachers need to run their students' programs themselves as an explicit goal.
Another goal was avoiding giving the inputs that the program was tested on to students.
These goals were mostly achieved using the UNIX =setuid= mechanism.
Note that students were using a true multi-user system, as in common use at the time.
Every attempt was also recorded in a log file in the teacher's personal directory.
Generality of programming language was achieved through intermediate build and test scripts that had to be provided by the teacher.

This is also the first study we could find that pays explicit attention to how expected and generated output is compared.
In addition to the basic character-by-character comparison, it is also supported to define the interface for a function that students have to call with their outputs.
The instructor can then link an implementation of this function in the build script.

Even later, automated assessment systems were built with graphical user interfaces.
A good example of this is ASSYST\nbsp{}[cite:@jacksonGradingStudentPrograms1997].
ASSYST also added evaluation on other metrics, such as runtime or cyclomatic complexity as suggested by\nbsp{}[cite/t:@hungAutomaticProgrammingAssessment1993].

*** Moving to the web
:PROPERTIES:
:CREATED:  [2024-02-06 Tue 17:29]
:END:

After Tim Berners-Lee invented the web in 1989\nbsp{}[cite:@berners-leeWorldWideWeb1992], automated assessment systems also started moving to the web.
Especially with the rise of Web 2.0\nbsp{}[cite:@oreillyWhatWebDesign2007] and its increased interactivity, this became more and more common.
Systems like the one by\nbsp{}[cite/t:@reekTRYSystemHow1989] also became impossible to use because of the rise of the personal computer.
Mainly because the typical multi-user system was used less and less, but also because the primary way people interacted with a computer was no longer through the command line, but through graphical user interfaces.

[cite/t:@higginsCourseMarkerCBASystem2003] developed CourseMarker, which is a more general assessment system (not exclusively developed for programming assessment).
This was initially not yet a web-based platform, but it did communicate over the network using Java's Remote Method Invocation mechanism.
The system it was designed to replace, Ceilidh, did have a basic web submission interface\nbsp{}[cite:@hughesCeilidhCollaborativeWriting1998].
Designing a web client was also mentioned as future work in the paper announcing CourseMarker.

Perhaps the most famous example of the first web-based platforms is Web-CAT\nbsp{}[cite:@shah2003web].
In addition to being one of the first web-based automated assessment platforms, it also asked the students to write their own tests.
The coverage that these tests achieved was part of the testing done by the platform.
Tests are written using standard unit testing frameworks\nbsp{}[cite:@edwardsExperiencesUsingTestdriven2007].
An example of Web-CAT's submission screen can be seen in Figure\nbsp{}[[fig:introductionwebcatsubmission]].

#+CAPTION: Example of Web-CAT's submission screen for students.
#+CAPTION: Image taken from\nbsp{}[cite/t:@edwardsWebCATWebbasedCenter2006].
#+NAME: fig:introductionwebcatsubmission
[[./images/introductionwebcatsubmission.png]]

This is also the time when we first start to see mentions of plagiarism and plagiarism detection in the context of automated assessment, presumably because the internet made plagiarizing a lot easier.
In one case at MIT over 30% of students were found to be plagiarizing\nbsp{}[cite:@wagner2000plagiarism].
[cite/t:@dalyPatternsPlagiarism2005] analysed plagiarizing behaviour by watermarking student submissions, where the watermark consisted of added whitespace at the end of lines.
If students carelessly copied another student's submission, they would also copy the whitespace.
[cite/t:@schleimerWinnowingLocalAlgorithms2003] also published MOSS around this time.

Another important platform[fn:: Especially in our context, since we used this platform for a long time.] is SPOJ\nbsp{}[cite:@kosowskiApplicationOnlineJudge2008].
SPOJ specifically notes the influence of online contest platforms (and in fact, is a platform that can be used to organize contests).
Online contest platforms usually differ from the automated assessment platforms for education in the way they handle feedback.
For online contests, the amount of feedback given to participants is often far less than the feedback given in education to students.
Although, depending on the educational vision of the teacher, this happens in education as well.

The SPOJ paper also details the security measures they took when executing untrusted code.
They use a patched Linux kernel's =rlimits=, the =chroot= mechanism, and traditional user isolation to prevent student code from malicious action.

Another interesting idea was contributed by\nbsp{}[cite/t:@brusilovskyIndividualizedExercisesSelfassessment2005] in QuizPACK.
They combined the idea of parametric exercises with automated assessment by executing source code.
In QuizPACK, teachers provide a parameterized piece of code, where the value of a specific variable is the answer that a student needs to give.
The piece of code is then evaluated, and the result is compared to the student's answer.
Note that in this platform, it is not the students themselves who are writing code.

*** Adding features
:PROPERTIES:
:CREATED:  [2024-02-06 Tue 15:31]
:END:

At this point in history, the idea of a web-based automated assessment system for programming education is no longer new.
But still, more and more new platforms are being written.
For a possible explanation, see Figure\nbsp{}[[fig:introductionxkcdstandards]].

#+CAPTION: Comic on the proliferation of standards.
#+CAPTION: Created by Randall Munroe, released under the CC BY-NC 2.5 licence.
#+NAME: fig:introductionxkcdstandards
[[./images/introductionxkcdstandards.png]]


All of these platforms support automated assessment of code submitted by students, but try to differentiate themselves through the features they offer.
The FPGE platform by\nbsp{}[cite/t:@paivaManagingGamifiedProgramming2022] offers gamification, iWeb-TD\nbsp{}[cite:@fonsecaWebbasedPlatformMethodology2023] integrates a full-fledged editor, PLearn\nbsp{}[cite:@vasyliukDesignImplementationUkrainianLanguage2023] recommends extra exercises to its users, JavAssess\nbsp{}[cite:@insaAutomaticAssessmentJava2018] tries to automate grading, and GradeIT\nbsp{}[cite:@pariharAutomaticGradingFeedback2017] features automatic hint generation.

** Learning analytics and educational data mining
:PROPERTIES:
:CREATED:  [2024-02-01 Thu 10:47]
:CUSTOM_ID: sec:introlaedm
:END:

Learning analytics and educational data mining stand at the intersection of computer science, data analytics, and the social sciences, and focus on understanding and improving learning.
They are made possible by the increased availability of data about students who are learning, due to the increasing move of education to digital platforms\nbsp{}[cite:@romeroDataMiningCourse2008].
They can also serve different actors in the educational landscape: they can help learners directly, help teachers to evaluate their own teaching, allow developers of educational platforms to know what to focus on, allow educational institutions to guide their decisions, and even allow governments to take on data-driven policies\nbsp{}[cite:@fergusonLearningAnalyticsDrivers2012].
Learning analytics and educational data mining are overlapping fields, but in general, learning analytics is seen as focusing on the educational challenge, while educational data mining is more focused on the technical challenge\nbsp{}[cite:@fergusonLearningAnalyticsDrivers2012].
The analytics focusing on governments or educational institutions is called academic analytics.

[cite/t:@chattiReferenceModelLearning2012] defined a reference model for learning analytics and educational data mining based on four dimensions:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{ }}, itemjoin*={{ }}]
- What data is gathered and used?
- Who is targeted by the analysis?
- Why is the data analysed?
- How is the data analysed?
This gives an idea to researchers what to focus on when conceptualizing, executing, and publishing their research.

An example of educational data mining research is\nbsp{}[cite/t:@daudPredictingStudentPerformance2017], where the students' background (including family income, family expenditures, gender, martial status,\nbsp{}...) is used to predict the student's learning outcome at the end of the semester.
Evaluating this study using the reference model by\nbsp{}[cite/t:@chattiReferenceModelLearning2012], we can see that the data used is very personal and hard to collect.
As mentioned in the study, the primary target audience of the study are policymakers.
The data is analysed to evaluate the influence of a student's background on their performance, and this is done by using a number of machine learning techniques (which are compared to one another).

Another example of the research in this field is a study by\nbsp{}[cite/t:@akcapinarUsingLearningAnalytics2019].
They focus on the concept of an early warning system, where student performance can be predicted early and appropriate action could be undertaken.
Their study uses data from a blended learning environment, where students can see the lesson's resources, participate in discussions, and write down their own thoughts about the lessons.
Here, the primary target audience is the student.
Although the related actions are not yet in scope of the study, the primary goal is to develop such an early warning system.
Again, a number of machine learning techniques are compared, to determine which one gives the best results.

** Structure of this dissertation
:PROPERTIES:
:CREATED:  [2024-02-01 Thu 10:18]
:CUSTOM_ID: sec:introstructure
:END:

Chapters\nbsp{}[[#chap:what]],\nbsp{}[[#chap:use]],\nbsp{}and\nbsp{}[[#chap:technical]] focus on Dodona[fn:: https://dodona.be/] itself.
In Chapter\nbsp{}[[#chap:what]] we will give an overview of the user-facing features of Dodona, from user management to how feedback is represented.
Chapter\nbsp{}[[#chap:use]] then focuses on how Dodona is used in practice, by presenting some facts and figures of its use, students' opinions of the platform, and an extensive case study on how Dodona's features are used to optimize teaching.
Chapter\nbsp{}[[#chap:technical]] focuses on the technical aspect of developing Dodona and its related ecosystem of software.
This includes a discussion of the technical challenges related to developing a platform like Dodona, and how the Dodona team adheres to modern standards of software development.

Chapter\nbsp{}[[#chap:passfail]] discusses an educational data mining study where we tried to predict whether students would pass or fail a course at the end of the semester based solely on their submission history in Dodona.
It also briefly details a study we collaborated on with researchers from Jyväskylä University in Finland, where we replicated our study in their educational context, with data from their educational platform.

In Chapter\nbsp{}[[#chap:feedback]], we first give an overview of how Dodona changed manual assessment in our own educational context.
We then finish the chapter with some recent work on a machine learning method we developed to predict what feedback teachers will give when manually assessing student submissions.

Finally, Chapter\nbsp{}[[#chap:discussion]] concludes the dissertation with some discussion on the current status of Dodona, the research related to it, and Dodona's challenges for the future.

* What is Dodona?
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:47]
:CUSTOM_ID: chap:what
:END:

Dodona is an online learning environment that recognizes the importance of active learning and just-in-time feedback in courses involving programming assignments.
Dodona was started because our own educational needs outgrew SPOJ\nbsp{}[cite:@kosowskiApplicationOnlineJudge2008], the platform we were using before.
In the following years, Dodona grew.
It is now used in most higher education institutions in Flanders, and a large number of secondary education institutions as well.
In this chapter, we will give an overview of Dodona's most important features.

This chapter is partially based on\nbsp{}[cite/t:@vanpetegemDodonaLearnCode2023], published in SoftwareX.

** User management
:PROPERTIES:
:CREATED: [2023-10-24 Tue 09:44]
:CUSTOM_ID: subsec:whatuser
:END:

Instead of providing its own authentication and authorization, Dodona delegates authentication to external identity providers (e.g.\nbsp{}educational and research institutions) through SAML\nbsp{}[cite:@farrellAssertionsProtocolOASIS2002], OAuth\nbsp{}[cite:@leibaOAuthWebAuthorization2012; @hardtOAuthAuthorizationFramework2012] and OpenID Connect\nbsp{}[cite:@sakimuraOpenidConnectCore2014].
This support for *decentralized authentication* allows users to benefit from single sign-on when using their institutional account across multiple platforms and teachers to trust their students' identities when taking high-stakes tests and exams in Dodona.

Dodona automatically creates user accounts upon successful authentication and uses the association with external identity providers to assign an institution to users.
These institutions can have multiple sign-in methods.
If a user uses more than one of those methods, these logins are linked to the same user.
Institutions are also categorized internally in secundary education, higher education, and other (e.g. the Flemish government).

By default, newly created users are assigned a student role.
Teachers and instructors who wish to create content (courses, learning activities and judges), must first request teacher rights using a streamlined form[fn:: https://dodona.be/rights_requests/new/].
The sign in page can be seen in Figure\nbsp{}[[fig:whatsignin]].
After logging in, a user sees an overview of the courses they are registered with.

#+CAPTION: Sign in page showing the different options for users to sign in.
#+NAME: fig:whatsignin
[[./images/whatsignin.png]]

** Course management
:PROPERTIES:
:CREATED: [2023-10-24 Tue 09:31]
:CUSTOM_ID: subsec:whatclassroom
:END:

In Dodona, a *course* is where teachers and instructors effectively manage a learning environment by instructing, monitoring and evaluating their students and interacting with them, either individually or as a group.
A Dodona user who created a course becomes its first administrator and can promote other registered users as *course administrators*.
In what follows, we will also use the generic term teacher as a synonym for course administrators if this Dodona-specific interpretation is clear from the context, but keep in mind that courses may have multiple administrators.

The course itself is laid out as a *learning path* that consists of course units called *series*, each containing a sequence of *learning activities* (Figure\nbsp{}[[fig:whatcourse]]).
Among the learning activities we differentiate between *reading activities* that can be marked as read and *programming assignments* with support for automated assessment of submitted solutions.
Learning paths are composed as a recommended sequence of learning activities to build knowledge progressively, allowing students to monitor their own progress at any point in time.
Courses can either be created from scratch or from copying an existing course and making additions, deletions and rearrangements to its learning path.

#+CAPTION: Main course page (administrator view) showing some series with deadlines, reading activities and programming assignments in its learning path.
#+CAPTION: At any point in time, students can see their own progress through the learning path of the course.
#+CAPTION: Teachers have some additional icons in the navigation bar (top) that lead to an overview of all students and their progress, an overview of all submissions for programming assignments, general learning analytics about the course, course management and a dashboard with questions from students in various stages from being answered (Figure\nbsp{}[[fig:whatquestions]]).
#+CAPTION: The red dot on the latter icon notifies that some student questions are still pending.
#+NAME: fig:whatcourse
[[./images/whatcourse.png]]

Students can *self-register* to courses in order to avoid unnecessary user management.
A course can either be announced in the public overview of Dodona for everyone to see, or be limited in visibility to students from a certain educational institution.
Alternatively, students can be invited to a hidden course by sharing a secret link.
Independent of course visibility, registration for a course can either be open to everyone, restricted to users from the institution the course is associated with, or new registrations can be disabled altogether.
Registrations are either approved automatically or require explicit approval by a teacher.
Registered users can be tagged with one or more labels to create subgroups that may play a role in learning analytics and reporting.

Students and teachers more or less see the same course page, except for some management features and learning analytics that are reserved for teachers.
Teachers can make content in the learning path temporarily inaccessible and/or invisible to students.
Content is typically made inaccessible when it is still in preparation or if it will be used for evaluating students during a specific period.
A token link can be used to grant access to invisible content, e.g.\nbsp{}when taking a test or exam from a subgroup of students.

Students can only mark reading activities as read once, but there is no restriction on the number of solutions they can submit for programming assignments.
Submitted solutions are automatically assessed and students receive immediate feedback as soon as the assessment has completed, usually within a few seconds.
Dodona stores all submissions, along with submission metadata and generated feedback, such that the submission and feedback history can be reclaimed at all times.
On top of automated assessment, student submissions may be further assessed and graded manually by a teacher.

Series can have a *deadline*.
Passed deadlines do not prevent students from marking reading activities or submitting solutions for programming assignments in their series.
However, learning analytics, reports and exports usually only take into account submissions before the deadline.
Because of the importance of deadlines and to avoid discussions with students about missed deadlines, series deadlines are not only announced on the course page.
The student's home page highlights upcoming deadlines for individual courses and across all courses.
While working on a programming assignment, students will also see a clear warning starting from ten minutes before a deadline.
Courses also provide an iCalendar link\nbsp{}[cite:@stenersonInternetCalendaringScheduling1998] that students can use to publish course deadlines in their personal calendar application.

Because Dodona logs all student submissions and their metadata, including feedback and grades from automated and manual assessment, we use that data to integrate reports and learning analytics in the course page\nbsp{}[cite:@fergusonLearningAnalyticsDrivers2012].
This includes heatmaps (Figure\nbsp{}[[fig:whatcourseheatmap]]) and punch cards (Figure\nbsp{}[[fig:whatcoursepunchcard]]) of user activity, graphs showing class progress (Figure\nbsp{}[[fig:whatcourseprogress]]), ...

#+CAPTION: Heatmap showing on which days in the semester students are more active or less active.
#+NAME: fig:whatcourseheatmap
[[./images/whatcourseheatmap.png]]

#+CAPTION: Punchcard showing when during the week students are working on their exercises.
#+NAME: fig:whatcoursepunchcard
[[./images/whatcoursepunchcard.png]]

#+CAPTION: Graph showing the percentage of students that correctly solved the exercises in a certain series over time.
#+NAME: fig:whatcourseprogress
[[./images/whatcourseprogress.png]]

We also provide export wizards that enable the extraction of raw and aggregated data in CSV format for downstream processing and educational data mining\nbsp{}[cite:@romeroEducationalDataMining2010; @bakerStateEducationalData2009].
This allows teachers to better understand student behaviour, progress and knowledge, and might give deeper insight into the underlying factors that contribute to student actions\nbsp{}[cite:@ihantolaReviewRecentSystems2010].
Understanding, knowledge and insights that can be used to make informed decisions about courses and their pedagogy, increase student engagement, and identify at-risk students\nbsp{}(see\nbsp{}Chapter\nbsp{}[[#chap:passfail]]).

** Exercises
:PROPERTIES:
:CREATED:  [2024-02-20 Tue 14:32]
:END:

There are two types of assignments in Dodona: reading activities and programming exercises.
While reading activities only consist of descriptions, programming exercises need an additional *assessment configuration* that sets a programming language and a judge (for more information on judges, see Section\nbsp{}[[#subsec:whatjudges]]).
The configuration may also set a Docker image, a time limit, a memory limit and grant Internet access to the container that is instantiated from the image, but these settings have proper default values.
The configuration might also provide additional *assessment resources*: files made accessible to the judge during assessment.
The specification of how these resources must be structured and how they are used during assessment is completely up to the judge developers.
Finally, the configuration might also contain *boilerplate code*: a skeleton students can use to start the implementation that is provided in the code editor along with the description.
Directories that contain a learning activity also have their own internal directory structure that includes a *description* in HTML or Markdown.
Descriptions may reference data files and multimedia content included in the repository, and such content can be shared across all learning activities in the repository.
Embedded images are automatically encapsulated in a responsive lightbox to improve readability.
Mathematical formulas in descriptions are supported through MathJax\nbsp{}[cite:@cervoneMathJaxPlatformMathematics2012].

Where automatic assessment and feedback generation is outsourced to the judge linked to an assignment, Dodona itself takes up the responsibility for rendering the feedback.
This frees judge developers from putting effort in feedback rendering and gives a coherent look-and-feel even for students that solve programming assignments assessed by different judges.
Because the way feedback is presented is very important\nbsp{}[cite:@maniBetterFeedbackEducational2014], we took great care in designing how feedback is displayed to make its interpretation as easy as possible (Figure\nbsp{}[[fig:whatfeedback]]).
Differences between generated and expected output are automatically highlighted for each failed test\nbsp{}[cite:@myersAnONDDifference1986], and users can swap between displaying the output lines side-by-side or interleaved to make differences more comparable.
We even provide specific support for highlighting differences between tabular data such as CSV files, database tables and data frames.
Users have the option to dynamically hide contexts whose test cases all succeeded, allowing them to immediately pinpoint reported mistakes in feedback that contains lots of succeeded test cases.
To ease debugging the source code of submissions for Python assignments, the Python Tutor\nbsp{}[cite:@guoOnlinePythonTutor2013] can be launched directly from any context with a combination of the submitted source code and the test code from the context.
Students typically report this as one of the most useful features of Dodona.

#+CAPTION: Dodona rendering of feedback generated for a submission of the Python programming assignment "Curling".
#+CAPTION: The feedback is split across three tabs: ~isinside~, ~isvalid~ and ~score~.
#+CAPTION: 48 tests under the ~score~ tab failed as can be seen from the badge in the tab header.
#+CAPTION: The "Code" tab displays the source code of the submission with annotations added during automatic and/or manual assessment (Figure\nbsp{}[[fig:whatannotations]]).
#+CAPTION: The differences between the generated and expected return values were automatically highlighted and the judge used HTML snippets to add a graphical representation (SVG) of the problem for the failed test cases.
#+CAPTION: In addition to highlighting differences between the generated and expected return values of the first (failed) test case, the judge also added a text snippet that points the user to a type error.
#+NAME: fig:whatfeedback
[[./images/whatfeedback.png]]

** Judges
:PROPERTIES:
:CREATED:  [2024-02-20 Tue 15:28]
:CUSTOM_ID: subsec:whatjudges
:END:

The range of approaches, techniques and tools for software testing that may underpin assessing the quality of software under test is incredibly diverse.
Static testing directly analyses the syntax, structure and data flow of source code, whereas dynamic testing involves running the code with a given set of test cases\nbsp{}[cite:@oberkampfVerificationValidationScientific2010; @grahamFoundationsSoftwareTesting2021].
Black-box testing uses test cases that examine functionality exposed to end-users without looking at the actual source code, whereas white-box testing hooks test cases onto the internal structure of the code to test specific paths within a single unit, between units during integration, or between subsystems\nbsp{}[cite:@nidhraBlackBoxWhite2012].
So, broadly speaking, there are three levels of white-box testing: unit testing, integration testing and system testing\nbsp{}[cite:@wiegersCreatingSoftwareEngineering1996; @dooleySoftwareDevelopmentProfessional2011].
Source code submitted by students can therefore be verified and validated against a multitude of criteria: functional completeness and correctness, architectural design, usability, performance and scalability in terms of speed, concurrency and memory footprint, security, readability (programming style), maintainability (test quality) and reliability\nbsp{}[cite:@staubitzPracticalProgrammingExercises2015].
This is also reflected by the fact that a diverse range of metrics for measuring software quality have come forward, such as cohesion/coupling\nbsp{}[cite:@yourdonStructuredDesignFundamentals1979; @stevensStructuredDesign1999], cyclomatic complexity\nbsp{}[cite:@mccabeComplexityMeasure1976] or test coverage\nbsp{}[cite:@millerSystematicMistakeAnalysis1963].

To cope with such a diversity in software testing alternatives, Dodona is centred around a generic infrastructure for *programming assignments that support automated assessment*.
Assessment of a student submission for an assignment comprises three loosely coupled components: containers, judges and assignment-specific assessment configurations.
Judges have a default Docker image that is used if the configuration of a programming assignment does not specify one explicitly.
Dodona builds the available images from Dockerfiles specified in a separate git repository.
More information on this underlying mechanism can be found in Chapter\nbsp{}[[#chap:technical]].
An overview of the existing judges and the corresponding number of exercises and submissions in Dodona can be found in Table\nbsp{}[[tab:whatoverviewjudges]].

#+CAPTION: Overview of the judges in Dodona, together with the corresponding number of exercises and submissions in Dodona.
#+CAPTION: The TESTed judge is a special case in that it supports multiple programming languages.
#+CAPTION: More information on it can be found in Section\nbsp{}[[#sec:techtested]].
#+CAPTION: The number of exercises and submissions for the JavaScript judge is undercounted: most of its exercises were converted to TESTed exercises, which also moved the submissions to those exercises to TESTed.
#+NAME: tab:whatoverviewjudges
| Judge      |   # exercises |              # submissions |
|------------+---------------+----------------------------|
| <l>        |           <r> |                        <r> |
| Bash       |           289 |            675\thinsp{}902 |
| C          |            77 |             31\thinsp{}822 |
| C#         |           256 |             44\thinsp{}294 |
| Compilers  |             3 |                         38 |
| HTML       |           187 |             24\thinsp{}947 |
| Haskell    |            76 |             76\thinsp{}556 |
| Java 8     |            93 |             90\thinsp{}084 |
| Java 21    |           450 |            730\thinsp{}383 |
| JavaScript |            36 |                         68 |
| Markdown   |            14 |                        354 |
| Prolog     |            54 |             37\thinsp{}609 |
| Python     | 8\thinsp{}481 | 13\thinsp{}798\thinsp{}051 |
| R          | 1\thinsp{}293 |            958\thinsp{}069 |
| SQL        |           298 |            114\thinsp{}725 |
| Scheme     |           277 |            125\thinsp{}138 |
| TESTed     | 1\thinsp{}139 |            333\thinsp{}507 |
| Turtle     |            17 |                        446 |

** Repositories
:PROPERTIES:
:CREATED:  [2024-02-20 Tue 15:20]
:END:

Where courses are created and managed in Dodona itself, other content is managed in external git *repositories* (Figure\nbsp{}[[fig:whatrepositories]]).
In this distributed content management model, a repository either contains a single judge or a collection of learning activities.
Setting up a *webhook* for the repository guarantees that any changes pushed to its default branch are automatically and immediately synchronized with Dodona.
This even works without the need to make repositories public, as they may contain information that should not be disclosed such as programming assignments that are under construction, contain model solutions, or will be used during tests or exams.
Instead, a *Dodona service account* must be granted push/pull access to the repository.
Some settings of a learning activity can be modified through the web interface of Dodona, but any changes are always pushed back to the repository in which the learning activity is configured so that it always remains the master copy.

#+CAPTION: Distributed content management model that allows to seamlessly integrate custom learning activities (reading activities and programming assignments with support for automated assessment) and judges (frameworks for automated assessment) into Dodona.
#+CAPTION: Content creators manage their content in external git repositories, keep ownership over their content, control who can co-create, and set up webhooks to automatically synchronize any changes with the content as published on Dodona.
#+NAME: fig:whatrepositories
[[./images/whatrepositories.png]]

Due to the distributed nature of content management, creators also keep ownership over their content and control who may co-create.
After all, access to a repository is completely independent of access to its learning activities that are published in Dodona.
The latter is part of the configuration of learning activities, with the option to either share learning activities so that all teachers can include them in their courses or to restrict inclusion of learning activities to courses that are explicitly granted access.
Dodona automatically stores metadata about all learning activities such as content type, natural language, programming language and repository to increase their findability in our large collection.
Learning activities may also be tagged with additional labels as part of their configuration.
Any repository containing learning activities must have a predefined directory structure[fn:: https://docs.dodona.be/en/references/exercise-directory-structure/].

** Internationalization and localization
:PROPERTIES:
:CREATED: [2023-10-24 Tue 10:55]
:CUSTOM_ID: subsec:whati18n
:END:
*Internationalization* (i18n) is a shared responsibility between Dodona, learning activities and judges.
All boilerplate text in the user interface that comes from Dodona itself is supported in English and Dutch, and users can select their preferred language.
Content creators can specify descriptions of learning activities in both languages, and Dodona will render a learning activity in the user's preferred language if available.
When users submit solutions for a programming assignment, their preferred language is passed as submission metadata to the judge.
It's then up to the judge to take this information into account while generating feedback.

Dodona always displays *localized deadlines* based on a time zone setting in the user profile, and users are warned when the current time zone detected by their browser differs from the one in their profile.

** Questions, answers and code reviews
:PROPERTIES:
:CREATED: [2023-10-24 Tue 10:56]
:CUSTOM_ID: subsec:whatqa
:END:

A downside of using discussion forums in programming courses is that students can ask questions about programming assignments that are either disconnected from their current implementation or contain code snippets that may give away (part of) the solution to other students\nbsp{}[cite:@nandiEvaluatingQualityInteraction2012].
Dodona therefore allows students to address teachers with questions they directly attach to their submitted source code.
We support both general questions and questions linked to specific lines of their submission (Figure\nbsp{}[[fig:whatquestion]]).
Questions are written in Markdown (e.g., to include markup, tables, syntax highlighted code snippets or multimedia), with support for MathJax (e.g., to include mathematical formulas).

#+CAPTION: A student (Matilda) previously asked a question that has already been answered by her teacher (Miss Honey).
#+CAPTION: Based on this response, the student is now asking a follow-up question that can be formatted using Markdown.
#+NAME: fig:whatquestion
[[./images/whatquestion.png]]

Teachers are notified whenever there are pending questions (Figure\nbsp{}[[fig:whatcourse]]).
They can process these questions from a dedicated dashboard with live updates (Figure\nbsp{}[[fig:whatquestions]]).
The dashboard immediately guides them from an incoming question to the location in the source code of the submission it relates to, where they can answer the question similar to how students ask questions.
To avoid questions being inadvertently handled simultaneously by multiple teachers, they have a three-state lifecycle: pending, in progress and answered.
In addition to teachers changing question states while answering them, students can also mark their own questions as being answered.
The latter might reflect the rubber duck debugging\nbsp{}[cite:@huntPragmaticProgrammer1999] effect that is triggered when students are forced to explain a problem to someone else while asking questions in Dodona.
Teachers can (temporarily) disable the option for students to ask questions in a course, e.g.\nbsp{}when a course is over or during hands-on sessions or exams when students are expected to ask questions face-to-face rather than online.

#+CAPTION: Live updated dashboard showing all incoming questions in a course while asking questions is enabled.
#+CAPTION: Questions are grouped into three categories: unanswered, in progress and answered.
#+NAME: fig:whatquestions
[[./images/whatquestions.png]]

Manual source code annotations from students (questions) and teachers (answers) are rendered in the same way as source code annotations resulting from automated assessment.
They are mixed in the source code displayed in the "Code" tab, showing their complementary nature.
It is not required that students take the initiative for the conversation.
Teachers can also start adding source code annotations while reviewing a submission.
Such *code reviews* will be used as a building block for manual assessment.

** Manual assessment
:PROPERTIES:
:CREATED: [2023-10-24 Tue 11:01]
:CUSTOM_ID: subsec:whateval
:END:

Teachers can create an *evaluation* for a series to manually assess student submissions for its programming assignments after a specific period, typically following the deadline of some homework, an intermediate test or a final exam.
An example of an evaluation overview can be seen on Figure\nbsp{}[[fig:whatevaluationoverview]].
The evaluation embodies all programming assignments in the series and a group of students that submitted solutions for these assignments.
Because a student may have submitted multiple solutions for the same assignment, the last submission before a given deadline is automatically selected for each student and each assignment in the evaluation.
This automatic selection can be manually overruled afterwards.
The evaluation deadline defaults to the deadline set for the associated series, if any, but an alternative deadline can be selected as well.

#+CAPTION: Pseudonymized overview of an evaluation in Dodona.
#+CAPTION: For each student, both the correctness of their submission and whether it has been graded yet is shown.
#+NAME: fig:whatevaluationoverview
[[./images/whatevaluationoverview.png]]

Evaluations support *two-way navigation* through all selected submissions: per assignment and per student.
For evaluations with multiple assignments, it is generally recommended to assess per assignment and not per student, as students can build a reputation throughout an assessment\nbsp{}[cite:@malouffBiasGradingMetaanalysis2016].
As a result, they might be rated more favourably with a moderate solution if they had excellent solutions for assignments that were assessed previously, and vice versa\nbsp{}[cite:@malouffRiskHaloBias2013].
Assessment per assignment breaks this reputation as it interferes less with the quality of previously assessed assignments from the same student.
Possible bias from the same sequence effect is reduced during assessment per assignment as students are visited in random order for each assignment in the evaluation.
In addition, *anonymous mode* can be activated as a measure to eliminate the actual or perceived halo effect conveyed through seeing a student's name during assessment\nbsp{}[cite:@lebudaTellMeYour2013].
While anonymous mode is active, all students are automatically pseudonymized.
Anonymous mode is not restricted to the context of assessment and can be used across Dodona, for example while giving in-class demos.

When reviewing a selected submission from a student, assessors have direct access to the feedback that was previously generated during automated assessment: source code annotations in the "Code" tab and other structured and unstructured feedback in the remaining tabs.
Moreover, next to the feedback that was made available to the student, the specification of the assignment may also add feedback generated by the judge that is only visible to the assessor.
Assessors might then complement the assessment made by the judge by adding *source code annotations* as formative feedback and by *grading* the evaluative criteria in a scoring rubric as summative feedback (Figure\nbsp{}[[fig:whatannotations]]).
Previous annotations can be reused to speed up the code review process, because remarks or suggestions tend to recur frequently when reviewing submissions for the same assignment.
Grading requires setting up a specific *scoring rubric* for each assignment in the evaluation, as a guidance for evaluating the quality of submissions\nbsp{}[cite:@dawsonAssessmentRubricsClearer2017; @pophamWhatWrongWhat1997].
The evaluation tracks which submissions have been manually assessed, so that analytics about the assessment progress can be displayed and to allow multiple assessors working simultaneously on the same evaluation, for example one (part of a) programming assignment each.

#+CAPTION: Manual assessment of a submission: a teacher (Miss Honey) is giving feedback on the source code by adding inline annotations and is grading the submission by filling up the scoring rubric that was set up for the programming assignment "The Feynman ciphers".
#+NAME: fig:whatannotations
[[./images/whatannotations.png]]

* Dodona in educational practice
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:48]
:CUSTOM_ID: chap:use
:END:

This chapter discusses the use of Dodona.
We start by mentioning some facts and figures, and discussing a user study we performed.
We then explain how Dodona can be used on the basis of a case study.
The chapter is partially based on\nbsp{}[cite/t:@vanpetegemDodonaLearnCode2023], published in SoftwareX.

** Facts and figures
:PROPERTIES:
:CREATED:  [2024-01-22 Mon 18:15]
:CUSTOM_ID: sec:usefacts
:END:

Dodona's design decisions have allowed it to spread to more than {{{num_schools}}} schools, colleges and universities, mainly in Flanders (Belgium) and the Netherlands.
The renewed interest in embedding computational thinking in formal education has undoubtedly been an important stimulus for such a wide uptake\nbsp{}[cite:@wingComputationalThinking2006].
All other educational institutions use the instance of Dodona hosted at Ghent University, which is free to use for educational purposes.

Dodona currently hosts a collection of {{{num_exercises}}} learning activities that are freely available to all teachers, allowing them to create their own learning paths tailored to their teaching practice.
In total, {{{num_users}}} students have submitted more than {{{num_submissions}}} solutions to Dodona in the seven years that it has been running (Figures\nbsp{}[[fig:useadoption1]]\nbsp{}&\nbsp{}[[fig:useadoption2]]).

#+CAPTION: Overview of the number of submitted solutions by academic year.
#+NAME: fig:useadoption1
[[./images/useadoption1.png]]

#+CAPTION: Overview of the number of active users by academic year.
#+CAPTION: Users were active when they submitted at least one solution for a programming assignment during the academic year.
#+NAME: fig:useadoption2
[[./images/useadoption2.png]]

** User study
:PROPERTIES:
:CREATED:  [2024-01-22 Mon 18:16]
:CUSTOM_ID: sec:usestudy
:END:

A qualitative user experience study of Dodona was performed in 2018.
271 tertiary education students responded to a questionnaire that contained the following three questions:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{ }}, itemjoin*={{ }}]
- What are the things you value while working with Dodona?
- What are the things that bother you while working with Dodona?
- What are your suggestions for improvements to Dodona?
Students praised its user-friendliness, beautiful interface, immediate feedback with visualized differences between expected and generated output, integration of the Python Tutor, linting feedback and large number of test cases.
Negative points were related to differences between the students' local execution environments and the environment in which Dodona runs the tests, and the strictness with which the tests are evaluated.
Other negative feedback was mostly related to individual courses the students were taking instead of the platform itself.

** Case study
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:48]
:CUSTOM_ID: subsec:usecasestudy
:END:

Since the academic year 2011--2012 we have organized an introductory Python course at Ghent University (Belgium) with a strong focus on active and online learning.
Initially the course was offered twice a year in the first and second term, but from academic year 2014--2015 onwards it was only offered in the first term.
The course is taken by a mix of undergraduate, graduate, and postgraduate students enrolled in various study programmes (mainly formal and natural sciences, but not computer science), with 442 students enrolled for the 2021--2022 edition[fn:: https://dodona.be/courses/773/].

*** Course structure
:PROPERTIES:
:CREATED: [2023-10-24 Tue 11:47]
:CUSTOM_ID: subsubsec:usecourse
:END:

Each course edition has a fixed structure, with 13 weeks of educational activities subdivided in two successive instructional units that each cover five topics of the Python programming language -- one topic per week -- followed by a graded test about all topics covered in the unit (Figure\nbsp{}[[fig:usefwecoursestructure]]).
The final exam at the end of the term evaluates all topics covered in the entire course.
Students who fail the course during the first exam in January can take a resit exam in August/September that gives them a second chance to pass the exam.

#+CAPTION: *Top*: Structure of the Python course that runs each academic year across a 13-week term (September--December).
#+CAPTION: Programming assignments from the same Dodona series are stacked vertically.
#+CAPTION: Students submit solutions for ten series with six mandatory assignments, two tests with two assignments and an exam with three assignments.
#+CAPTION: There is also a resit exam with three assignments in August/September if they failed the first exam in January.
#+CAPTION: *Bottom*: Heatmap from Dodona learning analytics page showing distribution per day of all 331\thinsp{}734 solutions submitted during the 2021--2022 edition of the course (442 students).
#+CAPTION: The darker the colour, the more solutions were submitted that day.
#+CAPTION: A light grey square means no solutions were submitted that day.
#+CAPTION: Weekly lab sessions for different groups on Monday afternoon, Friday morning and Friday afternoon, where we can see darker squares.
#+CAPTION: Weekly deadlines for mandatory assignments on Tuesdays at 22:00.
#+CAPTION: Three exam sessions for different groups in January.
#+CAPTION: Low activity in exam periods, except for days when an exam was taken.
#+CAPTION: The course is not taught in the second term, so this low-activity period was collapsed.
#+CAPTION: Two more exam sessions for different groups in August/September, granting an extra chance to students who failed on their exam in January.
#+NAME: fig:usefwecoursestructure
[[./images/usefwecoursestructure.png]]

Each week in which a new programming topic is covered, students must try to solve six programming assignments on that topic before a deadline one week later.
That results in 60 mandatory assignments across the semester.
Following the flipped classroom strategy\nbsp{}[cite:@bishopFlippedClassroomSurvey2013; @akcayirFlippedClassroomReview2018], students prepare themselves to achieve this goal by reading the textbook chapters covering the topic.
Lectures are interactive programming sessions that aim at bridging the initial gap between theory and practice, advancing concepts, and engaging in collaborative learning\nbsp{}[cite:@tuckerFlippedClassroom2012].
Along the same lines, the first assignment for each topic is an ISBN-themed programming challenge whose model solution is shared with the students, together with an instructional video that works step-by-step towards the model solution.
As soon as students feel they have enough understanding of the topic, they can start working on the five remaining mandatory assignments.
Students can work on their programming assignments during weekly computer labs, where they can collaborate in small groups and ask help from teaching assistants.
They can also work on their assignments and submit solutions outside lab sessions.
In addition to the mandatory assignments, students can further elaborate on their programming skills by tackling additional programming exercises they select from a pool of over 900 exercises linked to the ten programming topics.
Submissions for these additional exercises are not taken into account in the final grade.

*** Assessment, feedback and grading
:PROPERTIES:
:CREATED: [2023-10-24 Tue 11:47]
:CUSTOM_ID: subsubsec:useassessment
:END:

We use the online learning environment Dodona to promote active learning through problem-solving\nbsp{}[cite:@princeDoesActiveLearning2004].
Each course edition has its own dedicated course in Dodona, with a learning path containing all mandatory, test and exam assignments, grouped into series with corresponding deadlines.
Mandatory assignments for the first unit are published at the start of the semester, and those for the second unit after the test of the first unit.
For each test and exam we organize multiple sessions for different groups of students.
Assignments for test and exam sessions are provided in a hidden series that is only accessible for students participating in the session using a shared token link.
The test and exam assignments are published afterwards for all students, when grades are announced.
Students can see class progress when working on their mandatory assignments to nudge them to avoid procrastination.
Only teachers can see class progress for test and exam series so as not to accidentally stress out students.
For the same reason, we intentionally organize tests and exams following exactly the same procedure, so that students can take high-stake exams in a familiar context and adjust their approach based on previous experiences.
The only difference is that test assignments are not as hard as exam assignments, as students are still in the midst of learning programming skills when tests are taken.

Students are stimulated to use an integrated development environment (IDE) to work on their programming assignments.
IDEs bundle a battery of programming tools to support today's generation of software developers in writing, building, running, testing and debugging software.
Working with such tools can be a true blessing for both seasoned and novice programmers, but there is no silver bullet\nbsp{}[cite:@brooksNoSilverBullet1987].
Learning to code remains inherently hard\nbsp{}[cite:@kelleherAlice2ProgrammingSyntax2002] and consists of challenges that are different to reading and learning natural languages\nbsp{}[cite:@fincherWhatAreWe1999].
As an additional aid, students can continuously submit (intermediate) solutions for their programming assignments and immediately receive automatically generated feedback upon each submission, even during tests and exams.
Guided by that feedback, they can track potential errors in their code, remedy them and submit updated solutions.
There is no restriction on the number of solutions that can be submitted per assignment.
All submitted solutions are stored, but for each assignment only the last submission before the deadline is taken into account to grade students.
This allows students to update their solutions after the deadline (i.e.\nbsp{}after model solutions are published) without impacting their grades, as a way to further practice their programming skills.
One effect of active learning, triggered by mandatory assignments with weekly deadlines and intermediate tests, is that most learning happens during the term (Figure\nbsp{}[[fig:usefwecoursestructure]]).
In contrast to other courses, students do not spend a lot of time practising their coding skills for this course in the days before an exam.
We want to explicitly encourage this behaviour, because we strongly believe that one cannot learn to code in a few days' time\nbsp{}[cite:@peternorvigTeachYourselfProgramming2001].

For the assessment of tests and exams, we follow the line of thought that human expert feedback through source code annotations is a valuable complement to feedback coming from automated assessment, and that human interpretation is an absolute necessity when it comes to grading\nbsp{}[cite:@staubitzPracticalProgrammingExercises2015; @jacksonGradingStudentPrograms1997; @ala-mutkaSurveyAutomatedAssessment2005].
We shifted from paper-based to digital code reviews and grading when support for manual assessment was released in version 3.7 of Dodona (summer 2020).
Although online reviewing positively impacted our productivity, the biggest gain did not come from an immediate speed-up in the process of generating feedback and grades compared to the paper-based approach.
While time-on-task remained about the same, our online source code reviews were much more elaborate than what we produced before on printed copies of student submissions.
This was triggered by improved reusability of digital annotations and the foresight of streamlined feedback delivery.
Where delivering custom feedback only requires a single click after the assessment of an evaluation has been completed in Dodona, it took us much more effort before to distribute our paper-based feedback.
Students were direct beneficiaries from more and richer feedback, as observed from the fact that 75% of our students looked at their personalized feedback within 24 hours after it had been released, before we even published grades in Dodona.
What did not change is the fact that we complement personalized feedback with collective feedback sessions in which we discuss model solutions for test and exam assignments, and the low numbers of questions we received from students on their personalized feedback.
As a future development, we hope to reduce the time spent on manual assessment through improved computer-assisted reuse of digital source code annotations in Dodona.

We accept to primarily rely on automated assessment as a first step in providing formative feedback while students work on their mandatory assignments.
After all, a back-of-the-envelope calculation tells us it would take us 72 full-time equivalents (FTE) to generate equivalent amounts of manual feedback for mandatory assignments compared to what we do for tests and exams.
In addition to volume, automated assessment also yields the responsiveness needed to establish an interactive feedback loop throughout the iterative software development process while it still matters to students and in time for them to pay attention to further learning or receive further assistance\nbsp{}[cite:@gibbsConditionsWhichAssessment2005].
Automated assessment thus allows us to motivate students working through enough programming assignments and to stimulate their self-monitoring and self-regulated learning\nbsp{}[cite:@schunkSelfregulationLearningPerformance1994; @pintrichUnderstandingSelfregulatedLearning1995].
It results in triggering additional questions from students that we manage to respond to with one-to-one personalized human tutoring, either synchronously during hands-on sessions or asynchronously through Dodona's Q&A module.
We observe that individual students seem to have a strong bias towards either asking for face-to-face help during hands-on sessions or asking questions online.
This could be influenced by the time when they mainly work on their assignments, by their way of collaboration on assignments, or by reservations because of perceived threats to self-esteem or social embarrassment\nbsp{}[cite:@newmanStudentsPerceptionsTeacher1993; @karabenickRelationshipAcademicHelp1991].

In computing a final score for the course, we try to find an appropriate balance between stimulating students to find solutions for programming assignments themselves and collaborating with and learning from peers, instructors and teachers while working on assignments.
The final score is computed as the sum of a score obtained for the exam (80%) and a score for each unit that combines the student's performance on the mandatory and test assignments (10% per unit).
We use Dodona's grading module to determine scores for tests and exams based on correctness, programming style, choice made between the use of different programming techniques and the overall quality of the implementation.
The score for a unit is calculated as the score \(s\) for the two test assignments multiplied by the fraction \(f\) of mandatory assignments the student has solved correctly.
A solution for a mandatory assignment is considered correct if it passes all unit tests.
Evaluating mandatory assignments therefore doesn't require any human intervention, except for writing unit tests when designing the assignments, and is performed entirely by our Python judge.
In our experience, most students traditionally perform much better on mandatory assignments compared to test and exam assignments\nbsp{}[cite:@glassFewerStudentsAre2022], given the possibilities for collaboration on mandatory assignments.

*** Open and collaborative learning environment
:PROPERTIES:
:CREATED: [2023-10-24 Tue 11:59]
:CUSTOM_ID: subsubsec:useopen
:END:

We strongly believe that effective collaboration among small groups of students is beneficial for learning\nbsp{}[cite:@princeDoesActiveLearning2004], and encourage students to collaborate and ask questions to tutors and other students during and outside lab sessions.
We also demonstrate how they can embrace collaborative coding and pair programming services provided by modern integrated development environments\nbsp{}[cite:@williamsSupportPairProgramming2002; @hanksPairProgrammingEducation2011].
But we recommend them to collaborate in groups of no more than three students, and to exchange and discuss ideas and strategies for solving assignments rather than sharing literal code with each other.
After all, our main reason for working with mandatory assignments is to give students sufficient opportunity to learn topic-oriented programming skills by applying them in practice and shared solutions spoil the learning experience.
The factor \(f\) in the score for a unit encourages students to keep fine-tuning their solutions for programming assignments until all test cases succeed before the deadline passes.
But maximizing that factor without proper learning of programming skills will likely yield a low test score \(s\) and thus an overall low score for the unit, even if many mandatory exercises were solved correctly.

Fostering an open collaboration environment to work on mandatory assignments with strict deadlines and taking them into account for computing the final score is a potential promoter for plagiarism, but using it as a weight factor for the test score rather than as an independent score item should promote learning by avoiding that plagiarism is rewarded.
It takes some effort to properly explain this to students.
We initially used Moss\nbsp{}[cite:@schleimerWinnowingLocalAlgorithms2003] and now use Dolos\nbsp{}[cite:@maertensDolosLanguageagnosticPlagiarism2022] to monitor submitted solutions for mandatory assignments, both before and at the deadline.
The solution space for the first few mandatory assignments is too small for linking high similarity to plagiarism: submitted solutions only contain a few lines of code and the diversity of implementation strategies is small.
But at some point, as the solution space broadens, we start to see highly similar solutions that are reliable signals of code exchange among larger groups of students.
Strikingly this usually happens among students enrolled in the same study programme (Figure\nbsp{}[[fig:usefweplagiarism]]).
As soon as this happens -- typically in week 3 or 4 of the course -- plagiarism is discussed during the next lecture.
Usually this is a lecture about working with the string data type, so we can introduce plagiarism detection as a possible application of string processing.

#+CAPTION: Dolos plagiarism graphs for the Python programming assignment "\pi{}-ramidal constants" that was created and used for a test of the 2020--2021 edition of the course (left) and reused as a mandatory assignment in the 2021--2022 edition (right).
#+CAPTION: Graphs constructed from the last submission before the deadline of 142 and 382 students respectively.
#+CAPTION: The colour of each node represents the student's study programme.
#+CAPTION: Edges connect highly similar pairs of submissions, with similarity threshold set to 0.8 in both graphs.
#+CAPTION: Edge directions are based on submission timestamps in Dodona.
#+CAPTION: Clusters of connected nodes are highlighted with a distinct background colour and have one node with a solid border that indicates the first correct submission among all submissions in that cluster.
#+CAPTION: All students submitted unique solutions during the test, except for two students who confessed they exchanged a solution during the test.
#+CAPTION: Submissions for the mandatory assignment show that most students work either individually or in groups of two or three students, but we also observe some clusters of four or more students that exchanged solutions and submitted them with hardly any varying types and amounts of modifications.
#+NAME: fig:usefweplagiarism
[[./images/usefweplagiarism.png]]

In an announcement entitled "copy-paste \neq{} learn to code" we show students some pseudonymized Dolos plagiarism graphs that act as mirrors to make them reflect upon which node in the graph they could be (Figure\nbsp{}[[fig:usefweplagiarism]]).
We stress that the learning effect dramatically drops in groups of four or more students.
Typically, we notice that in such a group only one or a few students make the effort to learn to code, while the other students usually piggyback by copy-pasting solutions.
We make students aware that understanding someone else's code for programming assignments is a lot easier than trying to find solutions themselves.
Over the years, we have experienced that a lot of students are caught in the trap of genuinely believing that being able to understand code is the same as being able to write code that solves a problem until they take a test at the end of a unit.
That's where the \(s\) factor of the test score comes into play.
After all, the goal of summative tests is to evaluate if individual students have acquired the skills to solve programming challenges on their own.

When talking to students about plagiarism, we also point out that the plagiarism graphs are directed graphs, indicating which student is the potential source of exchanging a solution among a cluster of students.
We specifically address these students by pointing out that they are probably good at programming and might want to exchange their solutions with other students in a way to help their peers.
But instead of really helping them out, they actually take away learning opportunities from their fellow students by giving away the solution as a spoiler.
Stated differently, they help maximize the factor \(f\) but effectively also reduce the \(s\) factor of the test score, where both factors need to be high to yield a high score for the unit.
After this lecture, we usually notice a stark decline in the amount of plagiarized solutions.

The goal of plagiarism detection at this stage is prevention rather than penalization, because we want students to take responsibility over their learning.
The combination of realizing that teachers and instructors can easily detect plagiarism and an upcoming test that evaluates if students can solve programming challenges on their own, usually has an immediate and persistent effect on reducing cluster sizes in the plagiarism graphs to at most three students.
At the same time, the signal is given that plagiarism detection is one of the tools we have to detect fraud during tests and exams.
The entire group of students is only addressed once about plagiarism, without going into detail about how plagiarism detection itself works, because we believe that overemphasizing this topic is not very effective and explaining how it works might drive students towards spending time thinking on how they could bypass the detection process, which is time they'd better spend on learning to code.
Every three or four years we see a persistent cluster of students exchanging code for mandatory assignments over multiple weeks.
If this is the case, we individually address these students to point them again on their responsibilities, again differentiating between students that share their solution and students that receive solutions from others.

Tests and exams, on the other hand, are taken on-campus under human surveillance and allow no communication with fellow students or other persons (and more recently, also no generative AI).
Students can work on their personal computers and get exactly two hours to solve two programming assignments during a test, and three hours and thirty minutes to solve three programming assignments during an exam.

Tests and exams are "open book/open Internet", so any hard copy and digital resources can be consulted while solving test or exam assignments.
Students are instructed that they can only be passive users of the Internet: all information available on the Internet at the start of a test or exam can be consulted, but no new information can be added.
When taking over code fragments from the Internet, students have to add a proper citation as a comment in their submitted source code.

After each test and exam, we again use MOSS/Dolos to detect and inspect highly similar code snippets among submitted solutions and to find convincing evidence they result from exchange of code or other forms of interpersonal communication (Figure\nbsp{}[[fig:usefweplagiarism]]).
If we catalogue cases as plagiarism beyond reasonable doubt, the examination board is informed to take further action\nbsp{}[cite:@maertensDolosLanguageagnosticPlagiarism2022].

*** Workload for running a course edition
:PROPERTIES:
:CREATED: [2023-10-24 Tue 13:46]
:CUSTOM_ID: subsubsec:useworkload
:END:

To organize "open book/open Internet" tests and exams that are valid and reliable, we always create new assignments and avoid assignments whose solutions or parts thereof are readily available online.
At the start of a test or exam, we share a token link that gives students access to the assignments in a hidden series on Dodona.

For each edition of the course, mandatory assignments were initially a combination of selected test and exam exercises reused from the previous edition of the course and newly designed exercises.
The former to give students an idea about the level of exercises they can expect during tests and exams, and the latter to avoid solution slippage.
As feedback for the students we publish sample solutions for all mandatory exercises after the weekly deadline has passed.
This also indicates that students must strictly adhere to deadlines, because sample solutions are available afterwards.
As deadlines are very clear and adjusted to timezone settings in Dodona, we never experience discussions with students about deadlines.

After nine editions of the course, we felt we had a large enough portfolio of exercises to start reusing mandatory exercises from four or more years ago instead of designing new exercises for each edition.
However, we still continue to design new exercises for each test and exam.
After each test and exam, exercises are published and students receive manual reviews on the code they submitted, on top of the automated feedback they already got during the test or exam.
But in contrast to mandatory exercises we do not publish sample solutions for test and exam exercises, so that these exercises can be reused during the next edition of the course.
When students ask for sample solutions of test or exam exercises, we explain that we want to give the next generation of students the same learning opportunities they had.

So far, we have created more than 900 programming assignments for this introductory Python course alone.
All these assignments are publicly shared on Dodona as open educational resources\nbsp{}[cite:@hylenOpenEducationalResources2021; @tuomiOpenEducationalResources2013; @wileyOpenEducationalResources2014; @downesModelsSustainableOpen2007; @caswellOpenEducationalResources2008].
They are used in many other courses on Dodona (on average 10.8 courses per assignment) and by many students (on average 503.7 students and 4801.5 submitted solutions per assignment).
We estimate that it takes about 10 person-hours on average to create a new assignment for a test or an exam: 2 hours for ideation, 30 minutes for implementing and tweaking a sample solution that meets the educational goals of the assignment and can be used to generate a test suite for automated assessment, 4 hours for describing the assignment (including background research), 30 minutes for translating the description from Dutch into English, one hour to configure support for automated assessment, and another 2 hours for reviewing the result by some extra pair of eyes.

Generating a test suite usually takes 30 to 60 minutes for assignments that can rely on basic test and feedback generation features that are built into the judge.
The configuration for automated assessment might take 2 to 3 hours for assignments that require more elaborate test generation or that need to extend the judge with custom components for dedicated forms of assessment (e.g.\nbsp{}assessing non-deterministic behaviour) or feedback generation (e.g.\nbsp{}generating visual feedback).
[cite/t:@keuningSystematicLiteratureReview2018] found that publications rarely describe how difficult and time-consuming it is to add assignments to automated assessment platforms, or even if this is possible at all.

The ease of extending Dodona with new programming assignments is reflected by more than {{{num_exercises}}} assignments that have been added to the platform so far.
Our experience is that configuring support for automated assessment only takes a fraction of the total time for designing and implementing assignments for our programming course, and in absolute numbers stays far away from the one person-week reported for adding assignments to Bridge\nbsp{}[cite:@bonarBridgeIntelligentTutoring1988].
Because the automated assessment infrastructure of Dodona provides common resources and functionality through a Docker container and a judge, the assignment-specific configuration usually remains lightweight.
Only around 5% of the assignments need extensions on top of the built-in test and feedback generation features of the judge.

So how much effort does it cost us to run one edition of our programming course?
For the most recent 2021--2022 edition we estimate about 34 person-weeks in total (Table\nbsp{}[[tab:usefweworkload]]), the bulk of which is spent on on-campus tutoring of students during hands-on sessions (30%), manual assessment and grading (22%), and creating new assignments (21%).
About half of the workload (53%) is devoted to summative feedback through tests and exams: creating assignments, supervision, manual assessment and grading.
Most of the other work (42%) goes into providing formative feedback through on-campus and online assistance while students work on their mandatory assignments.
Out of 2215 questions that students asked through Dodona's online Q&A module, 1983 (90%) were answered by teaching assistants and 232 (10%) were marked as answered by the student who originally asked the question.
Because automated assessment provides first-line support, the need for human tutoring is already heavily reduced.
We have drastically cut the time we initially spent on mandatory assignments by reusing existing assignments and because the Python judge is stable enough to require hardly any maintenance or further development.

#+CAPTION: Estimated workload to run the 2021--2022 edition of the introductory Python programming course for 442 students with 1 lecturer, 7 teaching assistants and 3 undergraduate students who serve as teaching assistants\nbsp{}[cite:@gordonUndergraduateTeachingAssistants2013].
#+NAME: tab:usefweworkload
| Task                                | Estimated workload (hours) |
|-------------------------------------+----------------------------|
| <l>                                 |                        <r> |
| Lectures                            |                         60 |
|-------------------------------------+----------------------------|
| Mandatory assignments               |                        540 |
| \emsp{} Select assignments          |                         10 |
| \emsp{} Review selected assignments |                         30 |
| \emsp{} Tips & tricks               |                         10 |
| \emsp{} Automated assessment        |                          0 |
| \emsp{} Hands-on sessions           |                        390 |
| \emsp{} Answering online questions  |                        100 |
|-------------------------------------+----------------------------|
| Tests & exams                       |                        690 |
| \emsp{} Create new assignments      |                        270 |
| \emsp{} Supervise tests and exams   |                        130 |
| \emsp{} Automated assessment        |                          0 |
| \emsp{} Manual assessment           |                        288 |
| \emsp{} Plagiarism detection        |                          2 |
|-------------------------------------+----------------------------|
| Total                               |              1\thinsp{}290 |

*** Learning analytics and educational data mining
:PROPERTIES:
:CREATED: [2023-10-24 Tue 14:04]
:CUSTOM_ID: subsubsec:uselearninganalytics
:END:

A longitudinal analysis of student submissions across the term shows that most learning happens during the 13 weeks of educational activities and that students don't have to catch up practising their programming skills during the exam period (Figure\nbsp{}[[fig:usefwecoursestructure]]).
Active learning thus effectively avoids procrastination.
We observe that students submit solutions every day of the week and show increased activity around hands-on sessions and in the run-up to the weekly deadlines (Figure\nbsp{}[[fig:usefwepunchcard]]).
Weekends are also used to work further on programming assignments, but students seem to be watching over a good night's sleep.

#+CAPTION: Punchcard from the Dodona learning analytics page showing the distribution per weekday and per hour of all 331\thinsp{}734 solutions submitted during the 2021--2022 edition of the course (442 students).
#+NAME: fig:usefwepunchcard
[[./images/usefwepunchcard.png]]

Throughout a course edition, we use Dodona's series analytics to monitor how students perform on our selection of programming assignments (Figures\nbsp{}[[fig:usefweanalyticssubmissions]],\nbsp{}[[fig:usefweanalyticsstatuses]],\nbsp{}and\nbsp{}[[fig:usefweanalyticscorrect]]).
This allows us to make informed decisions and appropriate interventions, for example when students experience issues with the automated assessment configuration of a particular assignment or if the original order of assignments in a series does not seem to align with our design goal to present them in increasing order of difficulty.
The first students that start working on assignments usually are good performers.
Seeing these early birds having trouble with solving one of the assignments may give an early warning that action is needed, as in improving the problem specification, adding extra tips & tricks, or better explaining certain programming concepts to all students during lectures or hands-on sessions.
Reversely, observing that many students postpone working on their assignments until just before the deadline might indicate that some assignments are simply too hard at this moment in time through the learning pathway of the students or that completing the collection of programming assignments interferes with the workload from other courses.
Such "deadline hugging" patterns are also a good breeding ground for students to resort on exchanging solutions among each other.

#+CAPTION: Distribution of the number of student submissions per programming assignment.
#+CAPTION: The larger the zone, the more students submitted a particular number of solutions.
#+CAPTION: Black dot indicates the average number of submissions per student.
#+NAME: fig:usefweanalyticssubmissions
[[./images/usefweanalyticssubmissions.png]]

#+CAPTION: Distribution of top-level submission statuses per programming assignment.
#+NAME: fig:usefweanalyticsstatuses
[[./images/usefweanalyticsstatuses.png]]

#+CAPTION: Progression over time of the percentage of students that correctly solved each assignment.
#+CAPTION: The visualisation starts two weeks before the deadline, which is on the 19th of October.
#+NAME: fig:usefweanalyticscorrect
[[./images/usefweanalyticscorrect.png]]

Using educational data mining techniques on historical data exported from several editions of the course, we further investigated what aspects of practising programming skills promote or inhibit learning, or have no or minor effect on the learning process\nbsp{}(see Chapter\nbsp{}[[#chap:passfail]]).
It won't come as a surprise that midterm test scores are good predictors for a student's final grade, because tests and exams are both summative assessments that are organized and graded in the same way.
However, we found that organizing a final exam end-of-term is still a catalyst of learning, even for courses with a strong focus of active learning during weeks of educational activities.

In evaluating if students gain deeper understanding when learning from their mistakes while working progressively on their programming assignments, we found the old adage that practice makes perfect to depend on what kind of mistakes students make.
Learning to code requires mastering two major competences:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{, }}, itemjoin*={{, and }}]
- getting familiar with the syntax and semantics of a programming language to express the steps for solving a problem in a formal way, so that the algorithm can be executed by a computer
- problem-solving itself.
  It turns out that staying stuck longer on compilation errors (mistakes against the syntax of the programming language) inhibits learning, whereas taking progressively more time to get rid of logical errors (reflective of solving a problem with a wrong algorithm) as assignments get more complex actually promotes learning.
  After all, time spent in discovering solution strategies while thinking about logical errors can be reclaimed multifold when confronted with similar issues in later assignments\nbsp{}[cite:@glassFewerStudentsAre2022].

These findings neatly align with the claim of\nbsp{}[cite/t:@edwardsSeparationSyntaxProblem2018] that problem-solving is a higher-order learning task in the Taxonomy by\nbsp{}[cite/t:@bloom1956handbook] (analysis and synthesis) than language syntax (knowledge, comprehension, and application).

Using historical data from previous course editions, we can also make highly accurate predictions about what students will pass or fail the current course edition\nbsp{}(see Chapter\nbsp{}[[#chap:passfail]]).
This can already be done after a few weeks into the course, so remedial actions for at-risk students can be started well in time.
The approach is privacy-friendly as we only need to process metadata on student submissions for programming assignments and results from automated and manual assessment extracted from Dodona.
Given that cohort sizes are large enough, historical data from a single course edition are already enough to make accurate predictions.

* Technical description
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:49]
:CUSTOM_ID: chap:technical
:END:

Dodona and its ecosystem comprise a lot of code.
This chapter discusses the technical background of Dodona itself\nbsp{}[cite:@vanpetegemDodonaLearnCode2023] and a stand-alone online code editor, Papyros (\url{https://papyros.dodona.be}), that was integrated into Dodona\nbsp{}[cite:@deridderPapyrosSchrijvenUitvoeren2022].
We will also discuss two judges that we were involved with the development of.
The R judge was written entirely by myself\nbsp{}[cite:@nustRockerversePackagesApplications2020].
The TESTed judge was first prototyped in a master's thesis\nbsp{}[cite:@vanpetegemComputationeleBenaderingenVoor2018] and was further developed in two other master's theses\nbsp{}[cite:@selsTESTedProgrammeertaalonafhankelijkTesten2021; @strijbolTESTedOneJudge2020].
In this chapter we assume the reader is familiar with Dodona's features and how they are used, as detailed in Chapters\nbsp{}[[#chap:what]]\nbsp{}and\nbsp{}[[#chap:use]].

** Dodona
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:49]
:CUSTOM_ID: sec:techdodona
:END:

To ensure that Dodona[fn:: https://github.com/dodona-edu/dodona] is robust against sudden increases in workload and when serving hundreds of concurrent users, it has a multi-tier service architecture that delegates different parts of the application to different servers, as can be seen on Figure\nbsp{}[[fig:technicaldodonaservers]].
More specifically, the web server, database (MySQL) and caching system (Memcached) each run on their own machine.
In addition, a scalable pool of interchangeable worker servers are available to automatically assess incoming student submissions.
In this section, we will highlight a few of these components.

#+CAPTION: Diagram of all the servers involved with running and developing Dodona.
#+CAPTION: The role of each server in the deployment is listed below its name.
#+CAPTION: Servers are connected if they communicate.
#+CAPTION: The direction of the connection signifies which server initiates the connection.
#+CAPTION: Every server also has an implicit connection with Phocus (the monitoring server), since metrics such as load, CPU usage, disk usage, etc. are collected and sent to Phocus on every server.
#+CAPTION: The Pandora server is greyed out because it is not used anymore (see Section\nbsp{}[[#subsec:techdodonatutor]] for more info).
#+NAME: fig:technicaldodonaservers
[[./diagrams/technicaldodonaservers.svg]]

*** The Dodona web application
:PROPERTIES:
:CREATED: [2023-11-23 Thu 17:12]
:CUSTOM_ID: subsec:techdodonaweb
:END:

The user-facing part of Dodona runs on the main web server, which also called Dodona (see Figure\nbsp{}[[fig:technicaldodonaservers]]).
Dodona is a Ruby-on-Rails web application that follows the Rails-standard way of organizing functionality in models, views and controllers.

The way we handle complex logic in the frontend has seen a number of changes along the years.
When Dodona was started, there were only a few places where JavaScript was used.
Dodona also used the Rails-standard way of serving dynamically generated JavaScript to replace parts of pages (e.g. for pagination or search).
With the introduction of more complex features like evaluations, we switched to using lightweight web components where this made sense.
We also eliminated jQuery, because more and more of its functionality was implemented natively by browsers.
And lastly, all JavaScript was rewritten to TypeScript.

**** Security and performance
:PROPERTIES:
:CREATED:  [2024-01-10 Wed 14:23]
:CUSTOM_ID: subsubsec:techdodonawebsecurity
:END:

Another important aspect of running a public web application is its security.
Dodona needs to operate in a challenging environment where students simultaneously submit untrusted code to be executed on its servers ("remote code execution as a service") and expect automatically generated feedback, ideally within a few seconds.
Many design decisions are therefore aimed at maintaining and improving the reliability and security of its systems.

Since Dodona grew from being used to teach mostly by people we knew personally to being used in secondary schools all over Flanders, we went from being able to fully trust exercise authors to having this trust reduced (as it is impossible for a team of our size to vet all the people we give teacher's rights in Dodona).
This meant that our threat model and therefore the security measures we had to take also changed over the years.
Once Dodona was opened up to more and more teachers, we gradually locked down what teachers could do with e.g. their exercise descriptions.
Content where teachers can inject raw HTML into Dodona was moved to iframes, to make sure that teachers could still be as creative as they wanted while writing exercises, while simultaneously not allowing them to execute JavaScript in a session where users are logged in.
For user content where this creative freedom is not as necessary (e.g. series or course descriptions), but some Markdown/HTML content is still wanted, we sanitize the (generated) HTML so that it can only include HTML elements and attributes that are specifically allowed.

One of the most important components of Dodona is the feedback table.
It has, therefore, seen a lot of security, optimization and UI work over the years.
Judge and exercise authors (and even students, through their submissions) can determine a lot of the content that eventually ends up in the feedback table.
Therefore, the same sanitization that is used for series and course descriptions is used for the messages that are added to the feedback table (since these can contain Markdown and arbitrary HTML as well).
The increase in teachers that added exercises to Dodona also meant that the variety in feedback given grew, sometimes resulting in a huge volume of testcases and long output.

Optimization work was needed to cope with this volume of feedback.
For example, one of the biggest optimizations was in how expected and generated feedback are diffed and how these diffs are rendered.
When Dodona was first written, the library used for creating diffs of the generated and expected results (=diffy=[fn:: https://github.com/samg/diffy]) actually shelled out to the GNU =diff= command.
This output was parsed and transformed into HTML by the library using find and replace operations.
As one can expect, starting a new process and doing a lot of string operations every time outputs had to be diffed resulted in very slow loading times for the feedback table.
The library was replaced with a pure Ruby library (=diff-lcs=[fn:: https://github.com/halostatue/diff-lcs]), and its outputs were built into HTML using Rails' efficient =Builder= class.
This change of diffing method also fixed a number of bugs we were experiencing along the way.

Even this was not enough to handle the most extreme of exercises though.
Diffing hundreds of lines hundreds of times still takes a long time, even if done in-process while optimized by a JIT.
The resulting feedback tables also contained so much HTML that the browsers on our development machines (which are pretty powerful machines) noticeably slowed down when loading and rendering them.
To handle these cases, we needed to do less work and needed to output less HTML.
We decided to only diff line-by-line (instead of character-by-character) in most of these cases and to not diff at all in the most extreme cases, reducing the amount of HTML required to render them as well.
This was also motivated by usability.
If there are lots of small differences between a very long generated and expected output, the diff view in the feedback table could also become visually overwhelming for students.

*** Judging submissions
:PROPERTIES:
:CREATED:  [2024-01-10 Wed 14:01]
:CUSTOM_ID: subsec:techdodonajudging
:END:

Student submissions are automatically assessed in background jobs by our worker servers (Salmoneus, Sisyphus, Tantalus, Tityos and Ixion; Figure\nbsp{}[[fig:technicaldodonaservers]]).
To divide the work over these servers we make use of a job queue, based on =delayed_job=[fn:: https://github.com/collectiveidea/delayed_job].
Each worker server has 6 job runners, which regularly poll the job queue when idle.

For proper virtualization we use Docker containers\nbsp{}[cite:@pevelerComparingJailedSandboxes2019] that use OS-level containerization technologies and define runtime environments in which all data and executable software (e.g., scripts, compilers, interpreters, linters, database systems) are provided and executed.
These resources are typically pre-installed in the image of the container.
Prior to launching the actual assessment, the container is extended with the submission, the judge and the resources included in the assessment configuration (Figure\nbsp{}[[fig:technicaloutline]]).
Additional resources can be downloaded and/or installed during the assessment itself, provided that Internet access is granted to the container.
When the container is started, limits are placed on the amount of resources it can consume.
This includes a limit in runtime, memory usage, disk usage, network access and the amount of processes a container can have running at the same time.
Some of these limits are (partially) configurable per exercise, but sane upper bounds are always applied.
This is also the case for network access, where even if the container is allowed internet access, it can not access other Dodona hosts (such as the database server).

#+CAPTION: Outline of the procedure to automatically assess a student submission for a programming assignment.
#+CAPTION: Dodona instantiates a Docker container (1) from the image linked to the assignment (or from the default image linked to the judge of the assignment) and loads the submission and its metadata (2), the judge linked to the assignment (3) and the assessment resources of the assignment (4) into the container.
#+CAPTION: Dodona then launches the actual assessment, collects and bundles the generated feedback (5), and stores it into a database along with the submission and its metadata.
#+NAME: fig:technicaloutline
[[./images/technicaloutline.png]]

The actual assessment of the student submission is done by a software component called a /judge/\nbsp{}[cite:@wasikSurveyOnlineJudge2018].
The judge must be robust enough to provide feedback on all possible submissions for the assignment, especially submissions that are incorrect or deliberately want to tamper with the automatic assessment procedure\nbsp{}[cite:@forisekSuitabilityProgrammingTasks2006].
Following the principles of software reuse, the judge is ideally also a generic framework that can be used to assess submissions for multiple assignments.
This is enabled by the submission metadata that is passed when calling the judge, which includes the path to the source code of the submission, the path to the assessment resources of the assignment and other metadata such as programming language, natural language, time limit and memory limit.

Rather than providing a fixed set of judges, Dodona adopts a minimalistic interface that allows third parties to create new judges: automatic assessment is bootstrapped by launching the judge's =run= executable that can fetch the JSON formatted submission metadata from standard input and must generate JSON formatted feedback on standard output.
The feedback has a standardized hierarchical structure that is specified in a JSON schema[fn:: https://github.com/dodona-edu/dodona/tree/main/public/schemas].
At the lowest level, /tests/ are a form of structured feedback expressed as a pair of generated and expected results.
They typically test some behaviour of the submitted code against expected behaviour.
Tests can have a brief description and snippets of unstructured feedback called messages.
Descriptions and messages can be formatted as plain text, HTML (including images), Markdown, or source code.
Tests can be grouped into /test cases/, which in turn can be grouped into /contexts/ and eventually into /tabs/.
All these hierarchical levels can have descriptions and messages of their own and serve no other purpose than visually grouping tests in the user interface.
At the top level, a submission has a fine-grained status that reflects the overall assessment of the submission: =compilation error= (the submitted code did not compile), =runtime error= (executing the submitted code failed during assessment), =memory limit exceeded= (memory limit was exceeded during assessment), =time limit exceeded= (assessment did not complete within the given time), =output limit exceeded= (too much output was generated during assessment), =wrong= (assessment completed but not all strict requirements were fulfilled), or =correct= (assessment completed, and all strict requirements were fulfilled).

Taken together, a Docker image, a judge and a programming assignment configuration (including both a description and an assessment configuration) constitute a /task package/ as defined by\nbsp{}[cite:@verhoeffProgrammingTaskPackages2008]: a unit Dodona uses to render the description of the assignment and to automatically assess its submissions.
However, Dodona's layered design embodies the separation of concerns\nbsp{}[cite:@laplanteWhatEveryEngineer2007] needed to develop, update and maintain the three modules in isolation and to maximize their reuse: multiple judges can use the same docker image and multiple programming assignments can use the same judge.
Related to this, an explicit design goal for judges is to make the assessment configuration for individual assignments as lightweight as possible.
After all, minimal configurations reduce the time and effort teachers and instructors need to create programming assignments that support automated assessment.
Sharing of data files and multimedia content among the programming assignments in a repository also implements the inheritance mechanism for /bundle packages/ as hinted by\nbsp{}[cite/t:@verhoeffProgrammingTaskPackages2008].
Another form of inheritance is specifying default assessment configurations at the directory level, which takes advantage of the hierarchical grouping of learning activities in a repository to share common settings.

*** Python Tutor
:PROPERTIES:
:CREATED:  [2024-01-17 Wed 13:23]
:CUSTOM_ID: subsec:techdodonatutor
:END:

The deployment of the Python Tutor also saw a number of changes over the years.
The Python Tutor itself is written in Python by [cite/t:@guoOnlinePythonTutor2013], so could not be part of Dodona itself.
It started out as a Docker container on the same server as the main Dodona web application.
Because it is used mainly by students who want to figure out their mistakes, the service responsible for running student code could become overwhelmed and in extreme cases even make the entire server unresponsive.
After we identified this issue, the Python tutor was moved to its own server (Pandora in Figure\nbsp{}[[fig:technicaldodonaservers]]).
This did not fix the Tutor itself becoming overwhelmed however, which meant that students that depended on the Tutor were sometimes unable to use it.
This of course happened more during periods where the Tutor was being used a lot, such as evaluations and exams.
One can imagine that the experience for students who are already quite stressed out about the exam they are taking when the Tutor suddenly failed was not very good.
In the meantime, we had started to experiment with running Python code client-side in the browser (see Section\nbsp{}[[#sec:papyros]] for more info).
Because these experiments were successful, we migrated the Python Tutor from its own server to being run by students in their own browser using Pyodide.
This means that the only student that can be impacted by the Python Tutor failing for a testcase is the student themselves (and because the Tutor is being run on a device that is under a far less heavy load, the Python Tutor fails much less often).
In practice, we got no questions or complaints about the Python Tutor's performance after these changes, even during exams where 460 students were submitting simultaneously.

*** Development process
:PROPERTIES:
:CREATED: [2023-11-23 Thu 17:13]
:CUSTOM_ID: subsec:techdodonadevelopment
:END:

Development of Dodona is done on GitHub.
Over the years, Dodona has seen over {{{num_commits}}} commits by {{{num_contributors}}} contributors, and there have been {{{num_releases}}} releases.
All new features and bug fixes are added to the =main= branch through pull requests, of which there have been about {{{num_prs}}}.
These pull requests are reviewed by (at least) two developers of the Dodona team before they are merged.
We also treat pull requests as a form of internal documentation by writing an extensive PR description and adding screenshots for all visual changes or additions.
The extensive test suite also runs automatically for every pull request (using GitHub Actions), and developers are encouraged to add new tests for each feature or bug fix.
We've also made it very easy to deploy to our testing (Mestra) and staging (Naos) environments so that reviewers can test changes without having to spin up their local development instance of Dodona.
These are the two unconnected servers seen in Figure\nbsp{}[[fig:technicaldodonaservers]].
Mestra runs a Dodona instance much like the instance developers use locally.
There is no production data present and in fact, the database is wiped and reseeded on every deploy.
Naos is much closer to the production setup.
It runs on a pseudonymized version of the production database, and has all the judges configured.

We also make sure that our dependencies are always up-to-date using Dependabot[fn:: https://docs.github.com/en/code-security/dependabot/working-with-dependabot].
By updating our dependencies regularly, we make sure that we are not met by incompatibilities that take a long time to integrate when there is an important security update.
Since Dodona is accessible over the public web, it would be problematic if we could not quickly apply security updates.

The way we release Dodona has seen a few changes over the years.
We've gone from a few large releases with bugfix point-releases between them, to lots of smaller releases, to now a /release/ per pull request.
Since ours is the only instance of Dodona, releasing every pull request immediately after merging makes getting feedback from our users a very quick process.
When we did versioned releases we also wrote release notes at the time of release.
Because we don't have versioned releases any more, we now bundle the changes into release notes for every month.
They are mostly autogenerated from the merged PRs, but bigger features are given more context and explanation.

*** Deployment process
:PROPERTIES:
:CREATED: [2023-11-23 Thu 17:13]
:CUSTOM_ID: subsec:techdodonadeployment
:END:

After a pull request is merged, it is automatically deployed by a GitHub action.
This action first runs all the tests again, deploys to the staging server and then deploys to the production servers.
Since Naos has a copy of the production database, the deploy would be stopped if there are any migrations that fail in production.
This way we can be sure the actual production database is never in an inconsistent migration state.
The actual deployment is done by Capistrano[fn:: https://capistranorb.com/].
Capistrano allows us to roll back any deploys and makes clever use of symlinking to make sure that deploys happen without any service interruption.

Backups of the database are automatically saved every day and kept for 12 months.
The backups are rotated according to a grandfather-father-son scheme\nbsp{}[cite:@jessen2010overview].
The backups are taken by dumping a replica database.
The replica database is used because dumping the main database write-locks it while it is being dumped, which would result in Dodona being unusable for a significant amount of time.

We also have an extensive monitoring and alerting system in place, based on Grafana[fn:: https://grafana.com/].
This gives us some superficial analytics about Dodona usage, but can also tell us if there are problems with one of our servers.
The analytics are also calculated using the replica database to avoid putting unnecessary load on our main production database.
The web server and worker servers also send notifications when an error occurs in their runtime.
This is one of the main ways we discover bugs that got through our tests, since our users don't regularly report bugs themselves.
We also get notified when there are long-running requests, since we consider our users having to wait a long time to see the page they requested a bug in itself.
These notifications were an important driver to optimize some pages or to make certain operations asynchronous.

** Papyros
:PROPERTIES:
:CREATED: [2023-11-23 Thu 17:29]
:CUSTOM_ID: sec:papyros
:END:

Papyros[fn:: https://github.com/dodona-edu/papyros] is a stand-alone basic online IDE we developed, primarily focused on secondary education.
Recurring feedback we got from secondary education teachers when introducing Dodona to them was that Dodona did not have a simple way for students to run and test their code themselves.
Testing their code in this case also means manually typing a response to an input prompt when an =input= statement is run by the interpreter.
In the educational practice that Dodona was born out of, this was an explicit design goal.
We wanted to guide students to use an IDE locally instead of programming in Dodona directly, since if they needed to program later in life, they would not have Dodona available as their programming environment.
This same goal is not present in secondary education.
In that context, the challenge of programming is already big enough, without complicating things by installing a real IDE with a lot of buttons and menus that students will never use.
Students might also be working on devices that they don't own (PCs in the school), where installing an IDE might not even be possible.

There are a few reasons why we could not initially offer a simple online IDE.
Even though we can use a lot of the infrastructure very graciously offered by Ghent University, these resources are not limitless.
The extra (interactive) evaluation of student code was something we did not have the resources for, nor did we have any architectural components in place to easily integrate this into Dodona.
The main goal of Papyros was thus to provide a client-side Python execution environment we could then include in Dodona.
We focused on Python because it is the most widely used programming language in secondary education, at least in Flanders.
Note that we don't want to replace Dodona's entire execution model with client-side execution, as the client is an untrusted execution environment where debugging tools could be used to manipulate the results.
Because the main idea is integration in Dodona, we primarily wanted users to be able to execute entire programs, and not necessarily offer a REPL at first.

Given that the target audience for Papyros is secondary education students, we identified a number of secondary requirements:
- The editor of our online IDE should have syntax highlighting.
  Recent literature\nbsp{}[cite:@hannebauerDoesSyntaxHighlighting2018] has shown that this does not necessarily have an impact on students' learning, but as the authors point out, it was the prevailing wisdom for a long time that it does help.
- It should also include linting.
  Linters notify students about syntax errors, but also about style guide violations and antipatterns.
- Error messages for errors that occur during execution should be user-friendly\nbsp{}[cite:@beckerCompilerErrorMessages2019].
- Code completion should be available. When starting out with programming, it is hard to remember all the different functions available.
  Completion frameworks allow students to search for functions, and can show inline documentation for these functions.

*** Execution
:PROPERTIES:
:CREATED: [2023-11-27 Mon 17:28]
:CUSTOM_ID: subsec:papyrosexecution
:END:

Python can not be executed directly by a browser, since only JavaScript and WebAssembly are natively supported.
We investigated a number of solutions for running Python code in the browser.

The first of these is Brython[fn:: https://brython.info].
Brython works by transpiling Python code to JavaScript, where the transpilation is implemented in JavaScript.
The project is conceptualized as a way to develop web applications in Python, and not to run arbitrary Python code in the browser, so a lot of its tooling is not directly applicable to our use case, especially concerning interactive input prompts.
It also runs on the main thread of the browser, so executing a student's code would freeze the browser until it is done running.

Another solution we looked into is Skulpt[fn:: https://skulpt.org].
It also transpiles Python code to JavaScript, and supports Python 2 and Python 3.7.
After loading Skulpt, a global object is added to the page where Python code can be executed through JavaScript.

The final option we looked into was Pyodide[fn:: https://pyodide.org/en/stable].
Pyodide was initially developed by Mozilla as part of their Iodide project, aiming to make scientific research shareable and reproducible via the browser.
It is now a stand-alone project.
Pyodide is a port of the Python interpreter to WebAssembly, allowing it to be executed by the browser.
Since the project is focused on scientific research, it has wide support for external libraries such as NumPy.
Because Pyodide can be treated as a regular JavaScript library, it can be run in a web worker, making sure that the page stays responsive while the user's code is being executed.

We also looked into integrating other platforms such as Repl.it, but none of them were free or did not provide a suitable interface for integration.
We chose to base Papyros on Pyodide given its active development, support for recent Python versions and its ability to be executed on a separate thread.

*** Implementation
:PROPERTIES:
:CREATED: [2023-11-27 Mon 17:28]
:CUSTOM_ID: subsec:papyrosimplementation
:END:

There are two aspects to the implementation: the user interface and the technical inner workings.
Given that this work will primarily be used by secondary school students, the user interface is an important part of this work that should not be neglected.

**** User interface
:PROPERTIES:
:CREATED: [2023-11-29 Wed 14:48]
:CUSTOM_ID: subsubsec:papyrosui
:END:

The most important choice in the user interface was the choice of the editor.
There were three main options:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{, }}, itemjoin*={{, and }}]
- Ace[fn:: https://ace.c9.io/]
- Monaco[fn:: https://microsoft.github.io/monaco-editor/]
- CodeMirror[fn:: https://codemirror.net/].

Ace was the editor used by Dodona at the time.
It supports syntax highlighting and has some built-in linting.
However, it is not very extensible, it doesn't support mobile devices well, and it's no longer actively developed.

Monaco is the editor extracted from Visual Studio Code and often used by people building full-fledged web IDE's.
It also has syntax highlighting and linting and is much more extensible.
As with Ace though, support for mobile devices is lacking.

CodeMirror is a modern editor made for the web, and not linked to any specific project.
It is also extensible and has modular syntax highlighting and linting support.
In contrast with Ace and Monaco, it has very good support for mobile devices.
Its documentation is also very clear and extensive.
Given the clear advantages, we decided to use CodeMirror for Papyros.

The two other main components of Papyros are the output window and the input window.
The output window is a simple read-only textarea.
The input window is a text area that has two modes: interactive mode and batch input.
In interactive mode, the user is expected to write the input needed by their program the moment it asks for it (similar to running their program on the command line and answering the prompts when they appear).
In batch mode, the user can prefill all the input required by their program.
The full user interface can be seen in Figure\nbsp{}[[fig:technicalpapyros]].

#+CAPTION: User interface of Papyros.
#+CAPTION: The editor can be seen on the left, with the output window to the right of it.
#+CAPTION: The input window is below the output window and is currently in batch mode.
#+CAPTION: All empty text fields have placeholder text that explains how they can be used.
#+NAME: fig:technicalpapyros
[[./images/technicalpapyros.png]]

**** Inner workings
:PROPERTIES:
:CREATED: [2023-11-29 Wed 14:48]
:CUSTOM_ID: subsubsec:papyrosinner
:END:

Since Pyodide does the heavy lifting of executing the actual Python code, most of the implementation work consisted of making Pyodide run in a web worker and hooking up the Python internals to our user interface.
The communication between the main UI thread and the web worker happens via message passing.
With message passing, all data has to be copied.
To avoid having to copy large amounts of data, and to be able to copy actual functions, classes or HTML elements, shared memory can be used.
To work correctly with shared memory, synchronization primitives have to be used.

After loading Pyodide, we load a Python script that overwrites certain functions with our versions.
For example, base Pyodide will overwrite =input= with a function that calls into JavaScript-land and executes =prompt=.
Since we're running Pyodide in a web worker, =prompt= is not available (and we want to implement custom input handling anyway).
For =input= we actually run into another problem: =input= is synchronous in Python.
In a normal Python environment, =input= will only return a value once the user entered some value on the command line.
We don't want to edit user code (to make it asynchronous) because that process is error-prone and fragile.
So we need a way to make our overwritten version of =input= synchronous as well.

The best way to do this is by using the synchronization primitives of shared memory.
We can block on some other thread writing to a certain memory location, and since blocking is synchronous, this makes our =input= synchronous as well.
Unfortunately, not all browser supported shared memory at the time.
Other browsers also severely constrain the environment in which shared memory can be used, since a number of CPU side channel attacks related to it were discovered.

Luckily, there is another way we can make the browser perform indefinite synchronous operations from a web worker.
Web workers can perform synchronous HTTP requests.
We can then intercept these HTTP requests from a service worker.
Service workers were originally conceived to allow web applications to continue functioning even when devices go offline.
In that case, a service worker could respond to network requests with data it has in its cache.
So, putting this together, the web worker tells the main thread that it needs input and then fires off a synchronous HTTP request to some non-existent endpoint.
The service worker intercepts this request, and responds to the request once it receives some input from the main thread.

The functionality for performing synchronous communication with the main thread from a web worker was parcelled off into its own library (=sync-message=[fn:: https://github.com/alexmojaki/sync-message]).
This library could then decide which of these two methods to use, depending on the available environment.
Another package, =python_runner=[fn:: https://github.com/alexmojaki/python_runner], bundles all required modifications to the Python environment in Pyodide.
This work was done in collaboration with Alex Hall.

**** Extensions
:PROPERTIES:
:CREATED: [2023-12-07 Thu 15:19]
:CUSTOM_ID: subsubsec:papyrosextensions
:END:

CodeMirror already has a number of functionalities it supports out of the box such as linting and code completion.
It is, however, a pure JavaScript library.
This means that these functionalities had to be newly implemented, since the standard tooling for Python is almost entirely implemented in Python.
Fortunately CodeMirror also supports supplying one's own linting message and code completion.
Since we have a working Python environment, we can also use it to run the standard Python tools for linting (PyLint) and code completion (Jedi) and hook up their results to CodeMirror.
For code completion this has the added benefit of also showing the documentation for the autocompleted items, which is especially useful for people new to programming (which is exactly our target audience).

Usability was further improved by adding the =FriendlyTraceback= library.
=FriendlyTraceback= is a Python library that changes error messages in Python to be clearer to beginners, by explicitly answering questions such as where and why an error occurred.

** R judge
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:49]
:CUSTOM_ID: sec:techr
:END:

Because Dodona had proven itself as a useful tool for teaching Python and Java to students, colleagues teaching statistics started asking if we could build R support into Dodona.
We started working on an R judge[fn:: https://github.com/dodona-edu/judge-r] soon after.
By now, more than 1\thinsp{}250 R exercises have been added, and almost 1 million submissions have been made to an R exercise.

Because R is the /lingua franca/ of statistics, there are a few extra features that come to mind that are not typically handled by judges, such as handling of data frames and outputting visual graphs (or even evaluating that a graph was built correctly).
Another feature that teachers wanted that we had not built into a judge previously was support for inspecting the student's source code, e.g. for making sure that certain functions were or were not used.

*** Exercise API
:PROPERTIES:
:CREATED:  [2024-01-05 Fri 14:06]
:CUSTOM_ID: subsec:techrapi
:END:

The API for the R judge was designed to follow the visual structure of the feedback table as closely as possible, as can be seen in the sample evaluation code in Listing\nbsp{}[[lst:technicalrsample]].
Tabs are represented by different evaluation files.
In addition to the =testEqual= function demonstrated in Listing\nbsp{}[[lst:technicalrsample]] there are some other functions to specifically support the requested functionality.
=testImage= will set up some handlers in the R environment so that generated plots (or other images) are sent to the feedback table (in a base 64 encoded string) instead of the filesystem.
It will also by default make the test fail if no image was generated (but does not do any verification of the image contents).
An example of what the feedback table looks like when an image is generated can be seen in Figure\nbsp{}[[fig:technicalrplot]].
=testDF= has some extra functionality for testing the equality of data frames, where it is possible to ignore row and column order.
The generated feedback is also limited to 5 lines of output, to avoid overwhelming students (and their browsers) with the entire table.
=testGGPlot= can be used to introspect plots generated with GGPlot\nbsp{}[cite:@wickhamGgplot2CreateElegant2023].
To test whether students use certain functions, =testFunctionUsed= and =testFunctionUsedInVar= can be used.
The latter tests whether the specific function is used when initializing a specific variable.

#+CAPTION: Feedback table showing the feedback for an R exercise where the goal is to generate a plot.
#+CAPTION: The code generates a plot showing a simple sine function, which is reflected in the feedback table.
#+NAME: fig:technicalrplot
[[./images/technicalrplot.png]]

If some code needs to be executed in the student's environment before the student's code is run (e.g. to make some dataset available, or to fix a random seed), the =preExec= argument of the =context= function can be used to do so.

#+CAPTION: Sample evaluation code for a simple R exercise.
#+CAPTION: The feedback table will contain one context with two testcases in it.
#+CAPTION: The first testcase checks whether some t-test was performed correctly, and does this by performing two equality checks.
#+CAPTION: The second testcase checks that the \(p\)-value calculated by the t-test is correct.
#+CAPTION: The =preExec= is executed in the student's environment and here fixes a random seed for the student's execution.
#+NAME: lst:technicalrsample
#+ATTR_LATEX: :float t
#+BEGIN_SRC r
context({
  testcase('The correct method was used', {
    testEqual("test$alternative",
              function(studentEnv) {
                studentEnv$test$alternative
              },
              'two.sided')
    testEqual("test$method",
              function(studentEnv) {
                studentEnv$test$method
              },
              ' Two Sample t-test')
  })
  testcase('p value is correct', {
    testEqual("test$p.value",
              function(studentEnv) {
                studentEnv$test$p.value
              },
              0.175)
  })
}, preExec = {
  set.seed(20190322)
})
#+END_SRC

*** Security
:PROPERTIES:
:CREATED:  [2024-01-05 Fri 14:06]
:CUSTOM_ID: subsec:techrsecurity
:END:

Other than the API for teachers creating exercises, encapsulation of student code is also an important part of a judge.
Students should not be able to access functions defined by the judge, or be able to find the correct solution or the evaluating code.
The R judge makes sure of this by making extensive use of environments.
This is also reflected in the teacher API: they can access variables or execute functions in the student environment, but this environment has to be explicitly passed to the function generating the student result.
In R, all environments except the root environment have a parent, essentially creating a tree structure of environments.
In most cases, this tree will actually be a path, but in the R judge, the student environment is explicitly attached to the base environment.
This even makes sure that libraries loaded by the judge are not initially available to the student code (thus allowing teachers to test that students can correctly load libraries).
The judge itself runs in an anonymous environment, so that even students with intimate knowledge of the inner workings of R and the judge itself would not be able to find this environment.

The judge is also programmed very defensively.
Every time execution is handed off to student code (or even teacher code), appropriate error handlers and output redirections are installed.
This prevents the student and teacher code from e.g. writing to standard output (and thus messing up the JSON expected by Dodona).

** TESTed
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:49]
:CUSTOM_ID: sec:techtested
:END:

TESTed[fn:: https://github.com/dodona-edu/universal-judge] is a universal judge for Dodona.
TESTed was developed to solve two major drawbacks with the current judge system of Dodona:
- When creating the same exercise in multiple programming languages, the exercise description and test cases need to be redone for every programming language.
  This is especially relevant for very simple exercises that students almost always start with, and for exercises in algorithms courses, where the programming language a student solves an exercise in is of lesser importance than the way they solve it.
  Mistakes in exercises also have to be fixed in all instances of the exercise when there are multiple instances of the exercise.
- The judges themselves have to be created from scratch every time.
  Most judges offer the same basic concepts and features, most of which are independent of programming language (communication with Dodona, checking correctness, I/O,\nbsp{}...).

The goal of TESTed was to implement a judge so that programming exercises only have to be created once to be available in all programming languages TESTed supports.
An exercise should also not have to be changed when support for a new programming language is added.
As a secondary goal, we also wanted to make it as easy as possible to create new exercises.
Teachers who have not used Dodona before should be able to create a new basic exercise without too many issues.

We first developed it as a proof of concept in my master's thesis\nbsp{}[cite:@vanpetegemComputationeleBenaderingenVoor2018], which presented a method for estimating the computational complexity of solutions for programming exercises.
One of the goals was to make this method work over many programming languages.
To do this, we wrote a framework based on Jupyter kernels[fn:: https://jupyter.org] where the interaction with each programming language was abstracted away behind a common interface.
We realized this framework could be useful in itself, but it was only developed as far as we needed for the thesis.
Further work then developed this proof of concept into the full judge we will present in the following sections.

*** Overview
:PROPERTIES:
:CREATED:  [2024-01-05 Fri 14:03]
:CUSTOM_ID: subsec:techtestedoverview
:END:

TESTed generally works using the following steps:
1. Receive the submission, exercise test plan, and any auxiliary files from Dodona.
1. Validate the test plan and making sure the submission's programming language is supported for the given exercise.
1. Generate test code for each context in the test plan.
1. Optionally compile the test code, either in batch mode or per context.
   This step is skipped if evaluation a submission written in an interpreted language.
1. Execute the test code.
1. Evaluate the results, either with programming language-specific evaluation, programmed evaluation, or generic evaluation.
1. Send the evaluation results to Dodona.

In the following sections we will expand on these steps using an example exercise to demonstrate them in practice.
In this exercise, students need to rotate a list.
For example, in Python, ~rotate([0, 1, 2, 3, 4], 2)~ should return ~[3, 4, 0, 1, 2]~.

*** Test plan 
:PROPERTIES:
:CREATED: [2024-01-02 Tue 10:23]
:CUSTOM_ID: subsec:techtestedtestplan
:END:

One of the most important elements that is needed to perform these steps is the test plan.
This test plan is a hierarchical structure, which closely resembles the underlying structure of Dodona's feedback table.
There are, however, a few important differences.
The first of these is the /context testcase/.
This is a special testcase per context that executes the main function (or the entire program in case this is more appropriate for the language being executed).
The only possible inputs for this testcase are text for the standard input stream, command-line arguments and files in the working directory.
The exit status code can only be checked in this testcase as well.

Like the communication with Dodona, this test plan is a JSON document under the hood.
In the following sections, we will use the JSON representation of the test plan to discuss how TESTed works.
Exercise authors use the DSL to write their tests, which we discuss in Section\nbsp{}[[#subsec:techtesteddsl]].
This DSL is internally converted by TESTed to a JSON test plan before execution.
A test plan of the example exercise can be seen in Listing\nbsp{}[[lst:technicaltestedtestplan]].

#+CAPTION: Basic structure of a test plan.
#+CAPTION: The structure of Dodona's feedback table is followed closely.
#+CAPTION: The function arguments have been left out, as they are explained in Section\nbsp{}[[#subsec:techtestedserialization]].
#+NAME: lst:technicaltestedtestplan
#+ATTR_LATEX: :float t
#+BEGIN_SRC js
{
  "tabs": [
    {
      "name": "Feedback",
      "contexts": [
        {
          "testcases": [
            {
              "input": {
                "type": "function",
                "name": "rotate",
                "arguments": [
                  ...
                ]
              },
              "output": {
                "result": {
                  "value": {
                    ...
                  }
                }
              }
            },
            ...
          ]
        }
      ]
    }
  ]
}
#+END_SRC

*** Data serialization
:PROPERTIES:
:CREATED: [2024-01-02 Tue 10:50]
:CUSTOM_ID: subsec:techtestedserialization
:END:

As part of the test plan, we also need a way to generically describe values and their types.
This is what we will call the /serialization format/.
The serialization format should be able to represent all the basic data types we want to support in the programming language independent part of the test plan.
These data types are basic primitives like integers, reals (floating point numbers), booleans, and strings, but also more complex collection types like arrays (or lists), sets and mapping types (maps, dictionaries, and objects).
Note that the serialization format is also used on the side of the programming language, to receive (function) arguments and send back execution results.

Of course, a number of data serialization formats already exist, like =MessagePack=[fn:: https://msgpack.org/], =ProtoBuf=[fn:: https://protobuf.dev/],\nbsp{}...
Binary formats were excluded from the start, because they can't easily be embedded in our JSON test plan, but more importantly, they can neither be written nor read by humans.
Other formats did not support all the types we wanted to support and could not be extended to do so.
Because of our goal in supporting many programming languages, the format also had to be either widely implemented or be easily implementable.
None of the formats we investigated met all these requirements.
We opted to make the serialization format in JSON as well.
Values are represented by objects containing the encoded value and the accompanying type.
Note that this is a recursive format: the values in a collection are also serialized according to this specification.

The types of values are split in three categories.
The first category are the basic types listed above.
The second category are the advanced types.
These are specialized versions of the basic types, for example to specify the number of bits that a number should be, or whether a collection should be a tuple or a list.
The final category of types can only be used to specify an expected type.
In addition to the other categories, =any= can be specified.
Like the name says, =any= signifies that the expected type is unknown, and the student can therefore return any type.

The encoded expected return value of our example exercise can be seen in Listing\nbsp{}[[lst:technicaltestedtypes]].

#+CAPTION: A list encoded using TESTed's data serialization format.
#+CAPTION: The corresponding Python list would be ~[3, 4, 0, 1, 2]~.
#+NAME: lst:technicaltestedtypes
#+ATTR_LATEX: :float t
#+BEGIN_SRC js
{
  "type": "sequence",
  "data": [
    { "type": "integer", "data": 3 },
    { "type": "integer", "data": 4 },
    { "type": "integer", "data": 0 },
    { "type": "integer", "data": 1 },
    { "type": "integer", "data": 2 }
  ]
}
#+END_SRC

*** Statements
:PROPERTIES:
:CREATED:  [2024-01-03 Wed 17:09]
:CUSTOM_ID: subsec:techtestedstatements
:END:

There is more complexity hidden in the idea of creating a variable of a custom type.
It implies that we need to be able to create variables, instead of just capturing the result of function calls or other expressions.
To support this, specific structures were added to the test plan JSON schema.
Listing\nbsp{}[[lst:technicaltestedassignment]] shows what it would look like if we wanted to assign the function argument of our example exercise to a variable.

#+CAPTION: A TESTed testcase containing a statement.
#+CAPTION: The corresponding Python statement would be ~numbers01 = [0, 1, 2, 3, 4]~.
#+NAME: lst:technicaltestedassignment
#+ATTR_LATEX: :float t
#+BEGIN_SRC js
"testcases": [
  {
    "input": {
      "type": "sequence",
      "variable": "numbers01",
      "expression": {
        "type": "sequence",
        "data": [
          { "type": "integer", "data": 0 },
          { "type": "integer", "data": 1 },
          { "type": "integer", "data": 2 },
          { "type": "integer", "data": 3 },
          { "type": "integer", "data": 4 }
        ],
      }
    }
  }
]
#+END_SRC

*** Checking programming language support
:PROPERTIES:
:CREATED:  [2024-01-04 Thu 09:16]
:CUSTOM_ID: subsec:techtestedsupport
:END:

We also need to make sure that the programming language of the submission under test is supported by the test plan of its exercise.
The two things that are checked are whether a programming language supports all the types that are used and whether the language has all the necessary language constructs.
For example, if the test plan uses a =tuple=, but the language doesn't support it, it's obviously not possible to evaluate a submission in that language.
The same is true for overloaded functions: if it is necessary that a function can be called with a string and with a number, a language like C will not be able to support this.
Collections also are not yet supported for C, since the way arrays and their lengths work in C is quite different from other languages.
Our example exercise will not work in C for this reason.

*** Execution
:PROPERTIES:
:CREATED:  [2024-01-04 Thu 09:43]
:CUSTOM_ID: subsec:techtestedexecution
:END:

To go from the generic test plan to something that can actually be executed in the given language, we need to generate test code.
This is done by way of a templating system.
For each programming language supported by TESTed, a few templates need to be defined.
The serialization format also needs to be implemented in the given programming language.
Because the serialization format is based on JSON and JSON is a widely used format, this requirement is usually pretty easy to fulfil.

For some languages, the code needs to be compiled as well.
All test code is usually compiled into one executable, since this only results in one call to the compiler (which is usually a pretty slow process).
There is one big drawback to this way of compiling code: if there is a compilation error (for example because a student has not yet implemented all requested functions) the compilation will fail for all contexts.
Because of this, TESTed will fall back to separate compilations for each context if a compilation error occurs.
Subsequently, the test code is executed and its results collected.

*** Evaluation
:PROPERTIES:
:CREATED:  [2024-01-04 Thu 10:45]
:CUSTOM_ID: subsec:techtestedevaluation
:END:

The generated output is usually evaluated by TESTed itself.
TESTed can however only evaluate the output as far as it is programmed to do so.
There are two other ways the results can be evaluated: programmed evaluation and programming-language specific evaluation.
With programmed evaluation, the results are passed to code written by a teacher.
For efficiency's sake, this code has to be written in Python (which means TESTed does not need to launch a new process for the evaluation).
This code will then check the results, and generate appropriate feedback.
Programming-language specific evaluation is executed immediately after the test code in the process of the test code.
This can be used to evaluate programming-language specific concepts, for example the correct use of pointers in C.

*** Linting
:PROPERTIES:
:CREATED:  [2024-01-04 Thu 10:47]
:CUSTOM_ID: subsec:techtestedlinting
:END:

Next to correctness, style is also an important aspect of source code.
In a lot of contexts, linters are used to perform basic style checks.
Linting was also implemented in TESTed.
For each supported programming language, both the linter to be used and how its output should be interpreted are specified.

*** DSL
:PROPERTIES:
:CREATED:  [2023-12-11 Mon 17:22]
:CUSTOM_ID: subsec:techtesteddsl
:END:

As mentioned in Section\nbsp{}[[#subsec:techtestedtestplan]], exercise authors are not expected to write their test plans in JSON.
It is very verbose and error-prone when writing (trailing commas are not allowed, all object keys are strings and need to be written as such, etc.).
This aspect of usability was not the initial focus of TESTed, since most Dodona power users already use code to generate their test plans.
Because code is very good at outputting an exact and verbose format like JSON, this avoids its main drawback.
However, we wanted teachers in secondary education to be able to work with TESTed, and they mostly do not have enough experience with programming themselves to generate a test plan.
To solve this problem we wanted to integrate a domain-specific language (DSL) to describe TESTed test plans.

We first investigated whether we could use an existing format to do so.
The best option of these was PEML: the Programming Exercise Markup Language\nbsp{}[cite:@mishraProgrammingExerciseMarkup2023].
Envisioned as a universal format for programming exercise descriptions, their goals seemed to align with ours.
Unfortunately, they did not base themselves on any existing formats.
This means that there is little tooling around PEML.
Parsing it as part of TESTed would require a lot of implementation work, and IDEs or other editors don't do syntax highlighting for it.
The format itself is also quite error-prone when writing.
Because of these reasons, we discarded PEML and started working on our own DSL.

Our own DSL is based on YAML[fn:: https://yaml.org].
YAML is a superset of JSON and describes itself as "a human-friendly data serialization language for all programming languages".
The DSL structure is quite similar to the actual test plan, though it does limit the amount of repetition required for common operations.
YAML's concise nature also contributes to the read- and writability of its test plans.

The main addition of the DSL is an abstract programming language, made to look somewhat like Python 3.
Note that this is not a full programming language, but only supports language constructs as far as they are needed by TESTed.
Values are interpreted as basic types, but can be cast explicitly to one of the more advanced types.
The DSL version of the example exercise can be seen in Listing\nbsp{}[[lst:technicaltesteddsl]].

#+CAPTION: DSL version of the example exercise.
#+CAPTION: This version also demonstrates the use of an assignment.
#+NAME: lst:technicaltesteddsl
#+ATTR_LATEX: :float t
#+BEGIN_SRC yaml
  - tab: "Feedback"
    contexts:
      - testcases:
          - statement: "numbers01 = [0, 1, 2, 3, 4]"
          - expression: "rotate(numbers01, 2)"
            return: "[3, 4, 0, 1, 2]"
          - expression: "rotate(numbers01, 1)"
            return: "[4, 0, 1, 2, 3]"
      - testcases:
          - statement: "numbers02 = [0, 1, 2, 3, 4, 5]"
          - expression: "rotate(numbers02, 2)"
            return: "[4, 5, 0, 1, 2, 3]"
          - expression: "rotate(numbers02, 1)"
            return: "[5, 0, 1, 2, 3, 4]"
#+END_SRC

* Pass/fail prediction in programming courses
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:50]
:CUSTOM_ID: chap:passfail
:END:

This chapter is based on\nbsp{}[cite/t:@vanpetegemPassFailPrediction2022], published in the Journal of Educational Computing Research, and also briefly discusses the work performed in\nbsp{}[cite/t:@zhidkikhReproducingPredictiveLearning2024], published in the Journal of Learning Analytics.

** Introduction
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:50]
:CUSTOM_ID: sec:passfailintro
:END:

A lot of educational opportunities are missed by keeping assessment separate from learning\nbsp{}[cite:@wiliamWhatAssessmentLearning2011; @blackAssessmentClassroomLearning1998].
Educational technology can bridge this divide by providing real-time data and feedback to help students learn better, teachers teach better, and education systems become more effective\nbsp{}[cite:@oecdOECDDigitalEducation2021].
Earlier research demonstrated that the adoption of interactive platforms may lead to better learning outcomes\nbsp{}[cite:@khalifaWebbasedLearningEffects2002] and allows collecting rich data on student behaviour throughout the learning process in non-evasive ways.
Effectively using such data to extract knowledge and further improve the underlying processes, which is called educational data mining\nbsp{}[cite:@bakerStateEducationalData2009], is increasingly explored as a way to enhance learning and educational processes\nbsp{}[cite:@duttSystematicReviewEducational2017].

About one third of the students enrolled in introductory programming courses fail\nbsp{}[cite:@watsonFailureRatesIntroductory2014; @bennedsenFailureRatesIntroductory2007].
Such high failure rates are problematic in light of low enrolment numbers and high industrial demand for software engineering and data science profiles\nbsp{}[cite:@watsonFailureRatesIntroductory2014].
To remedy this situation, it is important to have detection systems for monitoring at-risk students, understand why they are failing, and develop preventive strategies.
Ideally, detection happens early on in the learning process to leave room for timely feedback and interventions that can help students increase their chances of passing a course.

Previous approaches for predicting performance on examinations either take into account prior knowledge such as educational history and socio-economic background of students or require extensive tracking of student behaviour.
Extensive behaviour tracking may directly impact the learning process itself.
[cite/t:@rountreeInteractingFactorsThat2004] used decision trees to find that the chance of failure strongly correlates with a combination of academic background, mathematical background, age, year of study, and expectation of a grade other than "A".
They conclude that students with a skewed view on workload and content are more likely to fail.
[cite/t:@kovacicPredictingStudentSuccess2012] used data mining techniques and logistic regression on enrolment data to conclude that ethnicity and curriculum are the most important factors for predicting student success.
They were able to predict success with 60% accuracy.
[cite/t:@asifAnalyzingUndergraduateStudents2017] combine examination results from the last two years in high school and the first two years in higher education to predict student performance in the remaining two years of their academic study program.
They used data from one cohort to train models and from another cohort to test that the accuracy of their predictions is about 80%.
This evaluates their models in a similar scenario in which they could be applied in practice.

A downside of the previous studies is that collecting uniform and complete data on student enrolment, educational history and socio-economic background is impractical for use in educational practice.
Data collection is time-consuming and the data itself can be considered privacy-sensitive.
Usability of predictive models therefore not only depends on their accuracy, but also on their dependency on findable, accessible, interoperable and reusable data\nbsp{}[cite:@wilkinsonFAIRGuidingPrinciples2016].
Predictions based on educational history and socio-economic background also raise ethical concerns.
Such background information definitely does not explain everything and lowers the perceived fairness of predictions\nbsp{}[cite:@grgic-hlacaCaseProcessFairness2018; @binnsItReducingHuman2018].
A student can also not change their background, so these items are not actionable for any corrective intervention.

It might be more convenient and acceptable if predictive models are restricted to data collected on student behaviour during the learning process of a single course.
An example of such an approach comes from\nbsp{}[cite/t:@vihavainenPredictingStudentsPerformance2013], using snapshots of source code written by students to capture their work attitude.
Students are actively monitored while writing source code and a snapshot is taken automatically each time they edit a document.
These snapshots undergo static and dynamic analysis to detect good practices and code smells, which are fed as features to a non-parametric Bayesian network classifier whose pass/fail predictions are 78% accurate by the end of the semester.
In a follow-up study they applied the same data and classifier to accurately predict learning outcomes for the same student cohort in another course\nbsp{}[cite:@vihavainenUsingStudentsProgramming2013].
In this case, their predictions were 98.1% accurate, although the sample size was rather small.
While this procedure does not rely on external background information, it has the drawback that data collection is more invasive and directly intervenes with the learning process.
Students can't work in their preferred programming environment and have to agree with extensive behaviour tracking.

Approaches that are not using machine learning also exist.
[cite/t:@feldmanAnsweringAmRight2019] try to answer the question "Am I on the right track?" on the level of individual exercises, by checking if the student's current progress can be used as a base to synthesise a correct program.
However, there is no clear way to transform this type of approach into an estimation of success on examinations.
[cite/t:@werthPredictingStudentPerformance1986] found significant (\(p < 0.05\)) correlations between students' college grades, the number of hours worked, the number of high school mathematics classes and the students' grades for an introductory programming course.
[cite/t:@gooldFactorsAffectingPerformance2000] also looked at learning style (surveyed using LSI2) as a factor in addition to demographics, academic ability, problem-solving ability and indicators of personal motivation.
The regressions in their study account for 42 to 65 percent of the variation in cohort performances.

In this chapter, we present an alternative framework (Figure\nbsp{}[[fig:passfailmethodoverview]]) to predict if students will pass or fail a course within the same context of learning to code.
The method only relies on submission behaviour for programming exercises to make accurate predictions and does not require any prior knowledge or intrusive behaviour tracking.
Interpretability of the resulting models was an important design goal to enable further investigation on learning habits.
We also focused on early detection of at-risk students, because predictive models are only effective for the cohort under investigation if remedial actions can be started long before students take their final exam.

#+CAPTION: Step-by-step process of the proposed pass/fail prediction framework for programming courses: 1) Collect metadata from student submissions during successive course editions.
#+CAPTION: 2) Align course editions by identifying corresponding time points and calculating snapshots at these time points.
#+CAPTION: A snapshot measures student performance only from metadata available in the course edition at the time the snapshot was taken.
#+CAPTION: 3) Train a machine learning model on snapshot data from previous course editions and predict which students will likely pass or fail the current course edition by applying the model on a snapshot of the current edition.
#+CAPTION: 4) Infer what learning behaviour has a positive or negative learning effect by interpreting feature weights of the machine learning model.
#+CAPTION: Teachers can use insights from both steps 3 and 4 to take actions in their teaching practice.
#+NAME: fig:passfailmethodoverview
[[./images/passfailmethodoverview.png]]

The chapter starts with a description of how data is collected, what data is used and which machine learning methods have been evaluated to make pass/fail predictions.
We evaluated the same models and features in multiple courses to test their robustness against differences in teaching styles and student backgrounds.
The results are discussed from a methodological and educational perspective with a focus on
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{, }}, itemjoin*={{, and }}]
- accuracy (What machine learning algorithms yield the best predictions?)
- early detection (Can we already make accurate predictions early on in the semester?)
- interpretability (Are resulting models clear about which features are important? Can we explain why certain features are identified as important? How self-evident are important features?).

** Materials and methods
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:50]
:CUSTOM_ID: sec:passfailmaterials
:END:

*** Course structures
:PROPERTIES:
:CREATED: [2023-10-23 Mon 16:28]
:CUSTOM_ID: subsec:passfailstructures
:END:

This study uses data from two introductory programming courses (referenced as course A and course B) collected during 3 editions of each course in academic years 2016--2017, 2017--2018 and 2018--2019.
Both courses run once per academic year across a 12-week semester (September--December).
They have separate lecturers and teaching assistants, and are taken by students of different faculties.
The courses have their own structure, but each edition of a course follows the same structure.
Table\nbsp{}[[tab:passfailcoursestatistics]] summarizes some statistics on the course editions included in this study.

#+ATTR_LATEX: :float t
#+CAPTION: Statistics for course editions included in this study.
#+CAPTION: The courses are taken by different student cohorts at different faculties and differ in structure, lecturers and teaching assistants.
#+CAPTION: The number of tries is the average number of solutions submitted by a student per exercise they worked on (i.e. for which the student submitted at least one solution in the course edition).
#+NAME: tab:passfailcoursestatistics
|     | year       | students | # ex. |       solutions | tries | pass rate |
|-----+------------+----------+-------+-----------------+-------+-----------|
| <l> | <l>        |      <r> |   <r> |             <r> |   <r> |       <r> |
| A   | 2016--2017 |      322 |    60 | 167\thinsp{}675 |  9.56 |    60.86% |
| A   | 2017--2018 |      249 |    60 | 125\thinsp{}920 |  9.19 |    61.44% |
| A   | 2018--2019 |      307 |    60 | 176\thinsp{}535 | 10.29 |    65.14% |
| B   | 2016--2017 |      372 |   138 | 371\thinsp{}891 |  9.10 |    56.72% |
| B   | 2017--2018 |      393 |   187 | 407\thinsp{}696 |  7.31 |    60.81% |
| B   | 2018--2019 |      437 |   201 | 421\thinsp{}461 |  6.26 |    62.47% |

Course A is subdivided into two successive instructional units that each cover five programming topics -- one topic per week -- followed by an evaluation about all topics covered in the unit.
Students must solve six programming exercises on each topic before a deadline one week later.
Submitted solutions for these mandatory exercises are automatically evaluated and considered correct if they pass all unit tests for the exercise.
Failing to submit a correct solution for a mandatory exercise has a small impact on the score for the evaluation at the end of the unit.
The final exam at the end of the semester evaluates all topics covered in the entire course.
Students need to solve new programming exercises during evaluations (2 exercises) and exams (3 exercises), where reviewers manually evaluate and grade submitted solutions based on correctness, programming style used, choice made between the use of different programming techniques, and the overall quality of the solution.
Each edition of the course is taken by about 300 students.

Course B has 20 lab sessions across the semester, with evaluations after the 10th and 17th lab session and a final exam at the end of the semester.
Each lab session comes with a set of exercises and has an indicative deadline for submitting solutions.
However, these exercises are not taken into account when computing the final score for the course, so students are completely free to work on exercises as a way to practice their coding skills.
Students need to solve new programming exercises during evaluations (3 exercises) and exams (4 exercises).
Solutions submitted during evaluations are automatically graded based on the number of passed unit tests for the exercise.
Solutions submitted during exams are manually graded in the same way as for course A.
Each edition of the course is taken by about 400 students.

We opted to use two different courses that are structured quite differently to make sure our framework is generally applicable in other courses where the same behavioural data can be collected.

*** Learning environment
:PROPERTIES:
:CREATED: [2023-10-23 Mon 16:28]
:CUSTOM_ID: subsec:passfaillearningenvironment
:END:

Both courses use Dodona as their online learning environment.
Dodona promotes active learning through problem-solving\nbsp{}[cite:@princeDoesActiveLearning2004].
Each course edition has its own Dodona course, with a learning path that groups exercises in separate series (Figure\nbsp{}[[fig:passfailstudentcourse]]).
Course A has one series per covered programming topic (10 series in total) and course B has one series per lab session (20 series in total).
A submission deadline is set for each series.
Dodona is also used to take tests and exams, within series that are only accessible for participating students.

#+CAPTION: Student view of a course in Dodona, showing two series of six exercises in the learning path of course A.
#+CAPTION: Each series has its own deadline.
#+CAPTION: The status column shows a global status for each exercise based on the last solution submitted.
#+CAPTION: The class progress column visualizes global status for each exercise for all students subscribed in the course.
#+CAPTION: Icons on the left show a global status for each exercise based on the last submission submitted before the series deadline.
#+NAME: fig:passfailstudentcourse
[[./images/passfailstudentcourse.png]]

Throughout an edition of a course, students can continuously submit solutions for programming exercises and immediately receive feedback upon each submission, even during tests and exams.
This rich feedback is automatically generated by an online judge and unit tests linked to each exercise\nbsp{}[cite:@wasikSurveyOnlineJudge2018].
Guided by that feedback, students can track potential errors in their code, remedy them and submit an updated solution.
There is no restriction on the number of solutions that can be submitted per exercise, and students can continue to submit solutions after a series deadline.
All submitted solutions are stored, but only the last submission before the deadline is taken into account to determine the status (and grade) of an exercise for a student.
One of the effects of active learning, triggered by exercises with deadlines and automated feedback, is that most learning happens during the semester as can be seen on the heatmap in Figure\nbsp{}[[fig:passfailheatmap]].

#+CAPTION: Heatmap showing the distribution per day of all 176\thinsp{}535 solutions submitted during the 2018--2019 edition of course A.
#+CAPTION: The darker the colour, the more submissions were made on that day.
#+CAPTION: A lighter red means there are few submissions on that day.
#+CAPTION: A light grey square means that no submissions were made that day.
#+CAPTION: Weekly lab sessions for different groups were organized on Monday afternoon, Friday morning and Friday afternoon.
#+CAPTION: Weekly deadlines for mandatory exercises were on Tuesdays at 22:00.
#+CAPTION: There were four exam sessions for different groups in January.
#+CAPTION: There is little activity in the exam periods, except for days on which there was an exam.
#+CAPTION: The course is not taught in the second semester, so there is very little activity there
#+CAPTION: Two exam sessions were organized in August/September granting an extra chance to students who failed on their exam in January/February.
#+NAME: fig:passfailheatmap
[[./images/passfailheatmap.png]]

*** Submission data
:PROPERTIES:
:CREATED: [2023-10-23 Mon 16:38]
:CUSTOM_ID: subsec:passfaildata
:END:

We exported data from Dodona on all solutions submitted by students during each course edition included in the study.
Each solution has a submission timestamp with precision down to the second and is linked to a course edition, series in the learning path, exercise and student.
We did not use the actual source code submitted by students, but did use the status describing the global assessment made by the learning environment: correct, wrong, compilation error, runtime error, time limit exceeded, memory limit exceeded, or output limit exceeded.

Comparison of student behaviour between different editions of the same course is enabled by computing snapshots for each edition at series deadlines.
Because course editions follow the same structure, we can align their series and compare snapshots for corresponding series.
Corresponding snapshots represent student performance at intermediate points during the semester and their chronology also allows longitudinal analysis within the semester.
Course A has snapshots for the five series of the first unit (labelled S1--S5), a snapshot for the evaluation of the first unit (labelled E1), snapshots for the five series of the second unit (labelled S6--S10), a snapshot for the evaluation of the second unit (labelled E2) and a snapshot for the exam (labelled E3).
Course B has snapshots for the first ten lab sessions (labelled S1--S10), a snapshot for the first evaluation (labelled E1), snapshots for the next series of seven lab sessions (labelled S11--S17), a snapshot for the second evaluation (labelled E2), snapshots for the last three lab sessions (S18--S20) and a snapshot for the exam (labelled E3).

It is important to stress that a snapshot of a course edition measures student performance only using the information available at the time of the snapshot.
As a result, the snapshot does not take into account submissions after its timestamp.
The behaviour of a student can then be expressed as a set of features extracted from the raw submission data.
We identified different types of features (see Appendix\nbsp{}[[#chap:featuretypes]]) that indirectly quantify certain behavioural aspects of students practising their programming skills.
When and how long do students work on their exercises?
Can students correctly solve an exercise and how much feedback do they need to accomplish this?
What kinds of mistakes do students make while solving programming exercises?
Do students further optimize the quality of their solution after it passes all unit tests, based on automated feedback or publication of sample solutions?
Note that there is no one-on-one relationship between these behavioural aspects and feature types.
Some aspects will be covered by multiple feature types, and some feature types incorporate multiple behavioural aspects.
We will therefore need to take into account possible dependencies between feature types while making predictions.

A feature type essentially makes one observation per student per series.
Each feature type thus results in multiple features: one for each series in the course (excluding series for evaluations and exams).
In addition, the snapshot also contains a feature for the average of each feature type across all series.
We do not use observations per individual exercise, as the actual exercises might differ between course editions.
Snapshots taken at the deadline of an evaluation or later, also contain the score a student obtained for the evaluation.
These features of the snapshot can be used to predict whether a student will finally pass/fail the course.
In addition, the snapshot also contains a label indicating whether the student passed or failed that is used during training and testing of classification algorithms.
Students that did not take part in the final examination, automatically fail the course.

Since course B has no hard deadlines, we left out deadline-related features from its snapshots (=first_dl=, =last_dl= and =nr_dl=; see Appendix\nbsp{}[[#chap:featuretypes]]).
To investigate the impact of deadline-related features, we also made predictions for course A that ignore these features.

*** Classification algorithms
:PROPERTIES:
:CREATED: [2023-10-23 Mon 16:45]
:CUSTOM_ID: subsec:passfailclassification
:END:

We evaluated four classification algorithms to make pass/fail predictions from student behaviour: stochastic gradient descent\nbsp{}[cite:@fergusonInconsistentMaximumLikelihood1982], logistic regression\nbsp{}[cite:@kleinbaumIntroductionLogisticRegression1994], support vector machines\nbsp{}[cite:@cortesSupportVectorNetworks1995], and random forests\nbsp{}[cite:@svetnikRandomForestClassification2003].
We used implementations of these algorithms from =scikit-learn=\nbsp{}[cite:@pedregosaScikitlearnMachineLearning2011] and optimized model parameters for each algorithm by cross-validated grid-search over a parameter grid.

Readers unfamiliar with machine learning can think of these specific algorithms as black boxes, but we briefly explain the basic principles of classification for their understanding.
Supervised learning algorithms use a dataset that contains both inputs and desired outputs to build a model that can be used to predict the output associated with new inputs.
The dataset used to build the model is called the training set and consists of training examples, with each example represented as an array of input values (feature vector).
Classification is a specific case of supervised learning where the outputs are restricted to a limited set of values (labels), in contrast to for example all possible numerical values with a range.
Classification algorithms are validated by splitting a dataset of labelled feature vectors into a training set and a test set, building a model from the training set, and evaluating the accuracy of its predictions on the test set.
Keeping training and test data separate is crucial to avoid bias during validation.
A standard method to make unbiased predictions for all examples in a dataset is \(k\)-fold cross-validation: partition the dataset in \(k\) subsets and then perform \(k\) experiments that each take one subset for evaluation and the other \(k-1\) subsets for training the model.

Pass/fail prediction is a binary classification problem with two possible outputs: passing or failing a course.
We evaluated the accuracy of the predictions for each snapshot and each classification algorithm with three different types of training sets.
As we have data from three editions of each course, the largest possible training set to make predictions for the snapshot of a course edition combines the corresponding snapshots from the two remaining course editions.
We also made predictions for a snapshot using each of its corresponding snapshots as individual training sets to see if we can still make accurate predictions based on data from only one other course edition.
Finally, we also made predictions for a snapshot using 5-fold cross-validation to compare the quality of predictions based on data from the same or another cohort of students.
Note that the latter strategy is not applicable to make predictions in practice, because we will not have pass/fail results as training labels while taking snapshots during the semester.
In practice, to make predictions for a snapshot, we can rely only on corresponding snapshots from previous course editions.
However, because we can assume that different editions of the same course yield independent data, we also used snapshots from future course editions in our experiments.

There are many metrics that can be used to evaluate how accurately a classifier predicted which students will pass or fail the course from the data in a given snapshot.
Predicting a student will pass the course is called a positive prediction, and predicting they will fail the course is called a negative prediction.
Predictions that correspond with the actual outcome are called true predictions, and predictions that differ from the actual outcome are called false predictions.
This results in four possible combinations of predictions: true positives (\(TP\)), true negatives (\(TN\)), false positives (\(FP\)) and false negatives (\(FN\)).
Two standard accuracy metrics used in information retrieval are precision (\(TP/(TP+FP)\)) and recall (\(TP/(TP+FN)\)).
The latter is also called sensitivity if used in combination with specificity (\(TN/(TN+FP)\)).

Many studies for pass/fail prediction use accuracy (\((TP+TN)/(TP+TN+FP+FN)\)) as a single performance metric.
However, this can yield misleading results.
For example, let's take a dummy classifier that always "predicts" students will pass, no matter what.
This is clearly a bad classifier, but it will nonetheless have an accuracy of 75% for a course where 75% of the students pass.

In our study, we will therefore use two more complex metrics that take these effects into account: balanced accuracy and F_1-score.
Balanced accuracy is the average of sensitivity and specificity.
The F_1-score is the harmonic mean of precision and recall.
If we go back to our example, the optimistic classifier that consistently predicts that all students will pass the course and thus fails to identify any failing student will have a balanced accuracy of 50% and an F_1-score of 75%.
Under the same circumstances, a pessimistic classifier that consistently predicts that all students will fail the course has a balanced accuracy of 50% and an F_1-score of 0%.

*** Pass/fail predictions
:PROPERTIES:
:CREATED:  [2024-01-22 Mon 17:17]
:CUSTOM_ID: subsec:passfailmaterialspredictions
:END:

In summary, Figure\nbsp{}[[fig:passfailmethodoverview]] outlines the entire flow of the proposed pass/fail prediction framework.
It starts by extracting metadata for all submissions students made so far within a course (timestamp, status, student, exercise, series) and collecting their marks on intermediate tests and final exams (step 1).
In practice, applying the framework on a student cohort in the current course edition only requires submission metadata and pass/fail outcomes from student cohorts in previous course editions.
Successive course editions are then aligned by identifying fixed time points throughout the course where predictions are made, for example at submission deadlines, intermediate tests or final exams (step 2).
We conducted a longitudinal study to evaluate the accuracy of pass/fail predictions at successive stages of a course (step 3).
This is done by extracting features from the raw submission metadata of one or more course editions and training machine learning models that can identify at-risk students during other course editions.
Our scripts that implement this framework are provided as supplementary material.
Teachers can also interpret the behaviour of students in their class by analysing the feature weights of the machine learning models (step 4).

** Results and discussion
:PROPERTIES:
:CREATED: [2023-10-23 Mon 16:55]
:CUSTOM_ID: sec:passfailresults
:END:

We evaluated the performance of four classification algorithms for pass/fail predictions in a longitudinal sequence of snapshots from course A and B: stochastic gradient descent (Figure\nbsp{}[[fig:passfailsgdresults]]), logistic regression (Figure\nbsp{}[[fig:passfaillrresults]]), support vector machines (Figure\nbsp{}[[fig:passfailsvmresults]]), and random forests (Figure\nbsp{}[[fig:passfailrfresults]]).
For each classifier, course and snapshot, we evaluated 12 predictions for the following combinations of training and test sets: train on one edition and test on another edition; train on two editions and test on the other edition; train and test on one edition using 5-fold cross validation.
In addition, we made predictions for course A using both the full set of features and a reduced feature set that ignores deadline-related features.
We discuss the results in terms of accuracy, potential for early detection, and interpretability.

#+CAPTION: Performance of stochastic gradient descent classifiers for pass/fail predictions in a longitudinal sequence of snapshots from courses A (all features and reduced set of features) and B, measured by balanced accuracy and F_1-score.
#+CAPTION: Dots represent performance of a single prediction, with 12 predictions for each group of corresponding snapshots (columns).
#+CAPTION: Solid line connects averages of the performances for each group of corresponding snapshots.
#+NAME: fig:passfailsgdresults
[[./images/passfailsgdresults.png]]

#+CAPTION: Performance of logistic regression classifiers for pass/fail predictions in a longitudinal sequence of snapshots from courses A (all features and reduced set of features) and B, measured by balanced accuracy and F_1-score.
#+CAPTION: Dots represent performance of a single prediction, with 12 predictions for each group of corresponding snapshots (columns).
#+CAPTION: Solid line connects averages of the performances for each group of corresponding snapshots.
#+NAME: fig:passfaillrresults
[[./images/passfaillrresults.png]]

#+CAPTION: Performance of support vector machine classifiers for pass/fail predictions in a longitudinal sequence of snapshots from courses A (all features and reduced set of features) and B, measured by balanced accuracy and F_1-score.
#+CAPTION: Dots represent performance of a single prediction, with 12 predictions for each group of corresponding snapshots (columns).
#+CAPTION: Solid line connects averages of the performances for each group of corresponding snapshots.
#+NAME: fig:passfailsvmresults
[[./images/passfailsvmresults.png]]

#+CAPTION: Performance of random forest classifiers for pass/fail predictions in a longitudinal sequence of snapshots from courses A (all features and reduced set of features) and B, measured by balanced accuracy and F_1-score.
#+CAPTION: Dots represent performance of a single prediction, with 12 predictions for each group of corresponding snapshots (columns).
#+CAPTION: Solid line connects averages of the performances for each group of corresponding snapshots.
#+NAME: fig:passfailrfresults
[[./images/passfailrfresults.png]]

*** Accuracy
:PROPERTIES:
:CREATED: [2023-10-23 Mon 17:03]
:CUSTOM_ID: subsec:passfailaccuracy
:END:

The overall conclusion from the longitudinal analysis is that indirectly measuring how students practice their coding skills by solving programming exercises (formative assessments) in combination with directly measuring how they perform on intermediate evaluations (summative assessments), allows us to predict with high accuracy if students will pass or fail a programming course.
The signals to make such predictions seem to be present in the data, as we come to the same conclusions irrespective of the course, classification algorithm, or performance metric evaluated in our study.
Overall, logistic regression was the best performing classifier for both courses, but the difference compared to the other classifiers is small.

When we compare the longitudinal trends of balanced accuracy for the predictions of both courses, we see that course A starts with a lower balanced accuracy at the first snapshot, but its accuracy increases faster and is slightly higher at the end of the semester.
At the start of the semester at snapshot S1, course A has an average balanced accuracy between 60% and 65% and course B around 70%.
Nearly halfway through the semester, before the first evaluation, we see an average balanced accuracy around 70% for course A at snapshot S5 and between 70% and 75% for course B at snapshot S8.
After the first evaluation, we can make predictions with a balanced accuracy between 75% and 80% for both courses.
The predictions for course B stay within this range for the rest of the semester, but for course A we can consistently make predictions with an average balanced accuracy of 80% near the end of the semester.

Compared to the accuracy results of\nbsp{}[cite/t:@kovacicPredictingStudentSuccess2012], we see a 15-20% increase for our balanced accuracy results.
Our balanced accuracy results are similar to the accuracy results of\nbsp{}[cite/t:@livierisPredictingSecondarySchool2019], who used semi-supervised machine learning.
[cite/t:@asifAnalyzingUndergraduateStudents2017] achieve an accuracy of about 80% when using one cohort of training and another cohort for testing, which is again similar to our balanced accuracy results.
All of these studies used prior academic history as the basis for their methods, which we do not use in our framework.
We also see similar results as compared to\nbsp{}[cite/t:@vihavainenPredictingStudentsPerformance2013] where we don't have to rely on data collection that interferes with the learning process.
Note that we are comparing the basic accuracy results of prior studies with the more reliable balanced accuracy results of our framework.

F_1-scores follow the same trend as balanced accuracy, but the inclination is even more pronounced because it starts lower and ends higher.
It shows another sharp improvement of predictive performance for both courses when students practice their programming skills in preparation of the final exam (snapshot E3).
This underscores the need to keep organizing final summative assessments as catalysts of learning, even for courses with a strong focus on active learning.

The variation in predictive accuracy for a group of corresponding snapshots is higher for course A than for course B.
This might be explained by the fact that successive editions of course B use the same set of exercises, supplemented with evaluation and exam exercises from the previous edition, whereas each edition of course A uses a different selection of exercises.

Predictions made with training sets from the same student cohort (5-fold cross-validation) perform better than those with training sets from different cohorts (see supplementary material for details).
This is more pronounced for F_1-scores than for balanced accuracy, but the differences are small enough so that nothing prevents us from building classification models with historical data from previous student cohorts to make pass/fail predictions for the current cohort, which is something that can't be done in practice with data from the same cohort as pass/fail information is needed during the training phase.
In addition, we found no significant performance differences for classification models using data from a single course edition or combining data from two course editions.
Given that cohort sizes are large enough, this tells us that accurate predictions can already be made in practice with historical data from a single course edition.
This is also relevant when the structure of a course changes, because we can only make predictions from historical data for course editions whose snapshots align.

The need to align snapshots is also the reason why we had to build separate models for courses A and B since both have differences in course structure.
The models, however, were built using the same set of feature types.
Because course B does not work with hard deadlines, deadline-related feature types could not be computed for its snapshots.
This missing data and associated features had no impact on the performance of the predictions.
Deliberately dropping the same feature types for course A also had no significant effect on the performance of predictions, illustrating that the training phase is where classification algorithms decide themselves how the individual features will contribute to the predictions.
This frees us from having to determine the importance of features beforehand, allows us to add new features that might contribute to predictions even if they correlate with other features, and makes it possible to investigate afterwards how important individual features are for a given classifier (see Section\nbsp{}[[#subsec:passfailinterpretability]]).

Even though the structure of the courses is quite different, our method achieves high accuracy results for both courses.
The results for course A with reduced features also still gives accurate results.
This indicates that the method should be generalizable to other courses where similar data can be collected, even if the structure is quite different or when some features are impossible to calculate due to the course structure.

*** Early detection
:PROPERTIES:
:CREATED: [2023-10-23 Mon 17:05]
:CUSTOM_ID: subsec:passfailearly
:END:

Accuracy of predictions systematically increases as we capture more of student behaviour during the semester.
But surprisingly we can already make quite accurate predictions early on in the semester, long before students take their first evaluation.
Because of the steady trend, predictions for course B at the start of the semester are already reliable enough to serve as input for student feedback or teacher interventions.
It takes some more time to identify at-risk students for course A, but from week four or five onwards the predictions may also become an instrument to design remedial actions for this course.
Hard deadlines and graded exercises are a strong enforcement of active learning behaviour on the students of course A, and might disguise somewhat more the motivation of students to work on their programming skills.
This might explain why it takes a bit longer to properly measure student motivation for course A than for course B.

*** Interpretability
:PROPERTIES:
:CREATED: [2023-10-23 Mon 17:05]
:CUSTOM_ID: subsec:passfailinterpretability
:END:

So far, we have considered classification models as black boxes in our longitudinal analysis of pass/fail predictions.
However, many machine learning techniques can tell us something about the contribution of individual features to make the predictions.
In the case of our pass/fail predictions, looking at the importance of feature types and linking them to aspects of practising programming skills, might give us insights into what kind of behaviour promotes or inhibits learning, or has no or a minor effect on the learning process.
Temporal information can tell us what behaviour makes a steady contribution to learning or where we see shifts throughout the semester.

This interpretability was a considerable factor in our choice of the classification algorithms we investigated in this study.
Since we identified logistic regression as the best-performing classifier, we will have a closer look at feature contributions in its models.
These models are explained by the feature weights in the logistic regression equation, so we will express the importance of a feature as its actual weight in the model.
We use a temperature scale when plotting importances: white for zero importance, a red gradient for positive importance values and a blue gradient for negative importance values.
A feature importance \(w\) can be interpreted as follows for logistic regression models: an increase of the feature value by one standard deviation increases the odds of passing the course by a factor of \(e^w\) when all other feature values remain the same\nbsp{}[cite:@molnarInterpretableMachineLearning2019].
The absolute value of the importance determines the impact the feature has on predictions.
Features with zero importance have no impact because \(e^0 = 1\).
Features represented with a light colour have a weak impact and features represented with a dark colour have a strong impact.
As a reference, an importance of 0.7 doubles the odds for passing the course with each standard deviation increase of the feature value, because \(e^{0.7} \sim 2\).
The sign of the importance determines whether the feature promotes or inhibits the odds of passing the course.
Features with a positive importance (red colour) will increase the odds with increasing feature values, and features with a negative importance (blue colour) will decrease the odds with increasing feature values.

To simulate that we want to make predictions for each course edition included in this study, we trained logistic regression models with data from the remaining two editions of the same course.
A label "edition 18--19" therefore means that we want to make predictions for the 2018--2019 edition of a course with a model built from the 2016--2017 and 2017--2018 editions of the course.
However, in this case we are not interested in the predictions themselves, but in the importance of the features in the models.
The importance of all features for each course edition can be found at https://github.com/dodona-edu/pass-fail-article.
We will restrict our discussion by highlighting the importance of a selection of feature types for the two courses.

For course A, we look into the evaluation scores (Figure\nbsp{}[[fig:passfailfeaturesAevaluation]]) and the feature types =correct_after_15m= (Figure\nbsp{}[[fig:passfailfeaturesAcorrect]]) and =wrong= (Figure\nbsp{}[[fig:passfailfeaturesAwrong]]).
Evaluation scores have a very strong impact on predictions, and high evaluation scores increase the odds of passing the course.
This comes as no surprise, as both the evaluations and exams are summative assessments that are organized and graded in the same way.
Although the difficulty of evaluation exercises is lower than those of exam exercises, evaluation scores already are good predictors for exam scores.
Also note that these features only show up in snapshots taken at or after the corresponding evaluation.
They have zero impact on predictions for earlier snapshots, as the information is not available at the time these snapshots are taken.

#+CAPTION: Importance of evaluation scores in the logistic regression models for course A (full feature set).
#+CAPTION: Reds mean that a growth in the feature value for a student increases the odds of passing the course for that student.
#+CAPTION: The darker the colour the larger this increase will be.
#+NAME: fig:passfailfeaturesAevaluation
[[./images/passfailfeaturesAevaluation.png]]

The second feature type we want to highlight is =correct_after_15m=: the number of exercises in a series where the first correct submission was made within fifteen minutes after the first submission (Figure\nbsp{}[[fig:passfailfeaturesAcorrect]]).
Note that we can't directly measure how long students work on an exercise, as they may write, run and test their solutions on their local machine before their first submission to Dodona.
Rather, this feature type measures how long it takes students to find and remedy errors in their code (debugging), after they start getting automatic feedback from Dodona.

For exercise series in the first unit of course A (series 1--5), we generally see that the features have a positive impact (red).
This means that students will more likely pass the course if they are able to quickly remedy errors in their solutions for these exercises.
The first and fourth series are an exception here.
The fact that students need more time for the first series might reflect that learning something new is hard at the beginning, even if the exercises are still relatively easy.
Series 4 of course A covers strings as the first compound data type of Python in combination with nested loops, where (non-nested) loops themselves are covered in series 3.
This complex combination might mean that students generally need more time to debug the exercises in series 4.

For the series of the second unit (series 6--10), we observe two different effects.
The impact of these features is zero for the first few snapshots (grey bottom left corner).
This is because the exercises from these series were not yet published at the time of those snapshots, where all series of the first unit were available from the start of the semester.
For the later snapshots, we generally see a negative (blue) weight associated with the features.
It might seem counterintuitive and contradicts the explanation given for the series of the first unit.
However, the exercises of the second unit are a lot more complex than those of the first unit.
This up to a point that even for good students it is hard to debug and correctly solve an exercise in only 15 minutes.
Students that need less than 15 minutes at this stage might be bypassing learning by copying solutions from fellow students instead of solving the exercises themselves.
An exception to this pattern are the few red squares forming a diagonal in the middle of the bottom half.
These squares correspond with exercises that are solved as soon as they become available as opposed to waiting for the deadline.
A possible explanation for these few slightly positive weights is that these exercises are solved by highly-motivated, top students.

#+CAPTION: Importance of feature type =correct_after_15m= (the number of exercises in a series where the first correct submission was made within fifteen minutes after the first submission) in logistic regression models for course A (full feature set).
#+CAPTION: Reds mean that a growth in the feature value for a student increases the odds of passing the course for that student.
#+CAPTION: The darker the colour the larger this increase will be.
#+CAPTION: Blues mean that a growth in the feature value decreases the odds.
#+CAPTION: The darker the colour the larger this decrease will be.
#+NAME: fig:passfailfeaturesAcorrect
[[./images/passfailfeaturesAcorrect.png]]

Finally, if we look at the feature type =wrong= (Figure\nbsp{}[[fig:passfailfeaturesAwrong]]), submitting a lot of submissions with logical errors mostly has a positive impact on the odds of passing the course.
This underscores the old adage that practice makes perfect, as real learning happens where students learn from their mistakes.
Exceptions to this rule are found for series 2, 3, 9 and 10.
The lecturer and teaching assistants identify the topics covered in series 2 and 9 by far as the easiest topics covered in the course, and identify the topics covered in series 3, 6 and 10 as the hardest.
However, it does not feel very intuitive that being stuck with logical exercises longer than other students either inhibits the odds for passing on topics that are extremely hard or easy or promotes the odds on topics with moderate difficulty.
This shows that interpreting the importance of feature types is not always straightforward.

#+CAPTION: Importance of feature type =wrong= (the number of wrong submissions in a series) in logistic regression models for course A (full feature set).
#+CAPTION: Reds mean that a growth in the feature value for a student increases the odds of passing the course for that student.
#+CAPTION: The darker the colour the larger this increase will be.
#+CAPTION: Blues mean that a growth in the feature value decreases the odds.
#+CAPTION: The darker the colour the larger this decrease will be.
#+NAME: fig:passfailfeaturesAwrong
[[./images/passfailfeaturesAwrong.png]]

For course B, we look into the evaluation scores (Figure\nbsp{}[[fig:passfailfeaturesBevaluation]]) and the feature types =comp_error= (Figure\nbsp{}[[fig:passfailfeaturesBcomp]]) and =wrong= (Figure\nbsp{}[[fig:passfailfeaturesBwrong]]).
The importance of evaluation scores is similar as for course A, although their relative impact on the predictions is slightly lower.
The latter might be caused by automatic grading of evaluation exercises, where exam exercises are graded by hand.
The fact that the second evaluation is scheduled a little bit earlier in the semester than for course A, makes that pass/fail predictions for course B can also rely earlier on this important feature.
However, we do not see a similar increase of the global performance metrics around the second evaluation of course B, as we see for the first evaluation.

#+CAPTION: Importance of evaluation scores in the logistic regression models for course B.
#+CAPTION: Reds mean that a growth in the feature value for a student increases the odds of passing the course for that student.
#+CAPTION: The darker the colour the larger this increase will be.
#+NAME: fig:passfailfeaturesBevaluation
[[./images/passfailfeaturesBevaluation.png]]

Learning to code requires mastering two major competences:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{, }}, itemjoin*={{, and }}]
- getting familiar with the syntax rules of a programming language
  to express the steps for solving a problem in a formal way, so that
  the algorithm can be executed by a computer
- problem-solving itself.
As a result, we can make a distinction between different kinds of errors in source code.
Compilation errors are mistakes against the syntax of the programming language, whereas logical errors result from solving a problem with a wrong algorithm.
When comparing the importance of the number of compilation (Figure\nbsp{}[[fig:passfailfeaturesBcomp]]) and logical errors (Figure\nbsp{}[[fig:passfailfeaturesBwrong]]) students make while practising their coding skills, we see a clear difference.
Making a lot of compilation errors has a negative impact on the odds for passing the course (blue colour dominates in Figure\nbsp{}[[fig:passfailfeaturesBcomp]]), whereas making a lot of logical errors makes a positive contribution (red colour dominates in Figure\nbsp{}[[fig:passfailfeaturesBwrong]]).
This aligns with the claim of\nbsp{}[cite/t:@edwardsSeparationSyntaxProblem2018] that problem-solving is a higher-order learning task in the Taxonomy by\nbsp{}[cite/t:@bloom1956handbook] (analysis and synthesis) than language syntax (knowledge, comprehension, and application).
Students that get stuck longer in the mechanics of a programming language will more likely fail the course, whereas students that make a lot of logical errors and properly learn from them will more likely pass the course.
So making mistakes is beneficial for learning, but it depends on what kind of mistakes.
We also looked at the number of solutions with logical errors while interpreting feature types for course A.
Although we hinted there towards the same conclusions as for course B, the signals were less consistent.
This shows that interpreting feature importances always needs to take the educational context into account.
This can also be seen in Figure\nbsp{}[[fig:passfailfeaturesAcorrect]], where some weeks contribute positively and some negatively.
The reasons for these differences depend on the content of the course, which requires knowledge of the course contents to interpret correctly.

#+CAPTION: Importance of feature type =comp_error= (the number of submissions with compilation errors in a series) in logistic regression models for course B.
#+CAPTION: Reds mean that a growth in the feature value for a student increases the odds of passing the course for that student.
#+CAPTION: The darker the colour the larger this increase will be.
#+CAPTION: Blues mean that a growth in the feature value decreases the odds.
#+CAPTION: The darker the colour the larger this decrease will be.
#+NAME: fig:passfailfeaturesBcomp
[[./images/passfailfeaturesBcomp.png]]

#+CAPTION: Importance of feature type =wrong= (the number of wrong submissions in a series) in logistic regression models for course B.
#+CAPTION: Reds mean that a growth in the feature value for a student increases the odds of passing the course for that student.
#+CAPTION: The darker the colour the larger this increase will be.
#+CAPTION: Blues mean that a growth in the feature value decreases the odds.
#+CAPTION: The darker the colour the larger this decrease will be.
#+NAME: fig:passfailfeaturesBwrong
[[./images/passfailfeaturesBwrong.png]]

** Replication study at Jyväskylä University in Finland
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:50]
:CUSTOM_ID: sec:passfailfinland
:END:

After our original study, we collaborated with researchers from Jyväskylä University (JYU) in Finland on replicating the study in their introductory programming course\nbsp{}[cite:@zhidkikhReproducingPredictiveLearning2024].
There are however, some notable differences to the study performed at Ghent University.
In the new study, self-reported data was added to the model to test whether this enhances its predictions.
Also, the focus shifted from pass/fail prediction to dropout prediction.
This happened because of the different way the course at JYU is taught.
By performing well enough in all weekly exercises and a project, students can already receive a passing grade.
This is impossible in the courses at Ghent University, where most of the final marks are earned at the exam at the end of the semester.

Another important difference in the two studies is the data that was available to feed into the machine learning model.
Dodona keeps rich data about the evaluation results of a student's submission.
In TIM (the learning environment used at JYU), only a score is kept for each submission.
This score represents the underlying evaluation results (compilation error/mistakes in the output/...).
While it is possible to reverse engineer the score into some underlying status, for some statuses that Dodona can make a distinction between this is not possible with TIM.
This means that a different set of features had to be used in the study at JYU than the feature set used in the study at Ghent University.
The specific feature types left out of the study at JYU are =comp_error= and =runtime_error=.

The course at JYU had been taught in the same way since 2015, resulting in behavioural and survey data from 2\thinsp{}615 students from the 2015--2021 academic years.
The snapshots were made weekly as well, since the course also works with weekly assignments and deadlines.
The self-reported data consists of pre-course and midterm surveys that inquire about aptitudes towards learning programming and motivation, including expectation about grades, prior programming experience, study year, attendance and amount of concurrent courses.

In the analysis, the same four classifiers as the original study were tested.
In addition to this, the dropout analysis was done for three datasets:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{, }}, itemjoin*={{, and }}]
- behavioural data only
- self-reported data only
- combined behavioural and self-reported data.

The results obtained in the study at JYU are very similar to the results obtained at Ghent University.
Again, logistic regression was found to yield the best and most stable results.
Even though no data about midterm evaluations or examinations was used (since this data was not available) a similar jump in accuracy around the midterm of the course was also observed.
The jump in accuracy can be explained here by the fact that the period around the middle of the term is when most students drop out.
It was also observed that the first weeks of the course play an important role in reducing dropout.

The addition of the self-reported data to the snapshots resulted in a statistically significant improvement of predictions in the first four weeks of the course.
For the remaining weeks, the change in predication performance was not statistically significant.
This again points to the conclusion that the first few weeks of a CS1 course play a significant role in student success.
The models trained only on self-reported data performed significantly worse than the other models.

The replication done at JYU showed that our prediction strategy can be used in significantly different educational contexts.
Of course adaptations to the method have to be made sometimes given differences in course structure and learning environment used, but these adaptations do not result in different prediction results.

** Conclusions and future work
:PROPERTIES:
:CREATED: [2023-10-23 Mon 17:30]
:CUSTOM_ID: sec:passfailconclusions
:END:

In this chapter, we presented a classification framework for predicting if students will likely pass or fail introductory programming courses.
The framework already yields high-accuracy predictions early on in the semester and is privacy-friendly because it only works with metadata from programming challenges solved by students while working on their programming skills.
Being able to identify at-risk students early on in the semester opens windows for remedial actions to improve the overall success rate of students.

We validated the framework by building separate classifiers for three courses because of differences in course structure, institute and learning platform, but using similar sets of features for training models.
The results showed that submission metadata from previous student cohorts can be used to make predictions about the current cohort of students, even if course editions use different sets of exercises, or the courses are structured differently.
Making predictions requires aligning snapshots between successive editions of a course, where students have the same expected progress at corresponding snapshots.
Historical metadata from a single course edition suffices if group sizes are large enough.
Different classification algorithms can be plugged into the framework, but logistic regression resulted in the best-performing classifiers.

Apart from their application to make pass/fail predictions, an interesting side effect of classification models that map indirect measurements of learning behaviour onto mastery of programming skills is that they allow us to interpret what behavioural aspects contribute to learning to code.
Visualization of feature importance turned out to be a useful instrument for linking individual feature types with student behaviour that promotes or inhibits learning.
We applied this interpretability to some important feature types that popped up for the three courses included in this study.

Our study has several strengths and promising implications for future practice and research.
First, we were able to predict success based on historical metadata from earlier cohorts, and we are already able to do that early on in the semester.
In addition to that, the accuracy of our predictions is similar to those of earlier efforts\nbsp{}[cite:@asifAnalyzingUndergraduateStudents2017; @vihavainenPredictingStudentsPerformance2013; @kovacicPredictingStudentSuccess2012] while we are not using prior academic history or interfering with the students' usual learning workflows.
However, there are also some limitations and work for the future.
While our visualizations of the features (Figures\nbsp{}[[fig:passfailfeaturesAevaluation]]\nbsp{}through\nbsp{}[[fig:passfailfeaturesBwrong]]) are helpful to indicate which features are important at which stage of the course in view of increasing versus decreasing the odds of passing the course, they may not be oversimplified and need to be carefully interpreted and placed into context.
This is where the expertise and experience of teachers comes in.
These visualizations can be interpreted by teachers and further contextualized towards the specific course objectives.
For example, teachers know the content and goals of every series of exercises, and they can use the information presented in our visualizations in order to investigate why certain series of exercises are more or less important in view of passing the course.
In addition, they may use the information to further redesign their course

We can thus conclude that the proposed framework achieves the objectives set for accuracy, early prediction and interpretability.
Having this new framework at hand immediately raises some follow-up research questions that urge for further exploration:
#+ATTR_LATEX: :environment enumerate*
#+ATTR_LATEX: :options [label={\emph{\roman*)}}, itemjoin={{ }}, itemjoin*={{ }}]
- Do we inform students about their odds of passing a course?
  How and when do we inform students about their performance in an educationally responsible way?
  What learning analytics do we use to present predictions to students, and do we only show results or also explain how the data led to the results?
  How do students react to the announcement of their chance at passing the course?
  How do we ensure that students are not demotivated?
- What actions could teachers take upon early insights which students will likely fail the course?
  What recommendations could they make to increase the odds that more students will pass the course?
  How could interpretations of important behavioural features be translated into learning analytics that give teachers more insight into how students learn to code?
- Can we combine student progress (what programming skills does a student already have and at what level of mastery), student preferences (which skills does a student want to improve on), and intrinsic properties of programming exercises (what skills are needed to solve an exercise and how difficult is it) into dynamic learning paths that recommend exercises to optimize the learning effect for individual students?

* Manual feedback
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:51]
:CUSTOM_ID: chap:feedback
:END:

This chapter discusses the history of manual feedback in the programming course taught at the faculty of Sciences at Ghent University and how it informed the development of evaluation and grading features within Dodona.
We will then expand on some further experiments using data mining techniques we did to try to further reduce the time spent adding manual feedback.

** Assessment on paper
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:04]
:CUSTOM_ID: sec:feedbackpaper
:END:

Since the academic year 2015--2016 the programming course has started taking two open-book/open-internet evaluations in addition to the regular exam.
The first is a midterm and the other happens at the end of the semester (but before the exam period).
The organization of these evaluations has been a learning process for everyone involved.
Although the basic idea has remained the same (solve two Python programming exercises in two hours, or three in 3.5 hours for the exam), almost every aspect surrounding this basic premise has changed.

To be able to give feedback, student solutions were printed at the end of the evaluation.
At first this happened by going around with a USB stick that students had to copy their solution to, later by using a submission platform developed at Ghent University (Indianio) that had support for printing to printers in the evaluation rooms.
Printing support in Indianio was added specifically for this course, in fact.
Students were then allowed to check their printed solutions to make sure that the correct code was graded.
This however means that the end of an evaluation takes a lot of time, since printing all these papers is a slow and badly parallelizable process (not the mention the environmental impact!).

It also has some important drawbacks while grading.
Even though Dodona was not yet in use at this point, SPOJ was used to generate automated feedback on correctness.
This automated feedback was not available when assessing a student's source code on paper.
It therefore takes either more mental energy to work out whether the student's code would behave correctly with all inputs or it takes some hassle to look up a student's automated assessment results every time.
Another important drawback is that students have a much harder time seeing their feedback.
While their numerical grades were posted online or emailed to them, to see the comments graders wrote alongside their code they had to come to a hands-on session and ask the assistant there to be able to view the annotated version of their code.
Very few students did so.
There are a few possible explanations for this.
They might experience social barriers for asking feedback on an evaluation they performed poorly on.
For students who performed well, it might not be worth the hassle of going to ask about feedback.
But maybe more importantly, a vicious cycle started to appear: because few students look at their feedback, graders did not spend much effort in writing out clear and useful feedback.
Code that was too complex or plain wrong usually received little more than a strikethrough, instead of an explanation on why the student's method did not work.

** Adding comments via Dodona
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:32]
:CUSTOM_ID: sec:feedbackcomments
:END:

Seeing the amount of hassle that assessing these evaluations brought with them, we decided to build support for manual feedback and grading into Dodona.
The first step of this was the functionality of adding comments to code.
This work was started in the academic year 2019--2020, so the onset of the COVID-19 pandemic brought a lot of momentum to this work.
Suddenly, the idea of printing student submissions became impossible, since the evaluations had to be taken by students in their own homes and the graders were working from home as well.
Graders could now add comments to a student's code which would allow the student to view the feedback from their own home as well.
An example of such a comment can be seen on Figure\nbsp{}[[fig:feedbackfirstcomment]]
There were still a few drawbacks to this system for assessing and grading though:
- Knowing which submissions to grade was not always trivial.
  For most students, the existing deadline system worked, since the solution they submitted right before the deadline was the submission taken into account when grading.
  There are however also students who receive extra time based on a special status granted to them by Ghent University (due to e.g. a learning disability).
  For these students, graders had to manually search for the submission made right before the extended deadline these students receive.
  This means that students could not be graded anonymously.
  It also makes the process a lot more error-prone.
- Comment visibility could not yet be time-gated towards students.
  This meant that graders had to write their comments in a local file with some extra metadata about the assessment.
  Afterwards this local file could be processed using some home-grown scripts to automatically add all comments at (nearly) the same time.
It is obvious that this was not a great user experience, and not something we could roll out more widely outside of Dodona developers that were also involved with teaching.

We could already do some anecdotal analysis of this new system.
One first observation that might seem counterintuitive is that graders did not feel like they spent less time grading.
If anything, they reported spending more time grading.
Another observation however is that graders gave more feedback and felt that the feedback they gave was of higher quality than before.
In the first trial of this system, the feedback was viewed by over 80% of students within 24 hours, which is something that we had never observed when grading on paper.

** Evaluations
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:32]
:CUSTOM_ID: sec:feedbackevaluations
:END:

To streamline and automate the process of grading even more, the concept of an evaluation was added to Dodona.
Evaluations address the two drawbacks identified above:
- Comments made within an evaluation are linked to this evaluation.
  They are only made visible to students once the feedback of the evaluation is released.
- They also add an overview of the submissions that need to receive feedback.
  Since the submissions are explicitly linked to the evaluation, changing the submissions for students who receive extra time is also a lot less error-prone, since it can be done before actually starting out with the assessment.
  Evaluations also have specific UI to do this, where the timestamps are shown to teachers as accurately as Dodona saves them.
The addition of evaluations resulted in a subjective feeling of time being saved by the graders, at least in comparison with the previous system of adding comments.
There is still one main drawback though, in the fact that student scores still had to be entered outside of Dodona.
This is again more error-prone, since this involves manually looking up the correct student and entering their scores in a global spreadsheet.
It is also less transparent towards students.
While rubrics were made for every exercise that had to be graded, every grader had their preferred way of aggregating and entering these scores.
This means that even though the rubrics exist, students had no option of seeing the different marks they received for different rubrics.

#+CAPTION: The first comment ever left on Dodona as part of a grading session.
#+NAME: fig:feedbackfirstcomment
[[./images/feedbackfirstcomment.png]]

To address this concern, another feature was implemented in Dodona.
We added rubrics and a user-friendly way of entering scores.
This means that students can view the scores they received for each rubric, and can do so right next to the feedback that was added manually.

** Feedback re-use
:PROPERTIES:
:CREATED: [2023-11-20 Mon 17:39]
:CUSTOM_ID: sec:feedbackreuse
:END:

Grading and giving feedback has always been a time-consuming process, and the move to digital grading did not improve this compared to grading on paper.
Even though the process itself was optimized, this optimization was used by graders to write out more and more comprehensive feedback.

Since evaluations are done with a few exercises solved by lots of students, there are usually a lot of mistakes that are common to a lot of students.
This leads to graders giving the same feedback a lot of times.
In fact, most graders maintained a list of commonly given feedback in a separate program or document.

We implemented the concept of feedback re-use to streamline giving commonly re-used feedback.
When giving feedback, the grader has the option to save the annotation they are currently writing.
When they later encounter a situation where they want to give that same feedback, the only thing they have to do is write a few letters of the annotation in the saved annotation search box, and they can quickly insert the text written earlier.
While originally conceptualized mainly for the benefit of graders, students can actually benefit from this feature as well.
Graders only need to write out a detailed and clear message once and can then re-use that message over a lot of submissions instead of writing a shorter message each time.

** Feedback prediction
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:04]
:CUSTOM_ID: sec:feedbackprediction
:END:

Given that we now have a system for re-using earlier feedback, we can ask ourselves if we can do this in a smarter way.
Instead of teachers having to search for the annotation they want to use, what if we could predict which annotation they want to use?
This is exactly what we will explore in this section, which is based on an article that is currently being prepared for submission.

*** Introduction
:PROPERTIES:
:CREATED:  [2024-01-19 Fri 15:47]
:CUSTOM_ID: subsec:feedbackpredictionintro
:END:

Feedback is a key factor in student learning\nbsp{}[cite:@hattiePowerFeedback2007; @blackAssessmentClassroomLearning1998].
In programming education, many steps have been taken to automate feedback using automated assessment systems\nbsp{}[cite:@paivaAutomatedAssessmentComputer2022; @ihantolaReviewRecentSystems2010; @ala-mutkaSurveyAutomatedAssessment2005].
These automated assessment systems provide feedback on correctness, and can provide some feedback on style and best practices by using linters.
They are, however, generally not able to provide the same high-level feedback on program design that a seasoned programmer can give.
In many educational practices, automated assessment is therefore supplemented with manual feedback, especially when grading evaluations or exams\nbsp{}[cite:@debuseEducatorsPerceptionsAutomated2008].
This requires a large time investment from teachers\nbsp{}[cite:@tuckFeedbackgivingSocialPractice2012].

Consequently, numerous researchers have explored the enhancement of feedback mechanisms through AI.
[cite/t:@vittoriniAIBasedSystemFormative2021] automated grading using natural language processing, and found that students who used this system during the semester were more likely to pass the course at the end of the semester.
[cite/t:@leeSupportingStudentsGeneration2023] has used supervised learning with ensemble learning to enable students to conduct peer and self-evaluation.
Furthermore,\nbsp{}[cite/t:@berniusMachineLearningBased2022] introduced a framework based on clustering text segments in textual exercises to reduce the grading workload.

The context of our work is in our own assessment system, called Dodona, developed at Ghent University\nbsp{}[cite:@vanpetegemDodonaLearnCode2023].
Dodona provides automated feedback on every submission, but also allows teachers to give manual feedback on student sumbmissions and assign scores to them, from within the platform.
In 2023, 3\thinsp{}663\thinsp{}749 submissions were made on our platform, of which 44\thinsp{}012 were manually assessed.
During those assessments, 22\thinsp{}888 annotations were added.
The process of giving feedback on a programming assignment in Dodona is very similar to a code review, where mistakes or suggestions for improvements are annotated at the relevant line(s), as can be seen on Figure\nbsp{}[[fig:feedbackintroductionreview]].

#+CAPTION: Manual assessment of a submission: a teacher gave feedback on the code by adding inline annotations and is grading the submission by filling up the scoring rubric.
#+NAME: fig:feedbackintroductionreview
[[./images/feedbackintroductionreview.png]]

However, there exists a crucial distinction between traditional code reviews and those in an educational context: instructors often provide feedback on numerous solutions to the same assignment.
Given that students frequently commit similar errors, it logically follows that instructors repeatedly deliver the same feedback across multiple student submissions.
In response to this repetitive nature of feedback, Dodona has implemented a feature enabling instructors to save and later retrieve specific messages.
This functionality facilitates the reuse of feedback by allowing teachers to search for previously saved messages.
By using this functionality, we have generated data that we can use in this study: code submissions, where those submissions have been annotated on specific lines with messages that are shared over those submissions. 

Note that there are two concepts here, whose distinction is important.
On the one hand, we have /annotations/.
Annotations are a specific instance of a message left by a grader: it consists of its text, and is also linked to a specific line of a specific submission.
On the other hand we have /messages/.
This is the text that can be reused by graders when adding an annotation.

In this section we give an answer to the following research question: Can we, in the context of grading code written by students during an evaluation, use previously given feedback to predict what feedback a grader will give on a particular line?

We present a machine learning method for suggesting re-use of previously given feedback.
The manuscript starts with an in-depth explanation of the design of the method.
We then present and discuss the experimental results we obtained when testing our method on student submissions.
Two datasets are used to evaluate our method, based on real (Python) code written by students for exams.
With the first dataset we predict automated PyLint messages.
For the second dataset we use actual annotations left by graders during the grading of an exam.

*** Methodology
:PROPERTIES:
:CREATED: [2024-01-08 Mon 13:18]
:CUSTOM_ID: subsec:feedbackpredictionmethodology
:END:

The approach we present to predict what feedback a grader will give on source code is based on pattern mining.
Pattern mining is a data mining technique for extracting frequently occurring patterns from data that can be represented as trees.
It was already developed in the early 2000s\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005; @asaiEfficientSubstructureDiscovery2004].
Program code can be represented as an abstract syntax tree (AST), where nodes of the tree represent the language constructs used in the program.
More recent work used this fact to look into how these pattern mining algorithms could be used to efficiently find frequent patterns in source code\nbsp{}[cite:@phamMiningPatternsSource2019].
In an educational context, these techniques could then be used to, for example, find patterns common to solutions that failed a given exercise\nbsp{}[cite:@mensGoodBadUgly2021].
Other work looked into generating unit tests from mined patterns\nbsp{}[cite:@lienard2023extracting].

We start with a general overview of our method (explained visually in Figure\nbsp{}[[fig:feedbackmethodoverview]]).
We start by using the tree-sitter library\nbsp{}[cite:@brunsfeldTreesitterTreesitterV02024] to generate ASTs for each submission.
For every annotation, a constrained AST context surrounding the annotated line is extracted.
Subsequently, we then aggregate all the subtrees for each occurrence of a message.
Every message's collection of subtrees is processed by the =TreeminerD= algorithm, yielding a set of frequently occuring patterns specific for that message.
We assign weights to these patterns based on their length and their frequency across the entire dataset of patterns for all messages.
The result of these operations is our trained model.

The model can now be used to suggest matching patterns and thus messages for a given code fragment.
In practice, the instructor first selects a line of code in a given student's submission.
Next, the AST of the selected line and its surrounding context is generated.
For all messages, each of its patterns is matched to the line, and a similarity score is calculated (given the weights determined earlier).
This similarity score is used to rank the messages and this ranking is shown to the teacher.

A detailed explanation of this process follows, with a particular emphasis on operational efficiency.
Speed is a paramount concern throughout the model's lifecycle, from training to deployment in real-time grading contexts.
Given the continuous generation of training data during the grading process, the model's training duration must be optimized to prevent significant delays, ensuring that the model remains practical for live grading situations.

#+CAPTION: Overview of our machine learning method for predicting feedback re-use.
#+CAPTION: Code is converted to its Abstract Syntax Tree form.
#+CAPTION: Annotations for the same message have been given the same colour.
#+CAPTION: Per message, the context of each annotation is extracted and mined for patterns using the =TreeminerD= algorithm.
#+CAPTION: These patterns are then weighted, after which they make up our model.
#+CAPTION: When a teacher wants to place an annotation on a line, messages are ranked based on the similarity determined for that line.
#+NAME: fig:feedbackmethodoverview
[[./diagrams/feedbackmethodoverview.svg]]

**** Extracting a subtree around a line
:PROPERTIES:
:CREATED:  [2024-01-19 Fri 15:44]
:CUSTOM_ID: subsubsec:feedbackpredictionsubtree
:END:

Currently, the context around a line is extracted by taking all the AST nodes that are solely on that line.
For example, the subtree extracted for the code on line 3 in Listing\nbsp{}[[lst:feedbacksubtreesample]] can be seen on Figure\nbsp{}[[fig:feedbacksubtree]].
Note that the context we extract here is very limited.
Previous iterations considered all the nodes that contained the relevant line (e.g. the function node for a line in a function), but these contexts turned out to be too large to process in an acceptable time frame.

#+ATTR_LATEX: :float t
#+CAPTION: Sample code that simply reads a number from standard input and prints its digits.
#+NAME: lst:feedbacksubtreesample
#+BEGIN_SRC python
number = input()
print(f'{number} has the following digits:')
for digit in number:
  print(digit)
#+END_SRC

#+CAPTION: AST subtree corresponding to line 3 in Listing\nbsp{}[[lst:feedbacksubtreesample]].
#+NAME: fig:feedbacksubtree
[[./diagrams/feedbacksubtree.svg]]

**** =TreeminerD=
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:33]
:CUSTOM_ID: subsubsec:feedbackpredictiontreeminer
:END:

=Treeminer=\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005] is an algorithm for discovering frequently occurring subtrees in datasets of rooted, ordered and labelled trees.
It does this by starting with a list of frequently occurring nodes, and then iteratively expanding the frequently occurring patterns.

In the base =Treeminer= algorithm, frequently occurring means that the amount of times the pattern occurs in all trees divided by the amount of trees is larger than some predefined threshold.
This is the =minimum support= parameter.

Patterns are embedded subtrees: the nodes in a pattern are a subset of the nodes of the tree, where the ancestor-descendant relationships are kept and the left-to-right ordering of nodes is also preserved.

=TreeminerD= is a more efficient version of the base =Treeminer= algorithm.
It achieves this efficiency by not counting the amount of occurrences of a frequent pattern within one tree.
Since we are not interested in this information for our method, it was an obvious choice to use the =TreeminerD= version.

We use our own Python implementation of the =TreeminerD= algorithm, in this case to find patterns in the AST subtrees for each message.
We set the =minimum support= parameter to 0.8 in our implementation.
This value was experimentally determined.

As an example, one message in our real-world dataset was placed 92 times on 47 submissions by students.
For this message =TreeminerD= finds 105\thinsp{}718 patterns.


**** Assigning weights to patterns
:PROPERTIES:
:CREATED: [2023-11-22 Wed 14:39]
:CUSTOM_ID: subsubsec:feedbackpredictionweights
:END:

Due to the iterative nature of =TreeminerD= a lot of patterns are (embedded) subtrees of other patterns.
We don't do any post-processing to remove these patterns since they might be relevant for code we have not seen yet, but we do assign weights to them.
Weights are assigned using two criteria.

The first criterion is the size of the pattern (i.e., the number of nodes in the pattern), since a pattern with twenty nodes is a lot more specific than a pattern with only one node.
The second criterion is the amount of times a pattern occurs across all messages.
If all messages contain a specific pattern, it can not be reliably used to determine which message should be predicted and will therefore be assigned a smaller weight.
The weights are calculated by the following formula: \[weight(pattern) = \frac{len(pattern)}{\#occurences(pattern)}\]

**** Matching patterns to subtrees
:PROPERTIES:
:CREATED:  [2024-02-01 Thu 14:25]
:CUSTOM_ID: subsubsec:feedbackpredictionmatching
:END:

To know whether a given pattern matches a given subtree, we iterate over all nodes in the subtree.
Simultaneously, we also iterate over the nodes in the patterns.
During iteration, we also store the current depth, both in the pattern and the subtree.
We also keep a stack to store (some of) the subtree depths.
If the current label in the subtree and the pattern are the same, we store the current subtree depth on the stack and move to the next node in the pattern.
For moving upwards in the tree things are more complicated.
If the current depth and the depth of the last match (stored on the stack) are the same, we can move forwards in the pattern (and the subtree).
If this is not the case, we need to check that we are still in the embedded subtree, otherwise we need to reset our position in the pattern to the start.
Because subtrees can contain the same label multiple times we also need to make sure we can backtrack.
The full pseudocode for this algorithm can be seen in Listing\nbsp{}[[lst:feedbackmatchingpseudocode]].

#+ATTR_LATEX: :float t
#+CAPTION: Pseudocode for checking whether a pattern matches a subtree.
#+CAPTION: Note that both pattern and subtree are stored in the encoding described by\nbsp{}[cite/t:@zakiEfficientlyMiningFrequent2005].
#+NAME: lst:feedbackmatchingpseudocode
#+BEGIN_SRC python
subtree_matches(subtree, pattern):
    p_i, pattern_depth, depth = 0
    depth_stack = []
    for item in subtree:
        if item == -1:
            if depth_stack is not empty and \
              depth - 1 == depth_stack.last:
                last_depth = depth_stack.pop()
                if pattern [p_i] != -1 and \
                  (last_depth < pattern_depth or \
                    depth_stack is empty):
                    p_i = 0
                if pattern[p_i] == -1:
                    pattern_depth -= 1
                    p_i += 1
            depth -= 1
        else:
            if pattern[p_i] == item:
                depth_stack.append(depth)
                pattern_depth += 1
                p_i += 1
            depth += 1
    if p_i == pattern_length:
        return True
    return False
#+END_SRC

Checking whether a pattern matches a subtree is an operation that needs to happen a lot of times.
For some messages, there are many patterns, and all patterns of all messages are checked.
One important optimization we added was therefore to only execute the algorithm in Listing\nbsp{}[[lst:feedbackmatchingpseudocode]] if the set of labels in the pattern is a subset of the labels in the subtree.

**** Ranking the messages
:PROPERTIES:
:CREATED: [2023-11-22 Wed 14:47]
:CUSTOM_ID: subsubsec:feedbackpredictionsimilarity
:END:

Given a model where we have weighted patterns for each message, and a method for matching patterns to subtrees, we can now put these two together to make a final ranking of the messages for a line of code.
We calculate a matching score for each message with the following formula: \[ score(message) = \frac{\displaystyle\sum_{pattern}^{patterns} \begin{cases} weight(pattern) & \quad \text{if } pattern \text{ matches} \\ 0               & \quad \text{otherwise} \end{cases}}{len(patterns)} \]
The messages are sorted using this score.

*** Results and discussion
:PROPERTIES:
:CREATED: [2024-01-08 Mon 13:18]
:CUSTOM_ID: subsec:feedbackpredictionresults
:END:

We used two datasets to evaluate our method.
Both are based on real (Python) code written by students for (different) exams.
To test our method, we split the datasets in half and used the first half to train and the second half to test.
During the test phase, we iterate over the places where annotations were added in the source data.
These are the lines we give to our model.
We look at whether the correct message is ranked first, or if it is ranked in the first five suggestions.
This gives us a good idea on how useful this would be in practice: if a message is ranked farther than fifth, we would expect the grader to need to search for it.

In the first dataset, we run PyLint on those student submissions, and use PyLint's annotations as our training data and test data.
Note that in this dataset, we don't make the distinction between the different assignments students had to solve, since the way Pylint annotates them does not differ between assignments.
In the second dataset, we use actual annotations left by graders on student code in Dodona.
Here we train and test per assignment, since the set of messages that were used is also different for each assignment.

We differentiate between these two datasets, because we expect PyLint to be more consistent in when it places an annotation and also where it places that annotation.
Most linting messages are detected through explicit pattern matching in the AST, so we expect our implicit pattern matching to perform rather well.
Real-world data is more difficult, since graders are humans, and might miss an issue in one student's code that they annotated in another student's code, or they might not place the annotation for a certain message in a consistent location.
The method by which graders place an annotation is also a lot more implicit than PyLint's pattern matching.

**** PyLint
:PROPERTIES:
:CUSTOM_ID: subsubsec:feedbackpredictionresultspylint
:CREATED: [2023-11-20 Mon 13:33]
:END:

We will first discuss the PyLint results.
As can be seen in Figure\nbsp{}[[fig:feedbackpredictionpylintglobal]], for about 70% of the annotations, the actual message is ranked in the top five of messages.
For about 30% of the annotations, the message is even ranked first.

#+CAPTION: Global overview of the performance of our method on PyLint data.
#+CAPTION: For 70% of the annotations, the expected message is ranked in the top five.
#+CAPTION: In 30% of the annotations, the expected message is ranked first.
#+NAME: fig:feedbackpredictionpylintglobal
[[./images/feedbackpredictionpylintglobal.png]]

In Figure\nbsp{}[[fig:feedbackpredictionpylintmessages]], we have highlighted some messages, some of which perform very well, and some of which perform worse.
The differences in performance can be explained through the content of the message and the underlying patterns sought by PyLint.
For example, the message "too many branches" performs rather poorly.
This can be explained through the fact that we prune too much context for the pattern that PyLint used to be picked up by =TreeminerD=.
There are also annotations that can not be predicted at all, because no patterns are found.

Other messages, like "consider using with", perform very well.
For these messages, =TreeminerD= does have enough context to pick up the underlying patterns.
The amount of times the message occurs in the training set also has an impact.
Messages which only have a few annotations are generally predicted worse than those with a lot of annotations.

#+CAPTION: Detailed view of predictions for a few PyLint messages.
#+CAPTION: Each bar is a message, and the amount of occurrences in the training set and in the test set (respectively) is denoted in brackets after the name.
#+NAME: fig:feedbackpredictionpylintmessages
[[./images/feedbackpredictionpylintmessages.png]]

**** Real-world data
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:33]
:CUSTOM_ID: subsubsec:feedbackpredictionresultsrealworld
:END:

For the real-world data, we applied two different techniques to test our method.
Aside from using the same 50/50 split as with the PyLint data, we also tried to simulate how a grader would use the method, gradually increasing the size of the training set and decreasing the size of the test set.

The results of the first test can be seen on Figure\nbsp{}[[fig:feedbackpredictionrealworldglobal]].
The results are similar to the PyLint data.
The percentages of annotations where the correct message is ranked first are generally higher (between 35 and 55 percent), and the percentages of annotations where it is ranked in the first five annotations are between 65 and 80 percent.

#+CAPTION: Prediction results for four exercises that were part of an exam.
#+CAPTION: Our model was trained on half of the submissions in the dataset and tested with the other half.
#+NAME: fig:feedbackpredictionrealworldglobal
[[./images/feedbackpredictionrealworldglobal.png]]

For the next test, an extra category of annotations was added, namely "Not yet seen".
This means that the message for that annotation was not part of the test set, and thus could never have been predicted.
Results of this test (for the same exercises as in Figure\nbsp{}[[fig:feedbackpredictionrealworldglobal]]) can be seen on Figures\nbsp{}[[fig:feedbackpredictionrealworldsimulation1]],\nbsp{}[[fig:feedbackpredictionrealworldsimulation2]],\nbsp{}[[fig:feedbackpredictionrealworldsimulation3]]\nbsp{}and\nbsp{}[[fig:feedbackpredictionrealworldsimulation4]].
These figures show that while some build-up is required, once a critical mass of annotations is reached, the accuracy of the system jumps to its final state.

#+CAPTION: Prediction results for the exercise "A last goodbye" over time.
#+CAPTION: The amount of annotations for which the message has never been seen is marked separately.
#+CAPTION: The chart on the right shows the messages that are in the training set.
#+NAME: fig:feedbackpredictionrealworldsimulation1
[[./images/feedbackpredictionrealworldsimulation1.png]]

#+CAPTION: Prediction results for the exercise "Narcissus cipher" over time.
#+CAPTION: The amount of annotations for which the message has never been seen is marked separately.
#+CAPTION: The chart on the right shows the messages that are in the training set.
#+NAME: fig:feedbackpredictionrealworldsimulation2
[[./images/feedbackpredictionrealworldsimulation2.png]]

#+CAPTION: Prediction results for the exercise "Cocktail bar" over time.
#+CAPTION: The amount of annotations for which the message has never been seen is marked separately.
#+CAPTION: The chart on the right shows the messages that are in the training set.
#+NAME: fig:feedbackpredictionrealworldsimulation3
[[./images/feedbackpredictionrealworldsimulation3.png]]

#+CAPTION: Prediction results for the exercise "Anthropomorphic emoji" over time.
#+CAPTION: The amount of annotations for which the message has never been seen is marked separately.
#+CAPTION: The chart on the right shows the messages that are in the training set.
#+NAME: fig:feedbackpredictionrealworldsimulation4
[[./images/feedbackpredictionrealworldsimulation4.png]]

*** Conclusions and future work
:PROPERTIES:
:CREATED: [2023-11-20 Mon 13:33]
:CUSTOM_ID: subsec:feedbackpredictionconclusion
:END:

We presented a prediction method to help when giving feedback during grading by re-using messages.
Improving re-use of messages can be both a time-saver, and improve consistency with which feedback is given.

The framework already has promising results.
We validated the framework by predicting both automated linting messages to establish a baseline and by using real-world data.
The method performs about the same for real-world data as it does for PyLint's linting messages.
We can thus answer our research question and say that yes, we can use previously given feedback to predict what feedback a grader will give on a particular line.

Of course, alternative methods could also be considered.
One cannot overlook the rise of Large Language Models (LLMs) and the way they could contribute to this problem.
LLMs can also generate feedback for students, based on their code and a well-chosen system prompt.
Fine-tuning of a model with feedback already given could also be considered.

We can conclude that the proposed method achieved the aimed-for objective.
Having this method at hand immediately raises some possible follow-up work.
Messages that don't lend themselves well to being predicted need further investigation.
The context used could also be extended (although the important caveat here is that the method also needs to still maintain its speed).
Right now the model is also reactive: we propose a group of most likely messages when a grader wants to add an annotation on a line.
By introducing a confidence score we could check beforehand if we have a confident match for each line and then immediately propose this to the grader.
Whether a grader accepts this suggestion could then also be used as an input into the model.
We could also look into applying some of the techniques for source code pattern mining proposed by\nbsp{}[cite/t:@phamMiningPatternsSource2019] to make further speed improvements.
Another important aspect that was explicitly left out of scope in this chapter is building it into a learning platform and doing user testing.

* Discussion and future work
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:51]
:CUSTOM_ID: chap:discussion
:END:

It's safe to say that Dodona is a successful automated assessment platform.
{{{num_users}}} users is quite a lot, and the fact that it is being actively used in every university in Flanders, a number of colleges, and a lot of secondary schools is a feat that not many other platforms like it have achieved.

As we have tried to show in this dissertation, its development has also led to interesting opportunities for new research.
Dodona generates a lot of data by being used, and we have shown that educational data mining can be used on this data.
It can even be used to develop new educational data mining techniques.
The work is, however, never finished.
There are still possibilities for interesting computer science and educational research.

** Research opportunities
:PROPERTIES:
:CREATED:  [2024-02-16 Fri 10:50]
:END:

A big question, left open in this work, is what to do with the results we obtained in Chapter\nbsp{}[[#chap:passfail]].
Teachers can use the results to figure out which aspects of their course students are struggling with, and take general measures to deal with this.
But should we, and if so, /how/ should we communicate predictions to individual students?

Chapter\nbsp{}[[#chap:feedback]] also suggests a number of improvements that could still be worked on.
It gives us a framework for suggesting the feedback a teacher probably wants to give when selecting a line, but we could also try to come up with a confidence score and use that to suggest feedback before the teacher has even done that.
Another interesting (more educational) line of research that this work suggests is building the method into an actual assessment platform, and looking at its effects on feedback consistency and quality, time saved by teachers,\nbsp{}...

A new idea for research using Dodona's data would be skill estimation.
There are a few ways we could try to infer what skills are being tested by exercises: we could try to use the model solution, or the labels assigned to the exercise in Dodona.
Using those skills, we could try to estimate a student's mastery of those skills, using their submissions.

This leads right into another possibility for future research: exercise recommendation.
Right now, learning paths in Dodona are static, determined by the author of the course the student is following.
Dodona has a rich library of extra exercises, which is linked to in some courses, but it is not always easy for students to know what exercises would be good for them.
The research from Chapter\nbsp{}[[#chap:passfail]] could also be used to help solve this problem.
If we know a student has a higher chance of failing the course, we might want to recommend some easier exercises.
The other way around, if a student has a higher chance of passing, we could suggest harder exercises, so they can keep up their good progress in their course.

The final possibility we will present here is to prepare suggestions for answers to student questions on Dodona.
At first glance, LLMs should be quite good at this.
If we use the LLM output as a suggestion for what the teacher could answer, this should be a big time-saver.
However, there are some issues around data quality.
Questions are sometimes asked on a specific line, but the question doesn't necessarily have anything to do with that line.
Sometimes the question also needs context that is hard to pass on to the LLM.
For example, if the question is just "I don't know what's wrong.", a human might look at the failed test cases and be able to answer the "question" in that way.
Passing on the failed test cases to the LLM is a harder problem to solve.
The actual assignment also needs to be passed on, but depending on its size this might also present a problem given token limitations/cost per token of some models.
Another important aspect of this research would be figuring out how to evaluate the quality of the suggestions.

** Challenges for the future
:PROPERTIES:
:CREATED:  [2024-02-16 Fri 10:50]
:END:

Even though Dodona is a successful project with some exciting possibilities for research that can still be done, the project also faces some challenges.

The most important of these challenges is the sustainability of the project.
Dodona was started in the spare time of some researchers.
After a few years, there was somebody working on it full-time.
However, the funding for a full-time developer was always, and still is, temporary.
PhD students who can devote some of their time to it are attracted, grants are applied for (and sometimes received), but there is no stable source of funding.
We have the advantage that we can kindly make use of Ghent University's data centre, resulting in very few operational costs.
A full-time developer, which Dodona is big enough to need, is expensive though.
This puts Dodona's future in a precarious situation, where we constantly have to be on the lookout for new funding opportunities.

As much as generative AI can be an asset for Dodona, it is also a threat.
Most exercises in Dodona can be solved by LLMs without issues.[fn:: Or at least with some nudging.]
This has some troubling implications for Dodona.
Students using ChatGPT or GitHub Copilot when solving their exercises, might not learn as much as students who do the work fully on their own (just like students who plagiarize have a lower chance of passing their courses, as seen in Chapter\nbsp{}[[#chap:passfail]]).
Another aspect is the fairness and integrity of evaluations using Dodona.
The case study in Chapter\nbsp{}[[#chap:use]] details the use of open-book/open-internet evaluations.
If students can use generative AI during these evaluations, and knowing that LLMs can solve most exercises on Dodona, these evaluations will test the students' abilities less and less, if students can use LLMs.
The way to solve these issues is not clear.
It seems like LLMs are here to stay, and just like the calculator is a commonplace tool these days, the same could be true for LLMs in the future.
Instead of banning the use of LLMs, teachers could integrate the use of them in their courses.
On the other hand, when children first learn to count and add, they do not use calculators.
The same might be necessary when learning to program: to learn the basics, students might need to do a lot of things themselves, to really get a feel for what they are doing.

#+LATEX: \appendix
* Pass/fail prediction feature types
:PROPERTIES:
:CREATED: [2023-10-23 Mon 18:09]
:CUSTOM_ID: chap:featuretypes
:APPENDIX: t
:END:

- =subm= :: numbers of submissions by student in series
- =nosubm= :: number of exercises student did not submit any solutions for in series
- =first_dl= :: time difference in seconds between student's first submission in series and deadline of series
- =last_dl= :: time difference in seconds between student's last submission in series before deadline and deadline of series
- =nr_dl= :: number of correct submissions in series by student before series' deadline
- =correct= :: number of correct submissions in series by student
- =after_correct= :: number of submissions by student after their first correct submission in the series
- =before_correct= :: number of submissions by student before their first correct submission in the series
- =time_series= :: time difference in seconds between the student's first and last submission in the series
- =time_correct= :: time difference in seconds between the student's first submission in the series and their first correct submission in the series
- =wrong= :: number of submissions by student in series with logical errors
- =comp_error= :: number of submissions by student in series with compilation errors
- =runtime_error= :: number of submissions by student in series with runtime errors
- =correct_after_5m= :: number of exercises where first correct submission by student was made within five minutes after first submission
- =correct_after_15m= :: number of exercises where first correct submission by student was made within fifteen minutes after first submission
- =correct_after_2h= :: number of exercises where first correct submission by student was made within two hours after first submission
- =correct_after_24h= :: number of exercises where first correct submission by student was made within twenty-four hours after first submission

* Bibliography
:PROPERTIES:
:CREATED: [2023-10-23 Mon 08:59]
:CUSTOM_ID: chap:bibliography
:UNNUMBERED: t
:END:

#+LATEX: {\setlength{\emergencystretch}{2em}
#+print_bibliography: 
#+LATEX: }