Update feedback prediction chapter with latest version
This commit is contained in:
Charlotte Van Petegem
parent
1c4079cd8a
commit
7a084fadae
14 changed files with 262 additions and 237 deletions
408
book.org
408
book.org
|
|
@ -2968,44 +2968,47 @@ This is exactly what we will explore in this section.
|
|||
Feedback is a key factor in student learning\nbsp{}[cite:@hattiePowerFeedback2007; @blackAssessmentClassroomLearning1998].
|
||||
In programming education, many steps have been taken to give feedback using automated assessment systems\nbsp{}[cite:@paivaAutomatedAssessmentComputer2022; @ihantolaReviewRecentSystems2010; @ala-mutkaSurveyAutomatedAssessment2005].
|
||||
These automated assessment systems give feedback on correctness, and can give some feedback on style and best practices through the use of linters.
|
||||
They are, however, generally not able to give the same high-level feedback on program design that an experienced programmer can give.
|
||||
However, they are generally unable to give the same high-level feedback on program design that an experienced programmer can give.
|
||||
In many educational practices, automated assessment is therefore supplemented with manual feedback, especially when grading evaluations or exams\nbsp{}[cite:@debuseEducatorsPerceptionsAutomated2008].
|
||||
This requires a significant time investment of teachers\nbsp{}[cite:@tuckFeedbackgivingSocialPractice2012].
|
||||
|
||||
As a result, many researchers have explored the use of AI to enhance giving feedback.
|
||||
[cite/t:@vittoriniAIBasedSystemFormative2021] used natural language processing to automate grading, and found that students who used the system during the semester were more likely to pass the course at the end of the semester.
|
||||
[cite/t:@leeSupportingStudentsGeneration2023] has used supervised learning with ensemble learning to enable students to perform peer and self-assessment.
|
||||
In addition, [cite/t:@berniusMachineLearningBased2022] introduced a framework based on clustering text segments in textual exercises to reduce the grading workload.
|
||||
In addition, [cite/t:@berniusMachineLearningBased2022] introduced a framework based on clustering text segments in free-form textual exercises to reduce the grading workload.
|
||||
[cite/t:@strickrothSupportingSemiautomaticFeedback2023] attempt to solve this problem specifically for programming exercises by clustering submissions based on failed tests cases and compiler error messages.
|
||||
|
||||
The context of our work is in our own assessment system, called Dodona, developed at Ghent University\nbsp{}[cite:@vanpetegemDodonaLearnCode2023].
|
||||
Dodona gives automated feedback on each submission, but also has a module that allows teachers to give manual feedback on student submissions and assign scores to them.
|
||||
The process of giving manual feedback on a submission for a programming exercise in Dodona is very similar to a code review, where errors or suggestions for improvements are annotated on the relevant line(s)\nbsp{}(Figure\nbsp{}[[fig:feedbackintroductionreview]]).
|
||||
3\thinsp{}663\thinsp{}749 submissions were submitted to Dodona in 2023 alone, of which 44\thinsp{}012 were then also assessed manually.
|
||||
The context of our work is the Dodona learning environment, developed at Ghent University\nbsp{}[cite:@vanpetegemDodonaLearnCode2023].
|
||||
Dodona gives automated feedback on each submitted solution to programming exercises, but also has a module that allows teachers to give manual feedback on student submissions and assign scores.
|
||||
The process of giving manual feedback on a submission to a programming exercise in Dodona is very similar to a code review, where errors or suggestions for improvements are annotated on the relevant line(s)\nbsp{}(Figure\nbsp{}[[fig:feedbackintroductionreview]]).
|
||||
n 2023 alone, 3\thinsp{}663\thinsp{}749 solutions were submitted to Dodona, of which 44\thinsp{}012 were manually assessed.
|
||||
During manual assessment, 22\thinsp{}888 annotations were added to specific lines of code.
|
||||
|
||||
#+CAPTION: Manual assessment of a submission: a teacher gives feedback on the code by adding inline annotations and scores the submission by filling out the exercise-specific scoring rubric.
|
||||
#+CAPTION: Automated assessment was already performed beforehand, where 22 test cases failed, as can be seen from the badge on the "Correctness" tab.
|
||||
#+CAPTION: The teacher just searched for a previously saved annotation so that they could reuse it.
|
||||
#+CAPTION: Assessment of a submitted solution in Dodona.
|
||||
#+CAPTION: An automated assessment has already been performed, with 22 failed test cases, as can be seen from the badge on the "Correctness" tab.
|
||||
#+CAPTION: An automated annotation left by Pylint can be seen on line 22.
|
||||
#+CAPTION: After manually assessing this submission, the teacher still needs to assess 23 other submissions, as can be seen on the progress bar on the right.
|
||||
#+CAPTION: A teacher gives feedback on the code by adding inline annotations and scores the submission by filling out the exercise-specific scoring rubric.
|
||||
#+CAPTION: The teacher has just searched for a previously saved annotation so that they could reuse it.
|
||||
#+CAPTION: After manually assessing this submission, the teacher still has another 23 submissions to assess, as shown in the progress bar on the right.
|
||||
#+NAME: fig:feedbackintroductionreview
|
||||
[[./images/feedbackintroductionreview.png]]
|
||||
|
||||
However, there is a crucial difference between traditional code reviews and those in an educational context: teachers often give feedback on numerous submissions to the same exercise.
|
||||
Since students often make similar mistakes in their submissions to an exercise, it logically follows that teachers will repeatedly give the same or similar feedback on multiple student submissions.
|
||||
Because of this repetitive nature of feedback, Dodona allows teachers to save and later search for and retrieve specific annotations.
|
||||
This feature facilitates the reuse of feedback by allowing teachers to search for previously saved annotations.
|
||||
In 2023, 777 annotations were saved by teachers on Dodona, which were reused 7\thinsp{}180 times in total.
|
||||
The utilisation of this functionality has generated data that we can use in this study: code submissions, annotated on specific lines with annotations that are being shared across submissions.
|
||||
To facilitate the reuse of feedback, Dodona allows teachers to save specific annotations for later search and retrieval.
|
||||
In 2023, 777 annotations were saved by teachers on Dodona, which were reused a total of 7\thinsp{}180 times.
|
||||
The usage of this functionality has generated data that we can use in this study: annotations that are shared between code submissions and that occur on specific lines of those submissions.
|
||||
|
||||
In this section we answer the following research question: Can previously added annotations be used to predict what annotations a reviewer is likely to add to a specific line of code, during the manual assessment of student-written code?
|
||||
In this section we answer the following research questions:
|
||||
(RQ1) Can previously added annotations be used to predict what annotations a reviewer is likely to add to a specific line of code during manual assessment of student-written code?
|
||||
(RQ2) Additionally, can this be done so that both training of and predictions by the method are fast enough to use in live reviewing situations with human reviewers?
|
||||
|
||||
We present a machine learning approach aimed at facilitating the reuse of previously given feedback.
|
||||
We begin with a detailed explanation of the design of our method.
|
||||
We then present and discuss the experimental results we obtained by testing our method on student submissions.
|
||||
The dataset we used for this experiment is based on real Python code written by students during examinations.
|
||||
First, we test our method by predicting machine annotations by Pylint.
|
||||
Second, we use manual annotations left by human reviewers during assessment.
|
||||
We present ECHO (Efficient Critique Harvesting and Ordering), a machine learning approach that aims to facilitate the reuse of previously given feedback.
|
||||
We begin with a detailed explanation of the design of ECHO.
|
||||
We then present and discuss the experimental results we obtained by testing ECHO on student submissions.
|
||||
The dataset we used for this experiment is based on real Python code written by students during exams.
|
||||
First, we test ECHO by predicting Pylint machine annotations.
|
||||
Next, we use annotations left by human reviewers during manual assessment.
|
||||
|
||||
*** Methodology
|
||||
:PROPERTIES:
|
||||
|
|
@ -3013,54 +3016,60 @@ Second, we use manual annotations left by human reviewers during assessment.
|
|||
:CUSTOM_ID: subsec:feedbackpredictionmethodology
|
||||
:END:
|
||||
|
||||
Our approach for predicting what feedback a reviewer will give on source code is based on tree mining.
|
||||
We consider predicting relevant annotations to be a ranking problem, which we solve by determining similarity between the lines of code where annotations are added.
|
||||
The approach to determine this similarity is based on tree mining.
|
||||
This is a data mining technique for extracting frequently occurring patterns from data that can be represented as trees\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005; @asaiEfficientSubstructureDiscovery2004].
|
||||
Program code can be represented as an abstract syntax tree (AST), where the nodes of the tree represent the language constructs used in the program.
|
||||
Recent work has used this fact to investigate how these pattern mining algorithms can be used to efficiently find frequent patterns in source code\nbsp{}[cite:@phamMiningPatternsSource2019].
|
||||
In an educational context, these techniques were already used, for example, to find patterns common to solutions that failed a given exercise\nbsp{}[cite:@mensGoodBadUgly2021].
|
||||
Other work has looked at automatically generating unit tests from mined patterns\nbsp{}[cite:@lienard2023extracting].
|
||||
We use tree mig to find commonalities between the lines of code where the same annotation was added.
|
||||
Recent work has used demonstrated the efficacy of this approach in efficiently identifying frequent patterns in source code\nbsp{}[cite:@phamMiningPatternsSource2019].
|
||||
In an educational context, these techniques have already been used to find patterns common to solutions that failed a given exercise\nbsp{}[cite:@mensGoodBadUgly2021].
|
||||
Other work has demonstrated the potential of automatically generating unit tests from mined patterns\nbsp{}[cite:@lienard2023extracting].
|
||||
We use tree minig to find commonalities between the lines of code where the same annotation has been added.
|
||||
|
||||
We start with a general overview of our method (Figure\nbsp{}[[fig:feedbackmethodoverview]]).
|
||||
We begin with a general overview of ECHO (Figure\nbsp{}[[fig:feedbackmethodoverview]]).
|
||||
The first step is to use the tree-sitter library\nbsp{}[cite:@brunsfeldTreesitterTreesitterV02024] to generate ASTs for each submission.
|
||||
Using tree-sitter makes our method independent of the programming language used, since it is a generic interface for generating syntax trees.
|
||||
Using tree-sitter makes ECHO independent of the programming language used, since it presents an interface for generating syntax trees independent of the programming language.
|
||||
The syntax trees are post-processed to include identifier names.
|
||||
For each annotation, we identify all occurrences and extract a constrained AST context around the annotated line for each instance.
|
||||
The resulting subtrees are then aggrefated for each annotation, after which they are processed by the =TreeminerD= algorithm\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005].
|
||||
The resulting subtrees are then aggregated for each annotation.
|
||||
If there are three or more subtrees, they are processed by the =TreeminerD= algorithm\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005].
|
||||
This yields a set of frequently occurring patterns specific to that annotation.
|
||||
We assign weights to these patterns based on their length and their frequency across the entire dataset of patterns for all annotations.
|
||||
We then assign weights to these patterns based on their length and their frequency across the entire dataset of patterns for all annotations.
|
||||
In addition to pattern mining, we also determine a set of unique nodes per forest of subtrees.
|
||||
The result of these operations is our trained model.
|
||||
|
||||
The model can then be used to score how well an annotation matches a given code fragment.
|
||||
In practice, the reviewer first selects a line of code in a given student's submission.
|
||||
Next, the AST of the selected line and its surrounding context is generated.
|
||||
For each annotation, each of its patterns is matched to the line, and a similarity score is calculated, given the previously determined weights.
|
||||
This similarity score is used to rank the annotations and this ranking is displayed to the teacher.
|
||||
The percentage of unique nodes which match in the current subtree is also taken into account.
|
||||
These scores are used to rank the annotations, which are then displayed to the reviewer.
|
||||
It is important to note that the reviewer remains in control of which annotation is used, if any.
|
||||
|
||||
A detailed explanation of this process follows, with a particular emphasis on operational efficiency.
|
||||
Speed is a paramount concern throughout the model's lifecycle, from training to deployment in real-time reviewing contexts.
|
||||
Given the continuous generation of training data during the reviewing process, the model's training time must be optimized to avoid significant delays, ensuring that the model remains practical for live reviewing situations.
|
||||
We will now provide a more in-depth explanation of this process, with a particular emphasis on operational efficiency.
|
||||
Speed is of the utmost importance throughout the model's lifecycle, from training to deployment in real-time reviewing contexts.
|
||||
Given the continuous generation of training data during the review process, the model's training time must be optimized to avoid significant delays, ensuring that the model remains practical for live review situations.
|
||||
|
||||
#+CAPTION: Overview of our machine learning method for predicting annotation reuse.
|
||||
#+CAPTION: Overview of ECHO.
|
||||
#+CAPTION: Code of previously reviewed submissions is converted to its abstract syntax tree (AST) form.
|
||||
#+CAPTION: Instances of the same annotation have the same colour.
|
||||
#+CAPTION: For each annotation, the context of each instance is extracted and mined for patterns using the =TreeminerD= algorithm.
|
||||
#+CAPTION: These patterns are then weighted to form our model.
|
||||
#+CAPTION: When a teacher wants to place an annotation on a line of the submissions they are currently reviewing, all previously given annotations are ranked based on the similarity determined for that line.
|
||||
#+CAPTION: The teacher can then choose which annotation they want to place, with the goal to have the selected annotation high up in the ranking.
|
||||
#+CAPTION: When a reviewer wants to place an annotation on a line of the submissions they are currently reviewing, all previously given annotations are ranked based on the similarity determined for that line.
|
||||
#+CAPTION: The reviewer can then choose which annotation they want to place, with the aim of having the selected annotation at the top of in the ranking.
|
||||
#+NAME: fig:feedbackmethodoverview
|
||||
[[./diagrams/feedbackmethodoverview.svg]]
|
||||
|
||||
**** Subtree extraction
|
||||
**** Training
|
||||
:PROPERTIES:
|
||||
:CREATED: [2024-01-19 Fri 15:44]
|
||||
:CUSTOM_ID: subsubsec:feedbackpredictionsubtree
|
||||
:END:
|
||||
|
||||
The first step of our method is to extract a subtree for every instance of an annotation and then aggregate them per annotation.
|
||||
Currently, the context around a line is extracted by taking all AST nodes from that line.
|
||||
For example, Figure\nbsp{}[[fig:feedbacksubtree]] shows that the subtree extracted for the code on line 3 in Listing\nbsp{}[[lst:feedbacksubtreesample]].
|
||||
The first step of ECHO is to extract a subtree for each instance of an annotation and then aggregate them per annotation.
|
||||
Currently, the context around a line is extracted by taking all the AST nodes from that line.
|
||||
For example, Figure\nbsp{}[[fig:feedbacksubtree]] shows that the subtree extracted for the code on line 3 of Listing\nbsp{}[[lst:feedbacksubtreesample]].
|
||||
Note that the context we extract here is very limited.
|
||||
Previous iterations of our method considered all nodes that contained the relevant line (e.g. the function node for a line in a function), but these contexts proved too large to process in an acceptable time frame.
|
||||
Previous iterations of our method considered all nodes that contained the relevant line (e.g. the function node for a line in a function), but these contexts proved too large to process in an acceptable time.
|
||||
|
||||
#+ATTR_LATEX: :float t
|
||||
#+CAPTION: Example code that simply adds a number to the ASCII value of a character and converts it back to a character.
|
||||
|
|
@ -3072,63 +3081,51 @@ def jump(alpha, n):
|
|||
return chr(adjusted)
|
||||
#+END_SRC
|
||||
|
||||
#+CAPTION: AST subtree corresponding to line 3 in Listing\nbsp{}[[lst:feedbacksubtreesample]], as generated by tree-sitter.
|
||||
#+ATTR_LATEX: :width 0.5\linewidth
|
||||
#+CAPTION: AST subtree corresponding to line 3 in Listing\nbsp{}[[lst:feedbacksubtreesample]] as generated by tree-sitter.
|
||||
#+NAME: fig:feedbacksubtree
|
||||
[[./diagrams/feedbacksubtree.svg]]
|
||||
|
||||
**** Pattern mining
|
||||
:PROPERTIES:
|
||||
:CREATED: [2023-11-20 Mon 13:33]
|
||||
:CUSTOM_ID: subsubsec:feedbackpredictiontreeminer
|
||||
:END:
|
||||
|
||||
After we have gathered a collection of subtrees for every annotation, we the need to mine patterns from these subtrees.
|
||||
=Treeminer=\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005] is an algorithm for discovering frequently occurring patterns in datasets of rooted, ordered and labelled trees.
|
||||
It does this by starting with a list of frequently occurring nodes, and then iteratively expanding the frequently occurring patterns.
|
||||
After collecting subtrees for each annotation, ECHO mines patterns from these subtrees using =TreeminerD=\nbsp{}[cite:@zakiEfficientlyMiningFrequent2005]: an algorithm for discovering frequently occurring patterns in datasets of rooted, ordered and labelled trees.
|
||||
=TreeminerD= starts with a list of frequently occurring nodes, and then iteratively expands the frequently occurring patterns.
|
||||
Patterns are embedded subtrees: the nodes in a pattern are a subset in the nodes of the tree, preserving the ancestor-descendant relationships and the left-to-right order of the nodes.
|
||||
An example of a valid pattern for the tree in Figure\nbsp{}[[fig:feedbacksubtree]] can be seen in Figure\nbsp{}[[fig:feedbackpattern]].
|
||||
An example of a valid pattern for the tree in Figure\nbsp{}[[fig:feedbacksubtree]] is shown in Figure\nbsp{}[[fig:feedbackpattern]].
|
||||
|
||||
#+ATTR_LATEX: :width 0.5\linewidth
|
||||
#+CAPTION: Valid pattern for the tree in Figure\nbsp{}[[fig:feedbacksubtree]].
|
||||
#+CAPTION: Indirect ancestor-descendant relationships have been marked with dashed lines.
|
||||
#+CAPTION: Indirect ancestor-descendant relationships are marked with dashed lines.
|
||||
#+NAME: fig:feedbackpattern
|
||||
[[./diagrams/feedbackpattern.svg]]
|
||||
|
||||
In the base =Treeminer= algorithm, frequently occurring means that the number of times the pattern occurs in all trees divided by the number of trees is greater than some predefined threshold.
|
||||
This threshold is called the =minimum support= parameter of the algorithm.
|
||||
|
||||
=TreeminerD= is a more efficient variant of the base =Treeminer= algorithm.
|
||||
It achieves this efficiency by not counting occurrences of a frequent pattern within a tree.
|
||||
Since we are not interested in this information for our method, it was an obvious choice to use the =TreeminerD= variant.
|
||||
In the =TreeminerD= algorithm, frequent means that the number of times the pattern occurs in all trees divided by the number of trees is greater than some predefined threshold.
|
||||
This is called the =minimum support= parameter of the algorithm.
|
||||
|
||||
We use a custom implementation of the =TreeminerD= algorithm, to find patterns in the AST subtrees for each annotation.
|
||||
In our implementation, we set the =minimum support= parameter to 0.8.
|
||||
This value was determined experimentally.
|
||||
Due to the exponential nature of the number of possible patterns in a tree, we only mine for patterns when there are at least three trees.
|
||||
|
||||
**** Weight assignment
|
||||
:PROPERTIES:
|
||||
:CREATED: [2023-11-22 Wed 14:39]
|
||||
:CUSTOM_ID: subsubsec:feedbackpredictionweights
|
||||
:END:
|
||||
|
||||
We now have a set of patterns corresponding to each annotation.
|
||||
Some patterns are more informative that others though.
|
||||
Therefore we assign weights to the patterns we obtain from =TreeminerD=
|
||||
ECHO now has a set of patterns corresponding to each annotation.
|
||||
However, some patterns are more informative that others.
|
||||
So it assigns weights to the patterns it gets from =TreeminerD=
|
||||
|
||||
Weights are assigned using two criteria.
|
||||
The first criterion is the size of the pattern (i.e., the number of nodes in the pattern), since a pattern with twenty nodes is much more specific than a pattern with only one node.
|
||||
The second criterion is the number of occurrences of a pattern across all annotations.
|
||||
If the pattern sets for all annotations contain a particular pattern, it can't be used reliably to determine which annotation should be predicted and is therefore given a lower weight.
|
||||
If the pattern sets for all annotations contain a particular pattern, it cannot be used reliably to determine which annotation should be predicted and is therefore given a lower weight.
|
||||
Weights are calculated using the formula below.
|
||||
\[\operatorname{weight}(pattern) = \frac{\operatorname{len}(pattern)}{\operatorname{\#occurences}(pattern)}\]
|
||||
|
||||
**** Pattern matching
|
||||
In addition to mining and weighting patterns, ECHO also determines a set of nodes that are unique to the subtrees of each annotation.
|
||||
This is done by taking the union of the nodes of all subtrees for that annotation, and then removing from that set any nodes that occur in the subtrees of at least three other annotations.
|
||||
This step does not require a minimum number of instances per annotation.
|
||||
|
||||
**** Ranking
|
||||
:PROPERTIES:
|
||||
:CREATED: [2024-02-01 Thu 14:25]
|
||||
:CUSTOM_ID: subsubsec:feedbackpredictionmatching
|
||||
:END:
|
||||
|
||||
After completing the steps above we have trained our model.
|
||||
To use our model, we need to know how to match patterns to subtrees.
|
||||
Having completed the above steps, ECHO has trained its model.
|
||||
To use the model, ECHO needs to know how to match patterns to subtrees.
|
||||
|
||||
To check whether a given pattern matches a given subtree, we iterate over all the nodes in the subtree.
|
||||
At the same time, we also iterate over the nodes in the pattern.
|
||||
|
|
@ -3139,7 +3136,7 @@ Moving up in the tree is more complicated.
|
|||
If the current depth and the depth of the last match (stored on the stack) are the same, we can move forwards in the pattern (and the subtree).
|
||||
If not, we need to check that we are still in the embedded subtree, otherwise we need to reset our position in the pattern to the start.
|
||||
Since subtrees can contain multiple instances of the same label, we also need to make sure that we can backtrack.
|
||||
Listings\nbsp{}[[lst:feedbackmatchingpseudocode1]]\nbsp{}and\nbsp{}[[lst:feedbackmatchingpseudocode2]] contain the full pseudocode for this algorithm.
|
||||
Listings\nbsp{}[[lst:feedbackmatchingpseudocode1]]\nbsp{}and\nbsp{}[[lst:feedbackmatchingpseudocode2]] give the full pseudocode for this algorithm.
|
||||
|
||||
#+ATTR_LATEX: :float t
|
||||
#+CAPTION: Pseudocode for checking whether a pattern matches a subtree.
|
||||
|
|
@ -3194,20 +3191,14 @@ find_in_subtree(subtree, current_subtree):
|
|||
return False
|
||||
#+END_SRC
|
||||
|
||||
Checking whether a pattern matches a subtree is an operation that needs to happen a lot.
|
||||
For some annotations, there are many patterns, and all patterns of all annotations are checked.
|
||||
Checking whether a pattern matches a subtree is an operation that ECHO has to perform many times.
|
||||
For some annotations there are many patterns, and all patterns of all annotations are checked.
|
||||
An important optimization we added was to run the algorithm in Listings\nbsp{}[[lst:feedbackmatchingpseudocode1]]\nbsp{}and\nbsp{}[[lst:feedbackmatchingpseudocode2]] only if the set of labels in the pattern is a subset of the labels in the subtree.
|
||||
|
||||
**** Annotation ranking
|
||||
:PROPERTIES:
|
||||
:CREATED: [2023-11-22 Wed 14:47]
|
||||
:CUSTOM_ID: subsubsec:feedbackpredictionsimilarity
|
||||
:END:
|
||||
|
||||
Given a model where we have weighted patterns for each annotation, and a method for matching patterns to subtrees, we can now put the two together to make a final ranking of the available annotations for a given line of code.
|
||||
We compute a match score for each annotation using the formula below.
|
||||
We calculate a match score for each annotation using the formula below.
|
||||
\[ \operatorname{score}(annotation) = \frac{\displaystyle\sum_{pattern \atop \in\, patterns} \begin{cases} \operatorname{weight}(pattern) & pattern \text{ matches} \\ 0 & \text{otherwise} \end{cases}}{\operatorname{len}(patterns)} \]
|
||||
The annotations are ranked according to this score.
|
||||
ECHO then ranks the annotations by combining the score and the percentage of nodes in the set of unique nodes for that annotation.
|
||||
|
||||
*** Results and discussion
|
||||
:PROPERTIES:
|
||||
|
|
@ -3215,39 +3206,52 @@ The annotations are ranked according to this score.
|
|||
:CUSTOM_ID: subsec:feedbackpredictionresults
|
||||
:END:
|
||||
|
||||
As a dataset to validate our method, we used Python code written by students for exercises from (different) exams.
|
||||
As a dataset to validate ECHO, we used Python code written by students for programming exercises from (different) exams.
|
||||
The dataset contains between 135 and 214 submissions per exercise.
|
||||
We initially split the datasets evenly into a training set and a test set.
|
||||
Each submission for a particular exercise is by a different student.
|
||||
We first split the datasets equally into a training set and a test set.
|
||||
This simulates the midpoint of an assessment session for the exercise.
|
||||
During testing, we let our model suggest annotations for each of the lines that had an actual annotation associated with it in the test set.
|
||||
We evaluate at which position the correct annotation is ranked.
|
||||
We only look at the top five to give us a good idea of how useful the suggested ranking would be in practice: if an annotation is not in the top five, we would expect the reviewer to have to search for it instead of directly selecting it from the suggested ranking.
|
||||
During testing, we let our model suggest annotations for each of the lines that had an actual annotation associated with them in the test set.
|
||||
We evaluate where the correct annotation is ranked.
|
||||
We only look at the top five to get a good idea of how useful the suggested ranking would be in practice: if an annotation is not in the top five, we would expect the reviewer to have to search for it manually, rather than selecting it directly from the suggested ranking.
|
||||
|
||||
We first ran Pylint[fn:: https://www.pylint.org/] (version 3.1.0) on the student submissions.
|
||||
Pylint is a static code analyser for Python that checks for erros and code smells, and enforces a standard programming style.
|
||||
We first ran Pylint[fn:: https://www.pylint.org/] (version 3.1.0) on the students' submissions.
|
||||
Pylint is a static code analyser for Python that checks for errors and code smells, and enforces a standard programming style.
|
||||
We used Pylint's machine annotations as our training and test data.
|
||||
We test per exercise since that's our main use case for this method, but also perform one test where all submissions of all exercises are combined.
|
||||
We test per exercise because that's our main use case for this method, but we also run a test that combines all submissions from all exercises.
|
||||
An overview of some annotation statistics for the data generated by Pylint can be found in Table\nbsp{}[[tab:feedbackresultsdatasetpylint]].
|
||||
|
||||
For a second experiment, we used the manual annotations left by human reviewers on student code in Dodona.
|
||||
Exercises were reviewed by different people, but all submissions for an exercise were reviewed by the same person.
|
||||
The reviewers were not aware at the time this method would be developed.
|
||||
In this case, there is no combined test, since the set of annotations used is also different for each exercise.
|
||||
#+CAPTION: Statistics of Pylint annotations for the programming exercises used in the benchmark.
|
||||
#+NAME: tab:feedbackresultsdataset
|
||||
| Exercise | subm. | ann. | inst. | max | avg |
|
||||
|-----------------------+-------+------+-------+-----+-------|
|
||||
| <l> | <r> | <r> | <r> | <r> | <r> |
|
||||
| A last goodbye | 135 | 25 | 189 | 29 | 7.56 |
|
||||
| Symbolic | 141 | 28 | 277 | 66 | 9.89 |
|
||||
| Narcissus cipher | 144 | 29 | 148 | 24 | 5.10 |
|
||||
| Cocktail bar | 211 | 31 | 162 | 29 | 5.23 |
|
||||
| Anthropomorphic emoji | 214 | 24 | 144 | 40 | 6.00 |
|
||||
| Hermit | 194 | 82 | 388 | 59 | 6.80 |
|
||||
| Combined | 1039 | 82 | 1479 | 196 | 18.04 |
|
||||
|
||||
We distinguish between these two sources of annotations because we expect Pylint to be more consistent both in when it places an instance of an annotation and also where it places the instance.
|
||||
Most linting annotations are detected by explicit pattern matching in the AST, so we expect our implicit pattern matching to work fairly well.
|
||||
However, we want to skip this explicit pattern matching for manual annotations because of the time required to assemble them and the fact that annotations will often be specific to a particular exercise and to a particular reviewer.
|
||||
Therefore we also test on manual annotations.
|
||||
Manual annotations are expected to be more inconsistent because reviewers may miss a problem in one student's code that they annotated in another student's code, or they may not place instances of a particular annotation in consistent locations.
|
||||
In a second experiment, we used the manual annotations left by human reviewers on student code in Dodona.
|
||||
Exercises were reviewed by different people, but all submissions for a specific exercise were reviewed by the same person.
|
||||
The reviewers were not aware of ECHO at the time they reviewed the submissions.
|
||||
In this case there is no combined test as the set of annotations used is different for each exercise.
|
||||
|
||||
We distinguish between these two sources of annotations because we expect Pylint to be more consistent in both when it places an instance of an annotation and also where it places the instance.
|
||||
Most linting annotations are detected by explicit pattern matching in the AST, so we expect the implicit pattern matching to work fairly well.
|
||||
However, we want to skip this explicit pattern matching for manual annotations because of the time it takes to compile them and the fact that annotations are often specific to a particular exercise and reviewer.
|
||||
Therefore, we also test on manual annotations.
|
||||
Manual annotations are expected to be more inconsistent because reviewers may miss a problem in one student's code that they have annotated in another student's code, or they may not place instances of a particular annotation in consistent locations.
|
||||
The method by which human reviewers place an annotation is also much more implicit than Pylint's pattern matching.
|
||||
|
||||
Exercises have between 55 and 469 instances of manual annotations.
|
||||
The reviewed programming exercises have between 55 and 469 instances of manual annotations.
|
||||
The number of distinct annotations varies between 7 and 34 per exercise.
|
||||
Table\nbsp{}[[tab:feedbackresultsdataset]] gives an overview of some of the features of the dataset.
|
||||
Timings mentioned in this section were measured on a 2019 Macbook Pro with a 1.4GHz Intel quad-core processor and 16 GB of RAM.
|
||||
Table\nbsp{}[[tab:feedbackresultsdataset]] gives an overview of some of characteristics of the dataset.
|
||||
Timings mentioned in this section were measured on a 2022 Dell laptop with a 3GHz Intel quad-core processor and 32 GB of RAM.
|
||||
|
||||
#+CAPTION: Statistics of manually added annotations for the exercises used in the benchmark.
|
||||
#+CAPTION: Max is the maximum amount of instances per annotation.
|
||||
#+CAPTION: Avg is the average amount of instances per annotation.
|
||||
#+CAPTION: Statistics of manually added annotations for the programming exercises used in the benchmark.
|
||||
#+NAME: tab:feedbackresultsdataset
|
||||
| Exercise | subm. | ann. | inst. | max | avg |
|
||||
|-----------------------+-------+------+-------+-----+-------|
|
||||
|
|
@ -3266,40 +3270,36 @@ Timings mentioned in this section were measured on a 2019 Macbook Pro with a 1.4
|
|||
:END:
|
||||
|
||||
We will first discuss the results for the Pylint annotations.
|
||||
During the experiment, a few Pylint annotations not related to the actual code were left out to avoid skewing the results.
|
||||
During the experiment, a few Pylint annotations that are not related to the structure of the code were omitted to avoid distorting the results.
|
||||
These are "line too long", "trailing whitespace", "trailing newlines", "missing module docstring", "missing class docstring", and "missing function docstring".
|
||||
Depending on the exercise, the actual annotation is ranked among the top five annotations for 45% to 75% of all test instances (Figure\nbsp{}[[fig:feedbackpredictionpylintglobal]]).
|
||||
The annotation is even ranked first for 19% to 59% of all test instances.
|
||||
Interestingly, the method mostly performs worse when the instances for all exercises are combined.
|
||||
This highlights the fact that our method is most useful in the context where similar code needs to be reviewed many times.
|
||||
For the submissions and instances in the training set, training took 1.5 to 52 seconds for an exercise.
|
||||
The entire testing phase took between 4 seconds to 9.5 minutes.
|
||||
The average time of one prediction ranges between 30 milliseconds and 6 seconds.
|
||||
The minima range between 6 milliseconds and 4 seconds, the maxima between 127 milliseconds and 55 seconds.
|
||||
Note that these are very wide ranges.
|
||||
These big differences can be explained through the number of patterns that are found for the annotations.
|
||||
If there are annotations with a very large amount of patterns, this will be reflected in both the training and testing time.
|
||||
Depending on the exercise, the actual annotation is ranked among the top five annotations in 45% to 77% of all test instances (Figure\nbsp{}[[fig:feedbackpredictionpylintglobal]]).
|
||||
The annotation is even ranked first for 23% to 52% of all test instances.
|
||||
Interestingly, the method performs worse when the instances for all exercises are combined.
|
||||
This highlights the fact that ECHO is most useful in the context of reviewing similar code many times.
|
||||
For the submissions and instances in the training set, training took between 70 and 245 milliseconds to process all submissions and instances for an exercise.
|
||||
The entire test phase took between 30 and 180 milliseconds per exercise.
|
||||
Individual predictions never exceed 15 milliseconds.
|
||||
|
||||
#+CAPTION: Predictive accuracy for suggesting instances of Pylint annotations.
|
||||
#+CAPTION: Numbers on the right are the total number of annotations and instances respectively.
|
||||
#+CAPTION: The "Combined" test evaluated our method on the entire set of submissions for all exercises.
|
||||
#+CAPTION: Prediction accuracy for suggesting instances of Pylint annotations.
|
||||
#+CAPTION: The numbers on the right are the total number of annotations and instances respectively.
|
||||
#+CAPTION: The "Combined" test evaluated ECHO on the entire set of submissions for all exercises.
|
||||
#+NAME: fig:feedbackpredictionpylintglobal
|
||||
[[./images/feedbackpredictionpylintglobal.png]]
|
||||
|
||||
We have selected some interesting annotations for further inspection, some of which perform very well, and some of which perform worse (Figure\nbsp{}[[fig:feedbackpredictionpylintmessages]]).
|
||||
We selected these specific annotations to demonstrate interesting behaviours our method exhibits.
|
||||
The differences in performance can be explained by the content of the annotation and the underlying patterns Pylint is looking for.
|
||||
For example, the annotation "unused variable"[fn:: https://pylint.pycqa.org/en/latest/user_guide/messages/warning/unused-variable.html] performs rather poorly.
|
||||
This can be explained by the fact that we do not feed enough context to =TreeminerD= to find predictive patterns for this Pylint annotation.
|
||||
We have selected some interesting annotations for further inspection, some of which perform very well, and some of which perform less well (Figure\nbsp{}[[fig:feedbackpredictionpylintmessages]]).
|
||||
We chose these specific annotations to demonstrate interesting behaviours exhibited by ECHO.
|
||||
The differences in performance can be explained by the content of the annotation and the underlying patterns that Pylint is looking for.
|
||||
For example, the "unused variable"[fn:: https://pylint.pycqa.org/en/latest/user_guide/messages/warning/unused-variable.html] annotation performs poorly.
|
||||
This can be explained by the fact that we do not feed =TreeminerD= with enough context to find predictive patterns for this Pylint annotation.
|
||||
There are also annotations that can't be predicted at all, because no patterns are found.
|
||||
|
||||
Other annotations, like "consider using in"[fn:: https://pylint.pycqa.org/en/latest/user_guide/messages/refactor/consider-using-in.html], work very well.
|
||||
Other annotations, such as "consider using with"[fn:: https://pylint.pycqa.org/en/latest/user_guide/messages/refactor/consider-using-with.html], work very well.
|
||||
For these annotations, =TreeminerD= does have enough context to automatically determine the underlying patterns.
|
||||
The number of instances of an annotation in the training set also has an impact.
|
||||
Annotations which have only a few instances are generally predicted worse than those with lots of instances.
|
||||
The number of instances of an annotation in the training set also has an effect.
|
||||
Annotations with few instances are generally predicted worse than those with many instances.
|
||||
|
||||
#+CAPTION: Predictive accuracy for a selection of machine annotations by Pylint.
|
||||
#+CAPTION: Each line corresponds to a Pylint annotation, with the number of instances in the training and test set denoted in brackets after the name of the annotation.
|
||||
#+CAPTION: Prediction accuracy for a selection of Pylint machine annotations.
|
||||
#+CAPTION: Each line corresponds to a Pylint annotation, with the number of instances in the training and test sets given in parentheses after the annotation name.
|
||||
#+NAME: fig:feedbackpredictionpylintmessages
|
||||
[[./images/feedbackpredictionpylintmessages.png]]
|
||||
|
||||
|
|
@ -3309,119 +3309,115 @@ Annotations which have only a few instances are generally predicted worse than t
|
|||
:CUSTOM_ID: subsubsec:feedbackpredictionresultsrealworld
|
||||
:END:
|
||||
|
||||
For the annotations added by human reviewers, we applied two different scenarios to evaluate our method.
|
||||
Besides using the same 50/50 split between training and testing data as with the Pylint data, we also simulated how a human reviewer would use the method in practice by gradually increasing the training set and decreasing the test set as the reviewer progresses through the submissions during the assessment.
|
||||
At the start of the assessment no annotations are available and the first instance of an annotation that applies to a reviewed submission cannot be predicted.
|
||||
As more submissions have been reviewed, and more instances of annotations are placed on those submissions, the training set for modelling predictions on the next submission under review grows gradually.
|
||||
For the annotations added by human reviewers, we used two different scenarios to evaluate ECHO.
|
||||
In addition to using the same 50/50 split between training and test data as for the Pylint data, we also simulated how a human reviewer would use ECHO in practice by gradually increasing the training set and decreasing the test set as the reviewer progresses through the submissions during the assessment.
|
||||
At the start of the assessment, no annotations are available and the first instance of an annotation that applies to a reviewed submission cannot be predicted.
|
||||
As more submissions are reviewed and more instances of annotations are placed on those submissions, the training set for modelling predictions on the next submission under review gradually grows.
|
||||
|
||||
If we evenly split submissions and the corresponding annotations from a human reviewer into a training and a test set, the predictive accuracy is similar or even slightly better compared to the Pylint annotations (Figure\nbsp{}[[fig:feedbackpredictionrealworldglobal]]).
|
||||
The number of instances where the true annotation is ranked first is generally higher (between 36.4% and 60% depending on the exercise), and the number of instances where it is ranked in the top five is between 61% and 89% depending on the exercise.
|
||||
However, there is quite some variance between exercises.
|
||||
This can be explained by the quality of the data.
|
||||
For example, for the exercise "Symbolic", very few instances were placed for most annotations, which makes it difficult to predict additional instances.
|
||||
If we split the submissions and the corresponding annotations of a human reviewer equally into a training and a test set, the prediction accuracy is similar or even slightly better compared to the Pylint annotations (Figure\nbsp{}[[fig:feedbackpredictionrealworldglobal]]).
|
||||
The number of instances where the true annotation is ranked first is generally higher (between 29% and 63% depending on the exercise), and the number of instances where it is ranked in the top five is between 63% and 93% depending on the exercise.
|
||||
|
||||
#+CAPTION: Prediction results for six exercises that were designed and used for an exam, using manual annotations.
|
||||
#+CAPTION: Models were trained on half of the submissions from the dataset and tested on the other half of the submissions from the dataset.
|
||||
#+CAPTION: Numbers on the right are the total number of annotations and instances respectively.
|
||||
#+CAPTION: Prediction accuracy for suggesting instances of annotations by human reviewers.
|
||||
#+CAPTION: The numbers on the right are the total number of annotations and instances respectively.
|
||||
#+NAME: fig:feedbackpredictionrealworldglobal
|
||||
[[./images/feedbackpredictionrealworldglobal.png]]
|
||||
|
||||
For this experiment, training took between 1.2 and 16.7 seconds depending on the exercise.
|
||||
The entire testing phase took between 1.5 and 35 seconds depending on the exercise.
|
||||
In this experiment, training took between 67 milliseconds and 22.4 seconds depending on the exercise.
|
||||
The entire testi phase took between 49 milliseconds and 27 seconds, depending on the exercise.
|
||||
These evaluations were run on the same hardware as those for the machine annotations.
|
||||
For one prediction, average times range between 0.1 milliseconds and 1 second.
|
||||
The minima range between 0.1 milliseconds and 240 milliseconds and the maxima range between 0.2 milliseconds and 3 seconds.
|
||||
The explanation for these wide ranges remains the same as for the Pylint predictions: everything depends on the number of patterns found.
|
||||
For one prediction, the average time ranged from 0.1 milliseconds to 150 milliseconds and the maxima from 0.5 milliseconds to 2.8 seconds.
|
||||
The explanation for these wide ranges remains the same as for the Pylint predictions: it all depends on the number of patterns found.
|
||||
|
||||
These results show that we can predict reuse with an accuracy that is quite high at the midpoint of a reviewing session for an exercise.
|
||||
The accuracy depends on the amount of instances per annotation and the consistency of the reviewer.
|
||||
Looking at the underlying data, we can also see that short, atomic messages can be predicted very well, as hinted by\nbsp{}[cite/t:@moonsAtomicReusableFeedback2022].
|
||||
We will now look at the accuracy of our method over time, to test how the accuracy evolves as the reviewing session progresses.
|
||||
These results show that we can predict reuse with a fairly high accuracy at the midpoint of a review session for a programming exercise.
|
||||
The accuracy depends on the number of instances per annotation and the consistency of the reviewer.
|
||||
Looking at the underlying data, we can also see that short, atomic messages can be predicted very well, as suggested by\nbsp{}[cite/t:@moonsAtomicReusableFeedback2022].
|
||||
We will now look at the longitudinal prediction accuracy of ECHO, to test how accuracy evolves over the course of a review session.
|
||||
|
||||
For the next experiment, we introduce two specific categories of negative predictive outcomes, namely "No training instances" and "No patterns".
|
||||
"No training instances means that the annotation corresponding to the true instance had no instances in the test set, and therefore could never have been predicted.
|
||||
"No patterns" means that =TreeminerD= was unable to find any frequent patterns for the set of subtrees extracted from the annotation instances.
|
||||
This could be because the collection of subtrees is too diverse, or because we have only seen one or two instances of a particular annotation, in which case we can't run =TreeminerD=.
|
||||
We know beforehand that test instances of such annotations cannot be predicted.
|
||||
For the next experiment, we introduce two specific categories of negative prediction results, namely "No training instances" and "No patterns".
|
||||
"No training instances" means that the annotation corresponding to the true instance had no instances in the training set, and therefore could never have been predicted.
|
||||
"No patterns" means that =TreeminerD= was unable to find any frequent patterns for the set of subtrees extracted from the annotation instances (and there were also no nodes unique to this set of subtrees in the entire set of subtrees).
|
||||
This could be because the collection of subtrees is too diverse to have common patterns.
|
||||
|
||||
Figures\nbsp{}[[fig:feedbackpredictionrealworldsimulation1]],\nbsp{}[[fig:feedbackpredictionrealworldsimulation2]],\nbsp{}[[fig:feedbackpredictionrealworldsimulation3]]\nbsp{}and\nbsp{}[[fig:feedbackpredictionrealworldsimulation4]] show the results of this experiment for four of the exercises we used in the previous experiments.
|
||||
The two exercises that performed worse in the previous experiment were not taken into account for this experiment.
|
||||
Figures\nbsp{}[[fig:feedbackpredictionrealworldsimulation1]],\nbsp{}[[fig:feedbackpredictionrealworldsimulation2]],\nbsp{}[[fig:feedbackpredictionrealworldsimulation3]]\nbsp{}and\nbsp{}[[fig:feedbackpredictionrealworldsimulation4]] show the results of this experiment for four of the programming exercises used in the previous experiments.
|
||||
The "Symbolic" exercise was excluded due to its low number of unique annotations, while the "Hermit" exercise was excluded due to its poor performance in the previous experiment.
|
||||
We also excluded submissions that received no annotations during the human review process, which explains the lower number of submissions compared to the numbers in Table\nbsp{}[[tab:feedbackresultsdataset]].
|
||||
This experiment shows that while the review process requires some build-up before sufficient training instances are available, once a critical mass of training instances is reached, the accuracy for suggesting new instances of annotations reaches its maximal predictive power.
|
||||
This critical mass is reached after about 20 to 30 reviews, which is quite early in the reviewing process\nbsp{}(Figure\nbsp{}[[fig:feedbackpredictionrealworldevolution]]).
|
||||
This experiment shows that while the review process requires some time to build up before sufficient training instances are available, once a critical mass of training instances is reached, the accuracy for suggesting new instances of annotations reaches its maximum predictive power.
|
||||
This critical mass is reached after about 20 to 30 submissions reviewed, which is quite early in the review process\nbsp{}(Figure\nbsp{}[[fig:feedbackpredictionrealworldevolution]]).
|
||||
This means that a lot of time could be saved during the review process when ECHO is integrated into an online learning environment.
|
||||
The point at which the critical mass is reached will of course depend on the nature of the exercises and the consistency of the reviewer.
|
||||
|
||||
#+CAPTION: Progression of the predictive accuracy for the exercise "A last goodbye" throughout the review process.
|
||||
#+CAPTION: Progression of the prediction accuracy for the "A last goodbye" exercise over the course of the review process.
|
||||
#+CAPTION: Predictions for instances whose annotation had no instances in the training set are classified as "No training instances.
|
||||
#+CAPTION: Predictions for instances whose annotation had no corresponding patterns in the model learned from the training set are classified as "No patterns".
|
||||
#+CAPTION: The graph on the right shows the number of annotations present with at least one instance in the training set.
|
||||
#+NAME: fig:feedbackpredictionrealworldsimulation1
|
||||
[[./images/feedbackpredictionrealworldsimulation1.png]]
|
||||
|
||||
#+CAPTION: Progression of the predictive accuracy for the exercise "Narcissus cipher" throughout the review process.
|
||||
#+CAPTION: Progression of the prediction accuracy for the "Narcissus cipher" exercise over the course of the review process.
|
||||
#+CAPTION: Predictions for instances whose annotation had no instances in the training set are classified as "No training instances.
|
||||
#+CAPTION: Predictions for instances whose annotation had no corresponding patterns in the model learned from the training set are classified as "No patterns".
|
||||
#+CAPTION: The graph on the right shows the number of annotations present with at least one instance in the training set.
|
||||
#+NAME: fig:feedbackpredictionrealworldsimulation2
|
||||
[[./images/feedbackpredictionrealworldsimulation2.png]]
|
||||
|
||||
#+CAPTION: Progression of the predictive accuracy for the exercise "Cocktail bar" throughout the review process.
|
||||
#+CAPTION: Progression of the prediction accuracy for the "Cocktail bar" exercise over the course of the review process.
|
||||
#+CAPTION: Predictions for instances whose annotation had no instances in the training set are classified as "No training instances.
|
||||
#+CAPTION: Predictions for instances whose annotation had no corresponding patterns in the model learned from the training set are classified as "No patterns".
|
||||
#+CAPTION: The graph on the right shows the number of annotations present with at least one instance in the training set.
|
||||
#+NAME: fig:feedbackpredictionrealworldsimulation3
|
||||
[[./images/feedbackpredictionrealworldsimulation3.png]]
|
||||
|
||||
#+CAPTION: Progression of the predictive accuracy for the exercise "Anthropomorphic emoji" throughout the review process.
|
||||
#+CAPTION: Progression of the prediction accuracy for the "Anthropomorphic emoji" exercise over the course of the review process.
|
||||
#+CAPTION: Predictions for instances whose annotation had no instances in the training set are classified as "No training instances.
|
||||
#+CAPTION: Predictions for instances whose annotation had no corresponding patterns in the model learned from the training set are classified as "No patterns".
|
||||
#+CAPTION: The graph on the right shows the number of annotations present with at least one instance in the training set.
|
||||
#+NAME: fig:feedbackpredictionrealworldsimulation4
|
||||
[[./images/feedbackpredictionrealworldsimulation4.png]]
|
||||
|
||||
#+CAPTION: Evolution of the percentage of suggestions that are in the top\nbsp{}5.
|
||||
#+CAPTION: The percentages are quite stable after 20 to 30 submissions have been reviewed.
|
||||
#+CAPTION: Evolution of the percentage of suggestions that are ranked in the top\nbsp{}5.
|
||||
#+CAPTION: The percentages are fairly stable after 20 to 30 submissions have been reviewed.
|
||||
#+NAME: fig:feedbackpredictionrealworldevolution
|
||||
[[./images/feedbackpredictionrealworldevolution.png]]
|
||||
|
||||
As mentioned before, we are working with a slightly inconsistent dataset when using annotations by human reviewers.
|
||||
As mentioned above, we are working with a slightly inconsistent dataset when using annotations from human reviewers.
|
||||
They will sometimes miss an instance of an annotation, place it inconsistently, or unnecessarily create duplicate annotations.
|
||||
If this system is used in practice, the predictions could possibly be even better, since knowing about its existence might further motivate a reviewer to be more consistent in their reviews.
|
||||
The exercises were also reviewed by different people, which could also be an explanation for the differences in accuracy of predictions between the exercises.
|
||||
If ECHO is used in practice, the predictions may be even better, as the knowledge of its existence may further motivate reviewers to be more consistent in their reviews.
|
||||
The programming exercises were also reviewed by different people, which may also explain the differences in prediction accuracy between the exercises.
|
||||
|
||||
To evaluate the performance of our model for these experiments, we measure the training times, and the times required for each prediction (this corresponds to a teacher wanting to add an annotation to a line in practice).
|
||||
Figures\nbsp{}[[fig:feedbackpredictionrealworldtimings1]],\nbsp{}[[fig:feedbackpredictionrealworldtimings2]],\nbsp{}[[fig:feedbackpredictionrealworldtimings3]],\nbsp{}and\nbsp{}[[fig:feedbackpredictionrealworldtimings4]] show the timings for these experiments.
|
||||
Like in the previous experiments, we can see that there is a big difference between exercises: for two of the exercises, the training time does not exceed 10 seconds, where for the others the training times go up to about a minute.
|
||||
Prediction times show a similar diversity: for one exercise the prediction times never exceed 3 seconds, while for the others there are some outliers up to 25 seconds.
|
||||
The average prediction times never exceed a few seconds though.
|
||||
To evaluate the performance of ECHO for these experiments, we measure the training times, and the times required for each prediction (this corresponds to a reviewer wanting to add an annotation to a line in practice).
|
||||
Figures\nbsp{}[[fig:feedbackpredictionrealworldtimings1]],\nbsp{}[[fig:feedbackpredictionrealworldtimings2]],\nbsp{}[[fig:feedbackpredictionrealworldtimings3]],\nbsp{}and\nbsp{}[[fig:feedbackpredictionrealworldtimings4]] show the performance of running these experiments.
|
||||
As in the previous experiments, we can see that there is a considerable difference between the exercises.
|
||||
However, the training time only exceeds one seconds in a few cases and remains well below that in most cases
|
||||
The prediction times are mostly below 50 milliseconds, except for a few outliers.
|
||||
The average prediction time never exceeds 500 milliseconds.
|
||||
|
||||
The timings show that even though there are some outliers, most predictions can be performed quickly enough to make this an interactive system.
|
||||
The outliers also correspond with higher training times, indicating this is mainly caused by a high number of underlying patterns for some annotations.
|
||||
The timings show that although there are some outliers, predictions can be made fast enough to make this an interactive system.
|
||||
The outliers also correspond to higher training times, indicating that this is mainly caused by a high number of underlying patterns for some annotations.
|
||||
Currently this process is also parallelized over the files, but in practice, the process could be parallelized over the patterns, which would speed up the prediction even more.
|
||||
Note that the training time can also go down given more training data.
|
||||
If there are more instances per annotation, the diversity in related subtrees will usually increase, which decreases the number of patterns that can be found, which also decreases the training time.
|
||||
Note that the training time may also decrease with more training data.
|
||||
If there are more instances per annotation, the diversity in the related subtrees will usually increase, which reduces the number of patterns that can be found and thus reduces the training time.
|
||||
|
||||
#+CAPTION: Time needed for training and testing throughout the review process for the exercise "A last goodbye".
|
||||
#+CAPTION: Time needed for training and testing during the entire review process for the exercise "A last goodbye".
|
||||
#+CAPTION: Top: training time.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time required for predicting a single instance.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time needed to predict a single instance.
|
||||
#+NAME: fig:feedbackpredictionrealworldtimings1
|
||||
[[./images/feedbackpredictionrealworldtimings1.png]]
|
||||
|
||||
#+CAPTION: Time needed for training and testing throughout the review process for the exercise "Narcissus cipher".
|
||||
#+CAPTION: Time needed for training and testing during the entire review process for the exercise "Narcissus cipher".
|
||||
#+CAPTION: Top: training time.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time required for predicting a single instance.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time needed to predict a single instance.
|
||||
#+NAME: fig:feedbackpredictionrealworldtimings2
|
||||
[[./images/feedbackpredictionrealworldtimings2.png]]
|
||||
|
||||
#+CAPTION: Time needed for training and testing throughout the review process for the exercise "Cocktail bar".
|
||||
#+CAPTION: Time needed for training and testing during the entire review process for the exercise "Cocktail bar".
|
||||
#+CAPTION: Top: training time.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time required for predicting a single instance.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time needed to predict a single instance.
|
||||
#+NAME: fig:feedbackpredictionrealworldtimings3
|
||||
[[./images/feedbackpredictionrealworldtimings3.png]]
|
||||
|
||||
#+CAPTION: Time needed for training and testing throughout the review process for the exercise "Anthropomorphic emoji".
|
||||
#+CAPTION: Time needed for training and testing during the entire review process for the exercise "Anthropomorphic emoji".
|
||||
#+CAPTION: Top: training time.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time required for predicting a single instance.
|
||||
#+CAPTION: Bottom: average (orange dot) and range (blue line) of time needed to predict a single instance.
|
||||
#+NAME: fig:feedbackpredictionrealworldtimings4
|
||||
[[./images/feedbackpredictionrealworldtimings4.png]]
|
||||
|
||||
|
|
@ -3431,32 +3427,32 @@ If there are more instances per annotation, the diversity in related subtrees wi
|
|||
:CUSTOM_ID: subsec:feedbackpredictionconclusion
|
||||
:END:
|
||||
|
||||
We presented a prediction method to assist human reviewers in giving feedback while reviewing students submissions for an exercise by reusing annotations.
|
||||
Improving annotation reuse can be both a time-saver, and improve the consistency with which feedback is given.
|
||||
The latter itself might further improve the accuracy of the predictions when the strategy is applied during the review process.
|
||||
We presented ECHO as a predictive method to assist human reviewers in giving feedback when reviewing students submissions to a programming exercise by reusing annotations.
|
||||
Improving the reuse of annotations can both save time and improve the consistency with which feedback is given.
|
||||
The latter in itself might further improve the accuracy of predictions if the strategy is applied during the review process.
|
||||
|
||||
The method has already shown promising results.
|
||||
We validated the framework both by predicting automated linting annotations to establish a baseline and by predicting annotations from human reviewers.
|
||||
The method has about the same predictive accuracy for machine (Pylint) and human annotations.
|
||||
Thus, we can give a positive answer to our research question that reuse of feedback given previously by a human reviewer can be predicted with high accuracy on a particular line of a new submission.
|
||||
ECHO has already shown promising results.
|
||||
We have validated the framework both by predicting automated linting annotations to establish a baseline, and by predicting annotations from human reviewers.
|
||||
The method has about the same prediction accuracy for machine (Pylint) and human annotations.
|
||||
Thus, we can answer both our research questions in an affirmative way, meaning that the reuse of feedback previously given by a human reviewer on a particular line of a new submission can be predicted with high accuracy (RQ1), and that this can be done fast enough to assist human reviewers in future reviews (RQ2).
|
||||
|
||||
We can conclude that the proposed method has achieved the desired objective as expressed in the research question.
|
||||
Having this method at hand immediately raises some possible follow-up work.
|
||||
Currently, the proposed model is reactive: we suggest a ranking of most likely annotations when a reviewer wants to add an annotation to a particular line of a submission.
|
||||
By introducing a confidence score, we could check beforehand if we have a confident match for each line, and then immediately propose those suggestions to the reviewer.
|
||||
Having ECHO at hand immediately raises some opportunities fo follow-up work.
|
||||
Currently, the proposed model is reactive: we suggest a ranking of the most likely annotations when a reviewer wants to add an annotation to a particular line of a submission.
|
||||
By introducing a confidence score, we could check beforehand whether we have a confident match for each line, and then immediately propose these suggestions to the reviewer.
|
||||
Whether or not a reviewer accepts these suggestions could then also be used as an input to the model.
|
||||
This could also have an extra advantage, since it could help reviewers be more consistent in where and when they place annotations.
|
||||
This could also have an additional benefit by helping reviewers to be more consistent in where and when they place annotations.
|
||||
|
||||
Annotations that don't lend themselves well to prediction also need further investigation.
|
||||
The context used could be expanded, although the important caveat here is that the method still needs to maintain sufficient performance.
|
||||
We could also consider applying some of the source code pattern mining techniques proposed by\nbsp{}[cite/t:@phamMiningPatternsSource2019] to achieve further speed improvements.
|
||||
This could help with the outliers seen in the timing data.
|
||||
Another important aspect that was explicitly left out of the scope of this chapter was its integration into a learning platform and user testing.
|
||||
Another important aspect that was explicitly outside of the scope of this chapter was the integration of ECHO into a learning platform and user testing.
|
||||
|
||||
Of course, alternative methods could also be considered.
|
||||
One cannot overlook the rise of Large Language Models (LLMs) and the way they could contribute to this problem.
|
||||
LLMs can generate feedback for students, based on their code and a well-chosen system prompt.
|
||||
Fine-tuning of a model with feedback already given is another option.
|
||||
One cannot overlook the rise of Large Language Models (LLMs) and the way in which they could contribute to this problem.
|
||||
LLMs can generate feedback for students based on their submitted solution and a well-chosen system prompt.
|
||||
Fine-tuning of an LLM with feedback already given is another possibility.
|
||||
Future applications could also combine user generated and LLM generated feedback, showing human reviewers the source of the feedback during their reviews.
|
||||
|
||||
* Looking ahead: opportunities and challenges
|
||||
:PROPERTIES:
|
||||
|
|
|
|||
|
|
@ -26,17 +26,42 @@
|
|||
inkscape:pagecheckerboard="0"
|
||||
inkscape:deskcolor="#d1d1d1"
|
||||
inkscape:document-units="mm"
|
||||
inkscape:zoom="0.94139615"
|
||||
inkscape:cx="391.97101"
|
||||
inkscape:cy="295.8372"
|
||||
inkscape:window-width="1534"
|
||||
inkscape:zoom="1.8827923"
|
||||
inkscape:cx="270.87428"
|
||||
inkscape:cy="93.212618"
|
||||
inkscape:window-width="2302"
|
||||
inkscape:window-height="2132"
|
||||
inkscape:window-x="10"
|
||||
inkscape:window-y="10"
|
||||
inkscape:window-x="11"
|
||||
inkscape:window-y="11"
|
||||
inkscape:window-maximized="0"
|
||||
inkscape:current-layer="layer1"
|
||||
inkscape:export-bgcolor="#ffffffff" /><defs
|
||||
id="defs1"><rect
|
||||
x="496.50118"
|
||||
y="88.720869"
|
||||
width="55.162418"
|
||||
height="23.122655"
|
||||
id="rect6" /><rect
|
||||
x="484.62553"
|
||||
y="222.32189"
|
||||
width="74.699972"
|
||||
height="22.047954"
|
||||
id="rect5" /><rect
|
||||
x="143.35424"
|
||||
y="217.61437"
|
||||
width="117.63612"
|
||||
height="19.502284"
|
||||
id="rect4" /><rect
|
||||
x="28.82811"
|
||||
y="145.70073"
|
||||
width="105.86213"
|
||||
height="24.006482"
|
||||
id="rect3" /><rect
|
||||
x="16.898522"
|
||||
y="80.401277"
|
||||
width="38.960999"
|
||||
height="23.896522"
|
||||
id="rect2" /><rect
|
||||
x="133.56245"
|
||||
y="130.90417"
|
||||
width="94.291634"
|
||||
|
|
@ -199,6 +224,11 @@
|
|||
inkscape:groupmode="layer"
|
||||
id="layer1"
|
||||
transform="translate(-119.58348,-121.61611)"><path
|
||||
d="m 503.09395,103.27871 -3.76,-10.016004 h -0.064 q 0.032,0.32 0.048,0.816 0.032,0.496 0.048,1.088 0.016,0.576 0.016,1.184 v 6.928004 h -1.328 V 91.854706 h 2.128 l 3.52,9.360004 h 0.064 l 3.584,-9.360004 h 2.112 v 11.424004 h -1.424 v -7.024004 q 0,-0.56 0.016,-1.104 0.016,-0.56 0.048,-1.04 0.032,-0.496 0.048,-0.832 h -0.064 l -3.808,10.000004 z m 12.528,-8.720004 q 1.568,0 2.32,0.688 0.752,0.688 0.752,2.192 v 5.840004 h -1.024 l -0.272,-1.216 h -0.064 q -0.368,0.464 -0.768,0.784 -0.384,0.304 -0.896,0.448 -0.496,0.144 -1.216,0.144 -0.768,0 -1.392,-0.272 -0.608,-0.272 -0.96,-0.832 -0.352,-0.576 -0.352,-1.44 0,-1.280004 1.008,-1.968004 1.008,-0.704 3.104,-0.768 l 1.456,-0.048 v -0.512 q 0,-1.072 -0.464,-1.488 -0.464,-0.416 -1.312,-0.416 -0.672,0 -1.28,0.208 -0.608,0.192 -1.136,0.448 l -0.432,-1.056 q 0.56,-0.304 1.328,-0.512 0.768,-0.224 1.6,-0.224 z m 0.416,4.576 q -1.6,0.064 -2.224,0.512 -0.608,0.448004 -0.608,1.264004 0,0.72 0.432,1.056 0.448,0.336 1.136,0.336 1.088,0 1.808,-0.592 0.72,-0.608 0.72,-1.856004 v -0.768 z m 8.17601,3.152004 q 0.32,0 0.656,-0.048 0.336,-0.064 0.544,-0.128 v 1.072 q -0.224,0.112 -0.64,0.176 -0.416,0.08 -0.8,0.08 -0.672,0 -1.248,-0.224 -0.56,-0.24 -0.912,-0.816 -0.352,-0.576 -0.352,-1.616 v -4.992004 h -1.216 v -0.672 l 1.232,-0.56 0.56,-1.824 h 0.832 v 1.968 h 2.48 v 1.088 h -2.48 v 4.960004 q 0,0.784 0.368,1.168 0.384,0.368 0.976,0.368 z m 6.352,1.152 q -1.136,0 -2.032,-0.464 -0.88,-0.464 -1.392,-1.44 -0.496,-0.976 -0.496,-2.496004 0,-1.584 0.528,-2.576 0.528,-0.992 1.424,-1.456 0.912,-0.464 2.064,-0.464 0.656,0 1.264,0.144 0.608,0.128 0.992,0.32 l -0.432,1.168 q -0.384,-0.144 -0.896,-0.272 -0.512,-0.128 -0.96,-0.128 -0.864,0 -1.424,0.368 -0.56,0.368 -0.832,1.088 -0.272,0.72 -0.272,1.792 0,1.024004 0.272,1.744004 0.272,0.72 0.816,1.088 0.544,0.368 1.36,0.368 0.704,0 1.232,-0.144 0.544,-0.144 0.992,-0.352 v 1.248 q -0.432,0.224 -0.96,0.336 -0.512,0.128 -1.248,0.128 z m 5.64799,-8.752004 q 0,0.304 -0.032,0.624 -0.016,0.32 -0.048,0.576 h 0.096 q 0.272,-0.448 0.688,-0.736 0.416,-0.288 0.928,-0.432 0.512,-0.16 1.056,-0.16 1.04,0 1.728,0.336 0.704,0.32 1.056,1.008 0.352,0.688 0.352,1.792 v 5.584004 h -1.392 v -5.488004 q 0,-1.04 -0.464,-1.552 -0.464,-0.512 -1.456,-0.512 -0.96,0 -1.504,0.368 -0.544,0.352 -0.784,1.056 -0.224,0.688 -0.224,1.696 v 4.432004 h -1.408 V 91.118706 h 1.408 z m 9.36,4.944 q 0,-0.608 0.112,-1.04 0.128,-0.448 0.416,-0.848 0.304,-0.4 0.816,-0.832 0.624,-0.528 0.96,-0.88 0.352,-0.352 0.496,-0.688 0.144,-0.336 0.144,-0.816 0,-0.768 -0.496,-1.184 -0.496,-0.416 -1.44,-0.416 -0.784,0 -1.392,0.208 -0.608,0.192 -1.168,0.464 l -0.496,-1.12 q 0.64,-0.336 1.408,-0.56 0.784,-0.224 1.744,-0.224 1.52,0 2.352,0.752 0.832,0.752 0.832,2.048 0,0.72 -0.24,1.232 -0.224,0.496 -0.656,0.928 -0.416,0.416 -0.992,0.896 -0.528,0.448 -0.816,0.784 -0.272,0.336 -0.368,0.672 -0.096,0.32 -0.096,0.784 v 0.272004 h -1.12 z m -0.368,2.784004 q 0,-0.592 0.272,-0.832 0.288,-0.24 0.72,-0.24 0.4,0 0.688,0.24 0.288,0.24 0.288,0.832 0,0.576 -0.288,0.832 -0.288,0.256 -0.688,0.256 -0.432,0 -0.72,-0.256 -0.272,-0.256 -0.272,-0.832 z"
|
||||
id="text5"
|
||||
style="font-size:16px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans';white-space:pre"
|
||||
transform="translate(116.47971,121.61611)"
|
||||
aria-label="Match?" /><path
|
||||
d="m 639.26875,134.19151 a 4,4 0 0 1 4,4 4,4 0 0 1 -4,4 4,4 0 0 1 -4,-4 4,4 0 0 1 4,-4 m 0,10 c 4.42,0 8,1.79 8,4 v 2 h -16 v -2 c 0,-2.21 3.58,-4 8,-4 z"
|
||||
id="path1" /><path
|
||||
d="m 582.13638,221.41487 c -0.13,0 -0.26,0 -0.39,0.04 l -1.61,-3.25 c 0.45,-0.45 0.75,-1.08 0.75,-1.79 0,-1.38 -1.12,-2.5 -2.5,-2.5 -0.13,0 -0.25,0 -0.39,0.04 l -1.63,-3.25 c 0.48,-0.45 0.77,-1.08 0.77,-1.79 0,-1.38 -1.12,-2.5 -2.5,-2.5 -1.38,0 -2.5,1.12 -2.5,2.5 0,0.71 0.29,1.34 0.76,1.79 l -1.62,3.25 c -0.14,-0.04 -0.26,-0.04 -0.39,-0.04 -1.38,0 -2.5,1.12 -2.5,2.5 0,0.71 0.3,1.34 0.75,1.79 l -1.61,3.25 c -0.13,-0.04 -0.26,-0.04 -0.39,-0.04 -1.38,0 -2.5,1.12 -2.5,2.5 0,1.38 1.12,2.5 2.5,2.5 1.38,0 2.5,-1.12 2.5,-2.5 0,-0.7 -0.29,-1.34 -0.76,-1.79 l 1.62,-3.25 c 0.14,0.04 0.26,0.04 0.39,0.04 0.13,0 0.25,0 0.39,-0.04 l 1.63,3.25 c -0.47,0.45 -0.77,1.09 -0.77,1.79 0,1.38 1.12,2.5 2.5,2.5 1.38,0 2.5,-1.12 2.5,-2.5 0,-1.38 -1.12,-2.5 -2.5,-2.5 -0.13,0 -0.26,0 -0.39,0.04 l -1.61,-3.25 c 0.46,-0.45 0.75,-1.08 0.75,-1.79 0,-0.71 -0.29,-1.34 -0.75,-1.79 l 1.61,-3.25 c 0.13,0.04 0.26,0.04 0.39,0.04 0.13,0 0.26,0 0.39,-0.04 l 1.61,3.25 c -0.45,0.45 -0.75,1.09 -0.75,1.79 0,1.38 1.12,2.5 2.5,2.5 0.13,0 0.25,0 0.39,-0.04 l 1.63,3.25 c -0.47,0.45 -0.77,1.09 -0.77,1.79 0,1.38 1.12,2.5 2.5,2.5 1.38,0 2.5,-1.12 2.5,-2.5 0,-1.38 -1.12,-2.5 -2.5,-2.5 z"
|
||||
|
|
@ -223,7 +253,7 @@
|
|||
id="path1-1-5" /><path
|
||||
d="m 147.78408,312.78657 h -1.5 v -4 c 0,-1.11 -0.9,-2 -2,-2 h -4 v -1.5 a 2.5,2.5 0 0 0 -2.5,-2.5 2.5,2.5 0 0 0 -2.5,2.5 v 1.5 h -4 a 2,2 0 0 0 -2,2 v 3.8 h 1.5 c 1.5,0 2.7,1.2 2.7,2.7 0,1.5 -1.2,2.7 -2.7,2.7 h -1.5 v 3.8 a 2,2 0 0 0 2,2 h 3.8 v -1.5 c 0,-1.5 1.2,-2.7 2.7,-2.7 1.5,0 2.7,1.2 2.7,2.7 v 1.5 h 3.8 a 2,2 0 0 0 2,-2 v -4 h 1.5 a 2.5,2.5 0 0 0 2.5,-2.5 2.5,2.5 0 0 0 -2.5,-2.5 z"
|
||||
id="path1-1-6" /></g><path
|
||||
d="m 636.3202,321.3581 h 12 v 1 h 3 v 2 h -3 v 2 h 3 v 2 h -3 v 2 h 3 v 2 h -3 v 2 h 3 v 2 h -3 v 1 h -12 v -1 h -3 v -2 h 3 v -2 h -3 v -2 h 3 v -2 h -3 v -2 h 3 v -2 h -3 v -2 h 3 v -1 m 5,11 v 3 h 1 v -3 h -1 m 2,0 v 3 h 1 v -3 h -1 m 2,0 v 3 h 1 v -3 z"
|
||||
d="m 633.35336,319.7585 h 12 v 1 h 3 v 2 h -3 v 2 h 3 v 2 h -3 v 2 h 3 v 2 h -3 v 2 h 3 v 2 h -3 v 1 h -12 v -1 h -3 v -2 h 3 v -2 h -3 v -2 h 3 v -2 h -3 v -2 h 3 v -2 h -3 v -2 h 3 v -1 m 5,11 v 3 h 1 v -3 h -1 m 2,0 v 3 h 1 v -3 h -1 m 2,0 v 3 h 1 v -3 z"
|
||||
id="path1-3" /><path
|
||||
d="m 636.2175,156.68051 -6.98397,6.98398 -3.20099,-3.20099 0.82062,-0.82062 2.38037,2.37455 6.16335,-6.15753 z"
|
||||
id="path1-2"
|
||||
|
|
@ -320,9 +350,6 @@
|
|||
style="fill:#000000;fill-opacity:1;stroke:#000000;stroke-width:0.852216;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle-8-76)"
|
||||
d="m 574.65898,168.07957 v 29.9481"
|
||||
id="path14-30" /><path
|
||||
style="fill:#000000;stroke:#000000;stroke-width:0.914531;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle-8)"
|
||||
d="m 623.84219,159.53915 -27.2592,-9.91803"
|
||||
id="path10" /><path
|
||||
style="fill:#000000;stroke:#000000;stroke-width:1.00157;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle)"
|
||||
d="m 198.90901,236.94821 103.22843,76.30967"
|
||||
id="path3" /><path
|
||||
|
|
@ -341,33 +368,35 @@
|
|||
style="fill:#000000;stroke:#000000;stroke-width:0.881701;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#DartArrow)"
|
||||
d="M 340.68104,328.0486 H 614.69917"
|
||||
id="path8" /><path
|
||||
style="font-weight:300;font-size:14.6667px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans Light';white-space:pre;stroke:#000000;stroke-width:1.00157"
|
||||
d="m 294.6704,344.06792 q 1.89201,0 2.83067,0.73334 0.93867,0.73333 0.93867,2.22934 0,0.748 -0.24933,1.34933 -0.23467,0.60134 -0.73334,1.02667 -0.484,0.41067 -1.24667,0.63067 -0.748,0.22 -1.77467,0.22 h -1.52533 v 4.28268 h -0.85067 v -10.47203 z m -0.088,0.748 h -1.672 v 4.70802 h 1.43733 q 1.01201,0 1.73067,-0.22 0.73334,-0.23467 1.10001,-0.76267 0.38133,-0.54267 0.38133,-1.46667 0,-1.17334 -0.71867,-1.71601 -0.71867,-0.54267 -2.25867,-0.54267 z m 8.69736,1.78934 q 1.32001,0 1.96534,0.64534 0.64534,0.64533 0.64534,2.05334 v 5.23601 h -0.63067 l -0.14667,-1.32001 h -0.044 q -0.27867,0.44001 -0.64533,0.77734 -0.352,0.32267 -0.86534,0.51333 -0.49867,0.176 -1.232,0.176 -0.73334,0 -1.276,-0.24933 -0.54267,-0.264 -0.83601,-0.748 -0.29333,-0.484 -0.29333,-1.188 0,-1.15867 0.95334,-1.76001 0.95333,-0.616 2.772,-0.704 l 1.42267,-0.0587 v -0.55733 q 0,-1.15867 -0.46933,-1.628 -0.45467,-0.48401 -1.36401,-0.48401 -0.60133,0 -1.15866,0.16134 -0.54267,0.14666 -1.10001,0.42533 l -0.264,-0.66 q 0.54267,-0.27867 1.20267,-0.45467 0.66,-0.176 1.364,-0.176 z m 0.45467,4.04801 q -1.48133,0.0733 -2.21467,0.528 -0.73333,0.44 -0.73333,1.33467 0,0.71867 0.45466,1.10001 0.45467,0.36666 1.23201,0.36666 1.24667,0 1.90667,-0.68933 0.67467,-0.68934 0.68933,-1.93601 v -0.76267 z m 6.84934,3.32934 q 0.32267,0 0.616,-0.044 0.29334,-0.0587 0.51334,-0.132 v 0.67467 q -0.22,0.088 -0.55734,0.14667 -0.32266,0.0587 -0.68933,0.0587 -0.64534,0 -1.1,-0.23466 -0.45467,-0.24934 -0.70401,-0.76267 -0.23466,-0.51334 -0.23466,-1.32 v -4.95735 h -1.14401 v -0.46933 l 1.14401,-0.264 0.308,-1.80401 h 0.528 v 1.86267 h 2.40534 v 0.67467 h -2.40534 v 4.92801 q 0,0.80667 0.32267,1.23201 0.32266,0.41066 0.99733,0.41066 z m 4.98668,0 q 0.32267,0 0.616,-0.044 0.29334,-0.0587 0.51334,-0.132 v 0.67467 q -0.22,0.088 -0.55734,0.14667 -0.32267,0.0587 -0.68933,0.0587 -0.64534,0 -1.10001,-0.23466 -0.45466,-0.24934 -0.704,-0.76267 -0.23466,-0.51334 -0.23466,-1.32 v -4.95735 h -1.14401 v -0.46933 l 1.14401,-0.264 0.308,-1.80401 h 0.528 v 1.86267 h 2.40534 v 0.67467 h -2.40534 v 4.92801 q 0,0.80667 0.32267,1.23201 0.32266,0.41066 0.99733,0.41066 z m 5.73468,-7.39201 q 1.01201,0 1.67201,0.46933 0.67467,0.45467 1.012,1.26134 0.352,0.80666 0.352,1.86267 v 0.55733 h -5.57335 q 0,1.56934 0.71867,2.39067 0.73334,0.82134 2.06801,0.82134 0.73333,0 1.276,-0.11733 0.54267,-0.11734 1.188,-0.41067 v 0.76267 q -0.58667,0.264 -1.15867,0.38133 -0.572,0.11733 -1.33467,0.11733 -1.15867,0 -1.98,-0.484 -0.80667,-0.484 -1.232,-1.37867 -0.41067,-0.90933 -0.41067,-2.112 0,-1.20267 0.396,-2.12667 0.396,-0.92401 1.15867,-1.45201 0.76267,-0.54266 1.848,-0.54266 z m 0,0.704 q -1.08533,0 -1.74533,0.71867 -0.66001,0.704 -0.77734,2.03867 h 4.69335 q 0,-0.82134 -0.23467,-1.43734 -0.23467,-0.616 -0.71867,-0.968 -0.46933,-0.352 -1.21734,-0.352 z m 8.34536,-0.68934 q 0.29333,0 0.55733,0.044 0.27867,0.0293 0.49867,0.088 l -0.11733,0.76267 q -0.23467,-0.0587 -0.46934,-0.088 -0.23466,-0.044 -0.51333,-0.044 -0.528,0 -0.95334,0.20533 -0.42533,0.20534 -0.73333,0.58667 -0.308,0.38134 -0.46934,0.90934 -0.16133,0.528 -0.16133,1.188 v 4.28268 h -0.836 v -7.80269 h 0.704 l 0.088,1.46667 h 0.044 q 0.19067,-0.45466 0.51333,-0.80667 0.33734,-0.36666 0.80667,-0.572 0.46934,-0.22 1.04134,-0.22 z m 6.072,-0.0147 q 1.33467,0 2.03867,0.68933 0.704,0.68934 0.704,2.17067 v 5.08935 h -0.82134 v -5.04535 q 0,-1.12933 -0.51333,-1.65733 -0.51333,-0.52801 -1.51067,-0.52801 -1.276,0 -1.92134,0.74801 -0.64533,0.748 -0.64533,2.22933 v 4.25335 h -0.82134 v -7.80269 h 0.66 l 0.132,1.34934 h 0.044 q 0.22,-0.42534 0.58667,-0.76267 0.36667,-0.33733 0.88,-0.528 0.52801,-0.20533 1.18801,-0.20533 z m 10.14936,5.88134 q 0,0.704 -0.352,1.20267 -0.352,0.49867 -1.02667,0.76267 -0.67466,0.24933 -1.64267,0.24933 -0.80667,0 -1.43733,-0.14666 -0.63067,-0.14667 -1.07067,-0.352 v -0.80667 q 0.528,0.264 1.188,0.44 0.66,0.16133 1.33467,0.16133 1.17334,0 1.68667,-0.38133 0.51333,-0.396 0.51333,-1.07067 0,-0.44 -0.24933,-0.71867 -0.23467,-0.29333 -0.71867,-0.51333 -0.46933,-0.23467 -1.17333,-0.46934 -0.71867,-0.24933 -1.27601,-0.49867 -0.55733,-0.264 -0.86533,-0.66 -0.308,-0.396 -0.308,-1.08533 0,-0.93867 0.76267,-1.46667 0.76266,-0.528 2.03867,-0.528 0.704,0 1.30533,0.132 0.61601,0.132 1.11467,0.352 l -0.29333,0.68933 q -0.45467,-0.20533 -1.02667,-0.33733 -0.572,-0.132 -1.144,-0.132 -0.924,0 -1.43734,0.32266 -0.51333,0.32267 -0.51333,0.93867 0,0.46934 0.23466,0.74801 0.24934,0.264 0.71867,0.45466 0.484,0.19067 1.15867,0.44 0.68934,0.23467 1.24667,0.49867 0.572,0.264 0.89467,0.67467 0.33733,0.41067 0.33733,1.1 z"
|
||||
id="text9"
|
||||
aria-label="Patterns" /><path
|
||||
style="font-weight:300;font-size:14.6667px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans Light';white-space:pre;stroke:#000000;stroke-width:1.00157"
|
||||
d="m 627.96423,356.12809 -3.76935,-9.57736 h -0.044 q 0.0293,0.264 0.0293,0.60134 0.0147,0.32266 0.0147,0.68933 0.0147,0.352 0.0147,0.73334 v 7.55335 h -0.792 v -10.47203 h 1.276 l 3.60801,9.18136 h 0.0587 l 3.65201,-9.18136 h 1.24666 v 10.47203 h -0.836 v -7.64135 q 0,-0.32267 0,-0.66001 0.0147,-0.33733 0.0293,-0.66 0.0147,-0.32266 0.0293,-0.60133 h -0.044 l -3.78401,9.56269 z m 14.59336,-3.90134 q 0,0.90933 -0.23467,1.65733 -0.22,0.73334 -0.67467,1.27601 -0.44,0.528 -1.1,0.82133 -0.66,0.29333 -1.51067,0.29333 -0.80667,0 -1.452,-0.27866 -0.63067,-0.29334 -1.08534,-0.82134 -0.45467,-0.54267 -0.68933,-1.29067 -0.23467,-0.748 -0.23467,-1.65733 0,-1.27601 0.41067,-2.17068 0.42533,-0.90933 1.21733,-1.39333 0.80667,-0.484 1.90667,-0.484 1.11467,0 1.87734,0.51333 0.77734,0.49867 1.17334,1.408 0.396,0.90934 0.396,2.12668 z m -6.11601,0 q 0,0.99733 0.27866,1.76 0.27867,0.748 0.86534,1.15867 0.58667,0.41067 1.46667,0.41067 0.90933,0 1.496,-0.41067 0.58667,-0.42534 0.86534,-1.17334 0.27866,-0.76267 0.27866,-1.74533 0,-0.98267 -0.264,-1.73068 -0.264,-0.748 -0.836,-1.17333 -0.572,-0.42534 -1.496,-0.42534 -1.33467,0 -1.99467,0.88001 -0.66,0.88 -0.66,2.44934 z m 11.04402,4.048 q -1.56933,0 -2.42,-0.99733 -0.836,-1.012 -0.836,-2.96267 0,-2.02401 0.88,-3.08001 0.89467,-1.056 2.44934,-1.056 0.66,0 1.144,0.20533 0.484,0.19067 0.82133,0.528 0.35201,0.33733 0.54267,0.76267 h 0.0587 q -0.0147,-0.352 -0.044,-0.76267 -0.0147,-0.42533 -0.0147,-0.76267 v -3.168 h 0.836 v 11.14669 h -0.67466 l -0.11734,-1.36401 h -0.044 q -0.20533,0.41067 -0.54267,0.76267 -0.33733,0.33734 -0.836,0.54267 -0.49867,0.20533 -1.20267,0.20533 z m 0.10267,-0.71866 q 1.364,0 1.92134,-0.82134 0.572,-0.82133 0.572,-2.376 v -0.132 q 0,-1.61334 -0.572,-2.46401 -0.55734,-0.86534 -1.86267,-0.86534 -1.26134,0 -1.90667,0.89467 -0.64534,0.88 -0.64534,2.53734 0,1.58401 0.616,2.40534 0.616,0.82134 1.87734,0.82134 z m 8.82937,-7.37735 q 1.012,0 1.672,0.46933 0.67467,0.45467 1.012,1.26134 0.352,0.80666 0.352,1.86267 v 0.55733 h -5.57334 q 0,1.56934 0.71867,2.39067 0.73333,0.82134 2.068,0.82134 0.73333,0 1.276,-0.11733 0.54267,-0.11734 1.18801,-0.41067 v 0.76267 q -0.58667,0.264 -1.15867,0.38133 -0.57201,0.11733 -1.33467,0.11733 -1.15867,0 -1.98001,-0.484 -0.80667,-0.484 -1.232,-1.37867 -0.41067,-0.90933 -0.41067,-2.112 0,-1.20267 0.396,-2.12667 0.396,-0.92401 1.15867,-1.45201 0.76267,-0.54266 1.84801,-0.54266 z m 0,0.704 q -1.08534,0 -1.74534,0.71867 -0.66,0.704 -0.77734,2.03867 h 4.69335 q 0,-0.82134 -0.23467,-1.43734 -0.23467,-0.616 -0.71867,-0.968 -0.46933,-0.352 -1.21733,-0.352 z m 5.98401,7.24535 h -0.836 V 344.9814 h 0.836 z"
|
||||
id="text9-8"
|
||||
aria-label="Model" /><path
|
||||
d="m 622.02912,225.816 -3.76934,-9.57736 h -0.044 q 0.0293,0.264 0.0293,0.60134 0.0147,0.32266 0.0147,0.68933 0.0147,0.352 0.0147,0.73334 v 7.55335 h -0.792 v -10.47203 h 1.276 l 3.60801,9.18136 h 0.0587 l 3.65201,-9.18136 h 1.24667 V 225.816 h -0.83601 v -7.64135 q 0,-0.32267 0,-0.66001 0.0147,-0.33733 0.0293,-0.66 0.0147,-0.32266 0.0293,-0.60133 h -0.044 l -3.78401,9.56269 z m 10.82402,-7.93469 q 1.32,0 1.96534,0.64534 0.64533,0.64533 0.64533,2.05334 v 5.23601 h -0.63067 l -0.14666,-1.32001 h -0.044 q -0.27867,0.44001 -0.64534,0.77734 -0.352,0.32267 -0.86533,0.51333 -0.49867,0.176 -1.23201,0.176 -0.73333,0 -1.276,-0.24933 -0.54267,-0.264 -0.836,-0.748 -0.29334,-0.484 -0.29334,-1.188 0,-1.15867 0.95334,-1.76001 0.95334,-0.616 2.77201,-0.704 l 1.42267,-0.0587 v -0.55733 q 0,-1.15867 -0.46934,-1.628 -0.45467,-0.48401 -1.364,-0.48401 -0.60134,0 -1.15867,0.16134 -0.54267,0.14666 -1.1,0.42533 l -0.264,-0.66 q 0.54266,-0.27867 1.20267,-0.45467 0.66,-0.176 1.364,-0.176 z m 0.45467,4.04801 q -1.48134,0.0733 -2.21468,0.528 -0.73333,0.44 -0.73333,1.33467 0,0.71867 0.45467,1.10001 0.45467,0.36666 1.232,0.36666 1.24667,0 1.90667,-0.68933 0.67467,-0.68934 0.68934,-1.93601 v -0.76267 z m 6.84933,3.32934 q 0.32267,0 0.61601,-0.044 0.29333,-0.0587 0.51333,-0.132 v 0.67467 q -0.22,0.088 -0.55733,0.14667 -0.32267,0.0587 -0.68934,0.0587 -0.64533,0 -1.1,-0.23466 -0.45467,-0.24934 -0.704,-0.76267 -0.23467,-0.51334 -0.23467,-1.32 v -4.95735 h -1.144 v -0.46933 l 1.144,-0.264 0.308,-1.80401 h 0.528 v 1.86267 h 2.40534 v 0.67467 h -2.40534 v 4.92801 q 0,0.80667 0.32267,1.23201 0.32267,0.41066 0.99733,0.41066 z m 5.91069,0.704 q -1.144,0 -1.95067,-0.46933 -0.792,-0.484 -1.21734,-1.37867 -0.41067,-0.89467 -0.41067,-2.156 0,-1.32001 0.46934,-2.22934 0.484,-0.90934 1.32,-1.37867 0.836,-0.484 1.96534,-0.484 0.55733,0 1.04133,0.11733 0.49867,0.10267 0.88001,0.27867 l -0.22,0.704 q -0.39601,-0.16134 -0.85067,-0.24934 -0.45467,-0.10266 -0.86534,-0.10266 -0.93867,0 -1.584,0.41067 -0.63067,0.396 -0.968,1.144 -0.32267,0.748 -0.32267,1.78934 0,0.968 0.27867,1.716 0.29333,0.73333 0.89466,1.144 0.60134,0.41067 1.54001,0.41067 0.572,0 1.07067,-0.11733 0.51333,-0.11734 0.93867,-0.30801 v 0.76267 q -0.38134,0.16134 -0.88001,0.27867 -0.49866,0.11733 -1.12933,0.11733 z m 4.67867,-7.71468 q 0,0.29333 -0.0147,0.572 -0.0147,0.27867 -0.0293,0.55734 h 0.0587 q 0.20533,-0.44001 0.572,-0.77734 0.36667,-0.33733 0.88,-0.528 0.52801,-0.19067 1.18801,-0.19067 0.89467,0 1.51067,0.308 0.616,0.29334 0.924,0.92401 0.32267,0.616 0.32267,1.61333 v 5.08935 h -0.82134 v -5.04535 q 0,-1.12933 -0.51333,-1.65733 -0.51334,-0.52801 -1.51067,-0.52801 -0.85067,0 -1.42267,0.33734 -0.572,0.32267 -0.86534,0.99733 -0.27867,0.66001 -0.27867,1.64267 v 4.25335 h -0.82133 v -11.14669 h 0.82133 z m 8.71203,4.53201 q 0,-0.64533 0.10267,-1.07067 0.11733,-0.44 0.38134,-0.792 0.27866,-0.36667 0.76266,-0.73334 0.54267,-0.44 0.89467,-0.77733 0.352,-0.33733 0.528,-0.71867 0.176,-0.38133 0.176,-0.95333 0,-0.89467 -0.55733,-1.33467 -0.55734,-0.45467 -1.51067,-0.45467 -0.63067,0 -1.15867,0.14666 -0.51333,0.14667 -1.02667,0.41067 l -0.308,-0.704 q 0.616,-0.29333 1.20267,-0.44 0.60133,-0.16133 1.33467,-0.16133 1.32,0 2.068,0.66 0.76267,0.64533 0.76267,1.86267 0,0.704 -0.23466,1.20267 -0.22001,0.484 -0.63067,0.88 -0.41067,0.396 -0.968,0.82134 -0.44001,0.33733 -0.68934,0.64533 -0.23467,0.29333 -0.33733,0.67467 -0.10267,0.36667 -0.10267,0.90933 v 0.22 h -0.68934 z m -0.24933,2.47867 q 0,-0.36666 0.16133,-0.54267 0.16134,-0.176 0.46934,-0.176 0.32267,0 0.484,0.176 0.16133,0.17601 0.16133,0.54267 0,0.352 -0.16133,0.54267 -0.16133,0.176 -0.484,0.176 -0.308,0 -0.46934,-0.176 -0.16133,-0.19067 -0.16133,-0.54267 z"
|
||||
id="text11"
|
||||
style="font-weight:300;font-size:14.6667px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans Light';white-space:pre;stroke:#000000;stroke-width:1.00157"
|
||||
aria-label="Match?" /><path
|
||||
style="fill:#000000;stroke:#000000;stroke-width:0.955696;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle-8)"
|
||||
d="M 642.5652,312.47404 V 242.36807"
|
||||
d="M 640.43766,312.47404 V 242.36807"
|
||||
id="path11" /><path
|
||||
style="fill:#000000;stroke:#000000;stroke-width:0.826931;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle-8)"
|
||||
d="m 594.01642,220.67681 h 10.88445"
|
||||
id="path12" /><path
|
||||
style="fill:#000000;stroke:#000000;stroke-width:0.928487;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle-8-86)"
|
||||
d="M 640.72665,205.11014 V 169.29993"
|
||||
d="M 640.43766,205.11014 V 169.29993"
|
||||
id="path2" /><path
|
||||
style="font-weight:300;font-size:14.6667px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans Light';white-space:pre;stroke:#000000;stroke-width:1.00157"
|
||||
d="m 141.13472,223.67412 -1.39334,-3.608 h -4.35601 l -1.408,3.608 h -0.86534 l 4.12134,-10.50135 h 0.76267 l 4.01868,10.50135 z m -3.06535,-8.15468 q -0.044,-0.14667 -0.132,-0.396 -0.088,-0.24934 -0.176,-0.528 -0.088,-0.27867 -0.176,-0.49867 -0.0733,0.24933 -0.16133,0.51333 -0.0733,0.264 -0.16133,0.49867 -0.0733,0.23467 -0.14667,0.42534 l -1.43734,3.76934 h 3.79868 z m 11.22003,5.44134 q 0,0.93867 -0.46934,1.58401 -0.45466,0.63067 -1.276,0.95333 -0.82134,0.32267 -1.86267,0.32267 -0.64534,0 -1.15867,-0.0733 -0.51333,-0.0587 -0.93867,-0.14667 -0.42533,-0.10266 -0.76267,-0.23466 v -0.85068 q 0.55734,0.22001 1.29067,0.39601 0.73334,0.16133 1.61334,0.16133 0.80667,0 1.408,-0.23467 0.616,-0.23466 0.95334,-0.68933 0.352,-0.45467 0.352,-1.14401 0,-0.63066 -0.29334,-1.02666 -0.27866,-0.39601 -0.88,-0.68934 -0.58667,-0.308 -1.51067,-0.616 -0.66,-0.23467 -1.188,-0.484 -0.51333,-0.24934 -0.88,-0.58667 -0.36667,-0.33733 -0.55734,-0.792 -0.19066,-0.45467 -0.19066,-1.1 0,-0.86534 0.44,-1.45201 0.44,-0.58667 1.188,-0.89467 0.76267,-0.308 1.716,-0.308 0.76267,0 1.45201,0.14667 0.68933,0.132 1.30533,0.41067 l -0.27866,0.73333 q -0.61601,-0.264 -1.24667,-0.396 -0.63067,-0.132 -1.26134,-0.132 -0.73333,0 -1.29067,0.22 -0.54267,0.20534 -0.85067,0.63067 -0.308,0.41067 -0.308,1.02667 0,0.67467 0.29334,1.08534 0.29333,0.41066 0.85066,0.68933 0.55734,0.264 1.36401,0.54267 0.924,0.32267 1.584,0.68933 0.67467,0.352 1.02667,0.88 0.36667,0.52801 0.36667,1.37867 z m 5.06,2.71334 h -0.85067 v -9.70935 h -3.37334 v -0.76267 h 7.59735 v 0.76267 h -3.37334 z m 8.78537,-2.068 q 0,0.704 -0.352,1.20267 -0.352,0.49867 -1.02667,0.76267 -0.67467,0.24933 -1.64267,0.24933 -0.80667,0 -1.43734,-0.14667 -0.63067,-0.14666 -1.07067,-0.352 v -0.80667 q 0.528,0.26401 1.188,0.44001 0.66001,0.16133 1.33467,0.16133 1.17334,0 1.68667,-0.38133 0.51334,-0.39601 0.51334,-1.07067 0,-0.44 -0.24934,-0.71867 -0.23466,-0.29334 -0.71866,-0.51334 -0.46934,-0.23466 -1.17334,-0.46933 -0.71867,-0.24933 -1.276,-0.49867 -0.55734,-0.264 -0.86534,-0.66 -0.308,-0.396 -0.308,-1.08534 0,-0.93866 0.76267,-1.46667 0.76267,-0.528 2.03867,-0.528 0.704,0 1.30534,0.132 0.616,0.132 1.11467,0.352 l -0.29334,0.68934 q -0.45467,-0.20534 -1.02667,-0.33734 -0.572,-0.132 -1.144,-0.132 -0.924,0 -1.43734,0.32267 -0.51333,0.32267 -0.51333,0.93867 0,0.46933 0.23467,0.748 0.24933,0.264 0.71867,0.45467 0.484,0.19067 1.15866,0.44 0.68934,0.23467 1.24667,0.49867 0.57201,0.264 0.89467,0.67467 0.33734,0.41066 0.33734,1.1 z"
|
||||
style="fill:#000000;stroke:#000000;stroke-width:1.01266;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#ConcaveTriangle-8)"
|
||||
d="m 626.0242,147.65131 -25.39453,0.17829"
|
||||
id="path4" /><path
|
||||
d="m 25.618437,94.958394 -1.376,-3.536 h -4.528 l -1.36,3.536 h -1.456 l 4.464,-11.472 h 1.296 l 4.448,11.472 z m -3.088,-8.272 q -0.048,-0.128 -0.159999,-0.464 -0.112001,-0.336 -0.224001,-0.688 -0.096,-0.368 -0.16,-0.56 -0.08,0.32 -0.176,0.656 -0.08,0.32 -0.176,0.592 -0.08,0.272 -0.144,0.464 l -1.296,3.456 h 3.616 z m 12.624,5.216 q 0,1.024 -0.496,1.744 -0.496,0.704 -1.424,1.088 -0.912,0.384 -2.16,0.384 -0.64,0 -1.232,-0.064 -0.576,-0.064 -1.056,-0.176 -0.48,-0.128 -0.848,-0.304 v -1.376 q 0.576,0.256 1.424,0.464 0.864,0.208 1.776,0.208 0.848,0 1.424,-0.224 0.576,-0.224 0.864,-0.64 0.288,-0.416 0.288,-0.976 0,-0.56 -0.24,-0.944 -0.24,-0.384 -0.832,-0.704 -0.576,-0.336 -1.616,-0.704 -0.736,-0.272 -1.296,-0.576 -0.544,-0.32 -0.912,-0.72 -0.368,-0.4 -0.56,-0.912 -0.176,-0.512 -0.176,-1.184 0,-0.912 0.464,-1.552 0.464,-0.656 1.28,-1.008 0.832,-0.352 1.904,-0.352 0.944,0 1.728,0.176 0.784,0.176 1.424,0.464 l -0.448,1.232 q -0.592,-0.256 -1.296,-0.432 -0.688,-0.176 -1.44,-0.176 -0.72,0 -1.2,0.208 -0.48,0.208 -0.72,0.592 -0.24,0.368 -0.24,0.864 0,0.576 0.24,0.96 0.24,0.384 0.784,0.688 0.544,0.304 1.472,0.656 1.008,0.368 1.696,0.8 0.704,0.416 1.056,1.008 0.368,0.592 0.368,1.488 z m 5.919996,3.056 h -1.44 v -10.16 h -3.568 v -1.264 h 8.56 v 1.264 h -3.552 z m 9.712005,-2.368 q 0,0.832 -0.416,1.392 -0.416,0.56 -1.184,0.848 -0.768,0.288 -1.824,0.288 -0.896,0 -1.552,-0.144 -0.64,-0.144 -1.136,-0.4 v -1.28 q 0.512,0.256 1.232,0.48 0.736,0.208 1.488,0.208 1.072,0 1.552,-0.336 0.48,-0.352 0.48,-0.928 0,-0.32 -0.176,-0.576 -0.176,-0.256 -0.64,-0.512 -0.448,-0.256 -1.296,-0.576 -0.832,-0.32 -1.424,-0.64 -0.592,-0.32 -0.912,-0.768 -0.32,-0.448 -0.32,-1.152 0,-1.088 0.88,-1.68 0.896,-0.592 2.336,-0.592 0.784,0 1.456,0.16 0.688,0.144 1.28,0.416 l -0.48,1.12 q -0.544,-0.224 -1.136,-0.384 -0.592,-0.16 -1.216,-0.16 -0.864,0 -1.328,0.288 -0.448,0.272 -0.448,0.752 0,0.352 0.208,0.608 0.208,0.24 0.688,0.48 0.496,0.224 1.312,0.544 0.816,0.304 1.392,0.624 0.576,0.32 0.88,0.784 0.304,0.448 0.304,1.136 z"
|
||||
id="text1"
|
||||
style="font-size:16px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans';white-space:pre"
|
||||
transform="translate(121.92998,127.22613)"
|
||||
aria-label="ASTs" /><path
|
||||
style="font-size:14.6667px;font-family:Hack;-inkscape-font-specification:Hack;text-align:center;text-anchor:middle;white-space:pre;stroke:#000000;stroke-width:1.00157"
|
||||
d="m 140.25284,134.77453 h -3.35873 v -1.21745 h 8.15691 v 1.21745 h -3.3444 v 9.47463 h -1.45378 z m 7.29754,1.45378 h 1.19597 l 0.1289,1.56836 q 0.33659,-0.85221 1.00977,-1.30338 0.68034,-0.45834 1.62565,-0.45834 0.98829,0 1.71876,0.50131 v 1.34635 q -0.77344,-0.63021 -1.83334,-0.63021 -1.21745,0 -1.86914,0.7806 -0.6517,0.77344 -0.6517,2.22722 v 3.98894 h -1.32487 z m 11.52281,8.22854 q -1.85482,0 -2.91472,-1.11003 -1.05274,-1.11719 -1.05274,-3.04363 0,-1.19597 0.41537,-2.16276 0.42253,-0.9668 1.2461,-1.53256 0.83073,-0.57292 2.04818,-0.57292 1.60417,0 2.48503,1.03125 0.88802,1.0241 0.88802,2.84311 v 0.64453 h -5.70769 v 0.043 q 0,1.25326 0.65885,1.99805 0.65886,0.7448 1.94792,0.7448 0.7448,0 1.43946,-0.23633 0.70183,-0.23633 1.35352,-0.57292 v 1.31055 q -0.68034,0.2793 -1.38216,0.44401 -0.69467,0.17188 -1.42514,0.17188 z m 1.79753,-4.93426 q 0,-0.66602 -0.20052,-1.20313 -0.20052,-0.53711 -0.64453,-0.85221 -0.43685,-0.31511 -1.16016,-0.31511 -0.73047,0 -1.22461,0.32227 -0.48698,0.3151 -0.76628,0.85221 -0.27214,0.53711 -0.34375,1.19597 z m 7.03257,4.93426 q -1.85482,0 -2.91472,-1.11003 -1.05274,-1.11719 -1.05274,-3.04363 0,-1.19597 0.41537,-2.16276 0.42253,-0.9668 1.2461,-1.53256 0.83073,-0.57292 2.04818,-0.57292 1.60417,0 2.48503,1.03125 0.88802,1.0241 0.88802,2.84311 v 0.64453 h -5.70769 v 0.043 q 0,1.25326 0.65885,1.99805 0.65886,0.7448 1.94792,0.7448 0.7448,0 1.43946,-0.23633 0.70182,-0.23633 1.35352,-0.57292 v 1.31055 q -0.68034,0.2793 -1.38217,0.44401 -0.69466,0.17188 -1.42513,0.17188 z m 1.79753,-4.93426 q 0,-0.66602 -0.20052,-1.20313 -0.20052,-0.53711 -0.64453,-0.85221 -0.43685,-0.31511 -1.16016,-0.31511 -0.73047,0 -1.22461,0.32227 -0.48698,0.3151 -0.76628,0.85221 -0.27214,0.53711 -0.34375,1.19597 z m 2.95769,-3.29428 h 1.08138 l 0.11459,0.6875 q 0.48698,-0.88086 1.43945,-0.88086 1.02409,0 1.42514,1.00977 0.48698,-1.00977 1.53255,-1.00977 0.97396,0 1.36068,0.73047 0.39388,0.75196 0.39388,2.83595 v 4.64779 h -1.20313 v -4.5905 q 0,-0.90235 -0.0501,-1.39649 -0.043,-0.49414 -0.14323,-0.73047 -0.17903,-0.39388 -0.70898,-0.39388 -0.54427,0 -0.75196,0.42969 -0.12174,0.25065 -0.17187,0.73047 -0.0501,0.47982 -0.0501,1.36068 v 4.5905 h -1.20313 v -4.5905 q 0,-0.86654 -0.0501,-1.375 -0.0501,-0.50847 -0.15755,-0.7448 -0.19336,-0.40104 -0.72331,-0.40104 -0.52995,0 -0.72331,0.42253 -0.11458,0.24349 -0.16471,0.74479 -0.0501,0.49414 -0.0501,1.35352 v 4.5905 h -1.19597 z m 11.79495,-1.45378 q -0.21485,0 -0.21485,-0.21484 v -1.23894 q 0,-0.21484 0.21485,-0.21484 h 1.03125 q 0.21484,0 0.21484,0.21484 v 1.23894 q 0,0.21484 -0.21484,0.21484 z m 2.40625,9.53193 q -1.18164,0 -1.83334,-0.75912 -0.64453,-0.76628 -0.64453,-2.14844 v -4.13934 h -1.75456 v -1.03125 h 3.07227 v 5.17059 q 0,0.8737 0.32943,1.33203 0.32943,0.45834 0.95964,0.45834 h 1.53972 v 1.11719 z m 4.07488,-8.07815 h 1.18881 l 0.12891,1.20313 q 0.7233,-1.39649 2.35612,-1.39649 2.45639,0 2.45639,3.24415 v 4.97006 h -1.32488 v -4.97006 q 0,-1.08138 -0.37955,-1.58985 -0.3724,-0.50846 -1.16016,-0.50846 -0.93816,0 -1.44662,0.66602 -0.5013,0.65885 -0.5013,1.86914 v 4.53321 h -1.31772 z m 12.28909,8.22854 q -1.85482,0 -2.91472,-1.11003 -1.05273,-1.11719 -1.05273,-3.04363 0,-1.19597 0.41536,-2.16276 0.42253,-0.9668 1.2461,-1.53256 0.83073,-0.57292 2.04818,-0.57292 1.60417,0 2.48503,1.03125 0.88803,1.0241 0.88803,2.84311 v 0.64453 h -5.7077 v 0.043 q 0,1.25326 0.65886,1.99805 0.65885,0.7448 1.94792,0.7448 0.74479,0 1.43945,-0.23633 0.70183,-0.23633 1.35352,-0.57292 v 1.31055 q -0.68034,0.2793 -1.38216,0.44401 -0.69467,0.17188 -1.42514,0.17188 z m 1.79753,-4.93426 q 0,-0.66602 -0.20052,-1.20313 -0.20052,-0.53711 -0.64453,-0.85221 -0.43685,-0.31511 -1.16016,-0.31511 -0.73047,0 -1.22461,0.32227 -0.48698,0.3151 -0.76628,0.85221 -0.27214,0.53711 -0.34375,1.19597 z m 4.33986,-3.29428 h 1.19596 l 0.12891,1.56836 q 0.33659,-0.85221 1.00977,-1.30338 0.68034,-0.45834 1.62565,-0.45834 0.98829,0 1.71876,0.50131 v 1.34635 q -0.77344,-0.63021 -1.83334,-0.63021 -1.21745,0 -1.86915,0.7806 -0.65169,0.77344 -0.65169,2.22722 v 3.98894 h -1.32487 z m 7.64845,-2.67123 h 2.16993 q 2.44923,0 3.60938,1.30339 1.16016,1.30339 1.16016,4.05339 0,2.72853 -1.16016,4.03192 -1.16015,1.30338 -3.60938,1.30338 h -2.16993 z m 2.14128,9.50328 q 1.83334,0 2.54949,-0.90235 0.72331,-0.89518 0.72331,-3.25847 0,-2.37045 -0.71615,-3.25847 -0.36523,-0.45833 -1.01693,-0.67318 -0.64453,-0.222 -1.53972,-0.222 h -0.6875 v 8.31447 z"
|
||||
id="text10"
|
||||
transform="translate(10.91427,141.75844)"
|
||||
aria-label="TreeminerD" /></g></svg>
|
||||
d="m 32.859375,149.52148 h -3.664063 v -1.32812 h 8.898438 v 1.32812 h -3.648438 v 10.33594 h -1.585937 z m 7.960937,1.58594 H 42.125 l 0.140625,1.71094 q 0.367187,-0.92969 1.101562,-1.42188 0.742188,-0.5 1.773438,-0.5 1.078125,0 1.875,0.54688 v 1.46875 q -0.84375,-0.6875 -2,-0.6875 -1.328125,0 -2.039063,0.85156 -0.710937,0.84375 -0.710937,2.42969 v 4.35156 h -1.445313 z m 12.570313,8.97656 q -2.023438,0 -3.179688,-1.21093 -1.148437,-1.21875 -1.148437,-3.32032 0,-1.30468 0.453125,-2.35937 0.460937,-1.05469 1.359375,-1.67188 0.90625,-0.625 2.234375,-0.625 1.75,0 2.710937,1.125 0.96875,1.11719 0.96875,3.10157 v 0.70312 H 50.5625 v 0.0469 q 0,1.36718 0.71875,2.17968 0.71875,0.8125 2.125,0.8125 0.8125,0 1.570312,-0.25781 0.765625,-0.25781 1.476563,-0.625 v 1.42969 q -0.742188,0.30469 -1.507813,0.48437 -0.757812,0.1875 -1.554687,0.1875 z m 1.960937,-5.38281 q 0,-0.72656 -0.21875,-1.3125 -0.21875,-0.58594 -0.703125,-0.92969 -0.476562,-0.34375 -1.265625,-0.34375 -0.796875,0 -1.335937,0.35157 -0.53125,0.34375 -0.835938,0.92968 -0.296875,0.58594 -0.375,1.30469 z m 7.671875,5.38281 q -2.023437,0 -3.179687,-1.21093 -1.148438,-1.21875 -1.148438,-3.32032 0,-1.30468 0.453125,-2.35937 0.460938,-1.05469 1.359375,-1.67188 0.90625,-0.625 2.234375,-0.625 1.75,0 2.710938,1.125 0.96875,1.11719 0.96875,3.10157 v 0.70312 h -6.226563 v 0.0469 q 0,1.36718 0.71875,2.17968 0.71875,0.8125 2.125,0.8125 0.8125,0 1.570313,-0.25781 0.765625,-0.25781 1.476562,-0.625 v 1.42969 q -0.742187,0.30469 -1.507812,0.48437 -0.757813,0.1875 -1.554688,0.1875 z m 1.960938,-5.38281 q 0,-0.72656 -0.21875,-1.3125 -0.21875,-0.58594 -0.703125,-0.92969 -0.476563,-0.34375 -1.265625,-0.34375 -0.796875,0 -1.335938,0.35157 -0.53125,0.34375 -0.835937,0.92968 -0.296875,0.58594 -0.375,1.30469 z m 3.226562,-3.59375 h 1.179688 l 0.125,0.75 q 0.53125,-0.96094 1.570312,-0.96094 1.117188,0 1.554688,1.10157 0.53125,-1.10157 1.671875,-1.10157 1.0625,0 1.484375,0.79688 0.429687,0.82031 0.429687,3.09375 v 5.07031 h -1.3125 v -5.00781 q 0,-0.98438 -0.05469,-1.52344 -0.04687,-0.53906 -0.15625,-0.79687 -0.195313,-0.42969 -0.773438,-0.42969 -0.59375,0 -0.820312,0.46875 -0.132813,0.27344 -0.1875,0.79687 -0.05469,0.52344 -0.05469,1.48438 v 5.00781 h -1.3125 v -5.00781 q 0,-0.94531 -0.05469,-1.5 -0.05469,-0.55469 -0.171875,-0.8125 -0.210938,-0.4375 -0.789063,-0.4375 -0.578125,0 -0.789062,0.46094 -0.125,0.26562 -0.179688,0.8125 -0.05469,0.53906 -0.05469,1.47656 v 5.00781 h -1.304688 z m 12.867188,-1.58594 q -0.234375,0 -0.234375,-0.23437 v -1.35156 q 0,-0.23438 0.234375,-0.23438 h 1.125 q 0.234375,0 0.234375,0.23438 v 1.35156 q 0,0.23437 -0.234375,0.23437 z m 2.625,10.39844 q -1.289063,0 -2,-0.82812 Q 81,158.25586 81,156.74805 v -4.51563 h -1.914063 v -1.125 H 82.4375 v 5.64063 q 0,0.95312 0.359375,1.45312 0.359375,0.5 1.046875,0.5 h 1.679687 v 1.21875 z m 4.445312,-8.8125 h 1.296875 l 0.140625,1.3125 q 0.789063,-1.52344 2.570313,-1.52344 2.679687,0 2.679687,3.53907 v 5.42187 h -1.445312 v -5.42187 q 0,-1.17969 -0.414063,-1.73438 -0.40625,-0.55469 -1.265625,-0.55469 -1.023437,0 -1.578125,0.72657 -0.546875,0.71875 -0.546875,2.03906 v 4.94531 h -1.4375 z m 13.406253,8.97656 q -2.02344,0 -3.17969,-1.21093 -1.148438,-1.21875 -1.148438,-3.32032 0,-1.30468 0.453125,-2.35937 0.460938,-1.05469 1.359375,-1.67188 0.90625,-0.625 2.234378,-0.625 1.75,0 2.71094,1.125 0.96875,1.11719 0.96875,3.10157 v 0.70312 h -6.226568 v 0.0469 q 0,1.36718 0.71875,2.17968 0.718748,0.8125 2.124998,0.8125 0.8125,0 1.57032,-0.25781 0.76562,-0.25781 1.47656,-0.625 v 1.42969 q -0.74219,0.30469 -1.50781,0.48437 -0.75782,0.1875 -1.55469,0.1875 z m 1.96094,-5.38281 q 0,-0.72656 -0.21875,-1.3125 -0.21875,-0.58594 -0.70313,-0.92969 -0.47656,-0.34375 -1.26562,-0.34375 -0.79688,0 -1.335943,0.35157 -0.53125,0.34375 -0.835937,0.92968 -0.296875,0.58594 -0.375,1.30469 z m 4.73437,-3.59375 h 1.30469 l 0.14062,1.71094 q 0.36719,-0.92969 1.10157,-1.42188 0.74218,-0.5 1.77343,-0.5 1.07813,0 1.875,0.54688 v 1.46875 q -0.84375,-0.6875 -2,-0.6875 -1.32812,0 -2.03906,0.85156 -0.71094,0.84375 -0.71094,2.42969 v 4.35156 H 108.25 Z m 8.34375,-2.91406 h 2.36719 q 2.67187,0 3.9375,1.42187 1.26562,1.42188 1.26562,4.42188 0,2.97656 -1.26562,4.39844 -1.26563,1.42187 -3.9375,1.42187 h -2.36719 z m 2.33594,10.36719 q 2,0 2.78125,-0.98438 0.78906,-0.97656 0.78906,-3.55469 0,-2.58593 -0.78125,-3.55468 -0.39844,-0.5 -1.10937,-0.73438 -0.70313,-0.24219 -1.67969,-0.24219 h -0.75 v 9.07032 z"
|
||||
id="text2"
|
||||
style="font-size:16px;font-family:Hack;-inkscape-font-specification:Hack;white-space:pre"
|
||||
transform="translate(118.56687,129.28009)"
|
||||
aria-label="TreeminerD" /><path
|
||||
d="m 147.92952,220.74924 q 2.24,0 3.264,0.88 1.024,0.88 1.024,2.48 0,0.704 -0.24,1.36 -0.224,0.64 -0.752,1.152 -0.528,0.512 -1.408,0.816 -0.88,0.288 -2.16,0.288 h -1.312 v 4.448 h -1.44 v -11.424 z m -0.128,1.232 h -1.456 v 4.512 h 1.152 q 1.088,0 1.808,-0.224 0.72,-0.24 1.072,-0.752 0.352,-0.512 0.352,-1.344 0,-1.104 -0.704,-1.648 -0.704,-0.544 -2.224,-0.544 z m 9.83999,1.472 q 1.568,0 2.32,0.688 0.752,0.688 0.752,2.192 v 5.84 h -1.024 l -0.272,-1.216 h -0.064 q -0.368,0.464 -0.768,0.784 -0.384,0.304 -0.896,0.448 -0.496,0.144 -1.216,0.144 -0.768,0 -1.392,-0.272 -0.608,-0.272 -0.96,-0.832 -0.352,-0.576 -0.352,-1.44 0,-1.28 1.008,-1.968 1.008,-0.704 3.104,-0.768 l 1.456,-0.048 v -0.512 q 0,-1.072 -0.464,-1.488 -0.464,-0.416 -1.312,-0.416 -0.672,0 -1.28,0.208 -0.608,0.192 -1.136,0.448 l -0.432,-1.056 q 0.56,-0.304 1.328,-0.512 0.768,-0.224 1.6,-0.224 z m 0.416,4.576 q -1.6,0.064 -2.224,0.512 -0.608,0.448 -0.608,1.264 0,0.72 0.432,1.056 0.448,0.336 1.136,0.336 1.088,0 1.808,-0.592 0.72,-0.608 0.72,-1.856 v -0.768 z m 8.17601,3.152 q 0.32,0 0.656,-0.048 0.336,-0.064 0.544,-0.128 v 1.072 q -0.224,0.112 -0.64,0.176 -0.416,0.08 -0.8,0.08 -0.672,0 -1.248,-0.224 -0.56,-0.24 -0.912,-0.816 -0.352,-0.576 -0.352,-1.616 v -4.992 h -1.216 v -0.672 l 1.232,-0.56 0.56,-1.824 h 0.832 v 1.968 h 2.48 v 1.088 h -2.48 v 4.96 q 0,0.784 0.368,1.168 0.384,0.368 0.976,0.368 z m 5.776,0 q 0.32,0 0.656,-0.048 0.336,-0.064 0.544,-0.128 v 1.072 q -0.224,0.112 -0.64,0.176 -0.416,0.08 -0.8,0.08 -0.672,0 -1.248,-0.224 -0.56,-0.24 -0.912,-0.816 -0.352,-0.576 -0.352,-1.616 v -4.992 h -1.216 v -0.672 l 1.232,-0.56 0.56,-1.824 h 0.832 v 1.968 h 2.48 v 1.088 h -2.48 v 4.96 q 0,0.784 0.368,1.168 0.384,0.368 0.976,0.368 z m 6.224,-7.744 q 1.104,0 1.888,0.48 0.8,0.48 1.216,1.36 0.432,0.864 0.432,2.032 v 0.848 h -5.872 q 0.032,1.456 0.736,2.224 0.72,0.752 2,0.752 0.816,0 1.44,-0.144 0.64,-0.16 1.312,-0.448 v 1.232 q -0.656,0.288 -1.296,0.416 -0.64,0.144 -1.52,0.144 -1.216,0 -2.16,-0.496 -0.928,-0.496 -1.456,-1.472 -0.512,-0.992 -0.512,-2.416 0,-1.408 0.464,-2.416 0.48,-1.008 1.328,-1.552 0.864,-0.544 2,-0.544 z m -0.016,1.152 q -1.008,0 -1.6,0.656 -0.576,0.64 -0.688,1.792 h 4.368 q 0,-0.736 -0.224,-1.28 -0.224,-0.544 -0.688,-0.848 -0.448,-0.32 -1.168,-0.32 z m 9.72799,-1.152 q 0.24,0 0.512,0.032 0.288,0.016 0.496,0.064 l -0.176,1.296 q -0.208,-0.048 -0.464,-0.08 -0.24,-0.032 -0.464,-0.032 -0.496,0 -0.944,0.208 -0.448,0.208 -0.8,0.592 -0.352,0.368 -0.56,0.896 -0.192,0.528 -0.192,1.184 v 4.576 h -1.408 v -8.576 h 1.152 l 0.16,1.568 h 0.064 q 0.272,-0.48 0.656,-0.864 0.384,-0.4 0.88,-0.624 0.496,-0.24 1.088,-0.24 z m 6.736,0 q 1.536,0 2.32,0.752 0.784,0.736 0.784,2.4 v 5.584 h -1.392 v -5.488 q 0,-1.04 -0.464,-1.552 -0.464,-0.512 -1.456,-0.512 -1.424,0 -1.968,0.8 -0.544,0.8 -0.544,2.304 v 4.448 h -1.408 v -8.576 h 1.136 l 0.208,1.168 h 0.08 q 0.288,-0.448 0.704,-0.736 0.432,-0.304 0.944,-0.448 0.512,-0.144 1.056,-0.144 z m 11.344,6.368 q 0,0.832 -0.416,1.392 -0.416,0.56 -1.184,0.848 -0.768,0.288 -1.824,0.288 -0.896,0 -1.552,-0.144 -0.64,-0.144 -1.136,-0.4 v -1.28 q 0.512,0.256 1.232,0.48 0.736,0.208 1.488,0.208 1.072,0 1.552,-0.336 0.48,-0.352 0.48,-0.928 0,-0.32 -0.176,-0.576 -0.176,-0.256 -0.64,-0.512 -0.448,-0.256 -1.296,-0.576 -0.832,-0.32 -1.424,-0.64 -0.592,-0.32 -0.912,-0.768 -0.32,-0.448 -0.32,-1.152 0,-1.088 0.88,-1.68 0.896,-0.592 2.336,-0.592 0.784,0 1.456,0.16 0.688,0.144 1.28,0.416 l -0.48,1.12 q -0.544,-0.224 -1.136,-0.384 -0.592,-0.16 -1.216,-0.16 -0.864,0 -1.328,0.288 -0.448,0.272 -0.448,0.752 0,0.352 0.208,0.608 0.208,0.24 0.688,0.48 0.496,0.224 1.312,0.544 0.816,0.304 1.392,0.624 0.576,0.32 0.88,0.784 0.304,0.448 0.304,1.136 z"
|
||||
id="text3"
|
||||
style="font-size:16px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans';white-space:pre"
|
||||
transform="translate(144.25802,124.6198)"
|
||||
aria-label="Patterns" /><path
|
||||
d="m 491.217,236.88027 -3.76,-10.016 h -0.064 q 0.032,0.32 0.048,0.816 0.032,0.496 0.048,1.088 0.016,0.576 0.016,1.184 v 6.928 h -1.328 v -11.424 h 2.128 l 3.52,9.36 h 0.064 l 3.584,-9.36 h 2.112 v 11.424 h -1.424 v -7.024 q 0,-0.56 0.016,-1.104 0.016,-0.56 0.048,-1.04 0.032,-0.496 0.048,-0.832 h -0.064 l -3.808,10 z m 16.73599,-4.304 q 0,1.072 -0.288,1.904 -0.272,0.816 -0.8,1.392 -0.512,0.576 -1.264,0.88 -0.736,0.288 -1.648,0.288 -0.848,0 -1.568,-0.288 -0.72,-0.304 -1.248,-0.88 -0.528,-0.576 -0.832,-1.392 -0.288,-0.832 -0.288,-1.904 0,-1.424 0.48,-2.4 0.48,-0.992 1.376,-1.504 0.896,-0.528 2.128,-0.528 1.168,0 2.048,0.528 0.896,0.512 1.392,1.504 0.512,0.976 0.512,2.4 z m -6.48,0 q 0,1.008 0.256,1.76 0.272,0.736 0.832,1.136 0.56,0.4 1.424,0.4 0.864,0 1.424,-0.4 0.56,-0.4 0.816,-1.136 0.272,-0.752 0.272,-1.76 0,-1.024 -0.272,-1.744 -0.272,-0.72 -0.832,-1.104 -0.544,-0.4 -1.424,-0.4 -1.312,0 -1.904,0.864 -0.592,0.864 -0.592,2.384 z m 11.744,4.464 q -1.6,0 -2.56,-1.104 -0.96,-1.12 -0.96,-3.328 0,-2.208 0.96,-3.328 0.976,-1.136 2.576,-1.136 0.672,0 1.168,0.176 0.496,0.16 0.864,0.448 0.368,0.288 0.624,0.64 h 0.096 q -0.016,-0.208 -0.064,-0.608 -0.032,-0.416 -0.032,-0.656 v -3.424 h 1.408 v 12.16 h -1.136 l -0.208,-1.152 h -0.064 q -0.256,0.368 -0.624,0.672 -0.368,0.288 -0.88,0.464 -0.496,0.176 -1.168,0.176 z m 0.224,-1.168 q 1.36,0 1.904,-0.736 0.56,-0.752 0.56,-2.256 v -0.256 q 0,-1.6 -0.528,-2.448 -0.528,-0.864 -1.952,-0.864 -1.136,0 -1.712,0.912 -0.56,0.896 -0.56,2.416 0,1.536 0.56,2.384 0.576,0.848 1.728,0.848 z m 9.88799,-7.728 q 1.104,0 1.888,0.48 0.8,0.48 1.216,1.36 0.432,0.864 0.432,2.032 v 0.848 h -5.872 q 0.032,1.456 0.736,2.224 0.72,0.752 2,0.752 0.816,0 1.44,-0.144 0.64,-0.16 1.312,-0.448 v 1.232 q -0.656,0.288 -1.296,0.416 -0.64,0.144 -1.52,0.144 -1.216,0 -2.16,-0.496 -0.928,-0.496 -1.456,-1.472 -0.512,-0.992 -0.512,-2.416 0,-1.408 0.464,-2.416 0.48,-1.008 1.328,-1.552 0.864,-0.544 2,-0.544 z m -0.016,1.152 q -1.008,0 -1.6,0.656 -0.576,0.64 -0.688,1.792 h 4.368 q 0,-0.736 -0.224,-1.28 -0.224,-0.544 -0.688,-0.848 -0.448,-0.32 -1.168,-0.32 z m 7.13599,7.584 h -1.408 v -12.16 h 1.408 z"
|
||||
id="text4"
|
||||
style="font-size:16px;font-family:'Noto Sans';-inkscape-font-specification:'Noto Sans';white-space:pre"
|
||||
transform="translate(135.28904,119.91277)"
|
||||
aria-label="Model" /></g></svg>
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 42 KiB After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 235 KiB After Width: | Height: | Size: 224 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 298 KiB After Width: | Height: | Size: 330 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 395 KiB After Width: | Height: | Size: 309 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 213 KiB After Width: | Height: | Size: 215 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 366 KiB After Width: | Height: | Size: 351 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 320 KiB After Width: | Height: | Size: 317 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 515 KiB After Width: | Height: | Size: 482 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 441 KiB After Width: | Height: | Size: 420 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 218 KiB After Width: | Height: | Size: 226 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 205 KiB After Width: | Height: | Size: 209 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 222 KiB After Width: | Height: | Size: 215 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 232 KiB After Width: | Height: | Size: 234 KiB |
Loading…
Add table
Add a link
Reference in a new issue