13hannes11/bachelor_thesis
1
0
Fork 0
mirror of https://github.com/13hannes11/bachelor_thesis.git synced 2024-09-04 01:11:00 +02:00
Code Issues Projects Releases Wiki Activity

changed metric used by evaluation

Browse Source
This commit is contained in:
hannes.kuchelmeister
2020-03-25 12:24:50 +01:00
parent 31bf2fe11b
commit 85b4fc51fb
1 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
30_Thesis/sections/60_evaluation.tex
View File

@@ -10,7 +10,8 @@ Generate groups with preferences (explicit preferences) and configuration state
\section{Metric}
\label{sec:Evaluation:Metrics}
For the evaluation a metric to evaluate by is needed. The proposed metric for usage is that of satisfactions. Satisfaction will be quantified by a threshold metric. A user's preference is used to calculate a rating for each possible solution. The score will be calculated using the average of a user's rating for each characteristic that is part of the solution. The result allows that a configuration can be compared to all other configurations and ranked according to the percentage of configurations that it beats. 50\% is used as base line and the parameter this metric accepts is something that will be called satisfaction mean distance $smd$. The users counting as satisfied with a solution find the solution to be better than $50\% + smd$ of all possible solutions. Respectively a solution that ranks among the lowest scored $50\% - smd$ is classified as dissatisfied.
For the evaluation a metric to evaluate by is needed. The proposed metric for usage is that of satisfactions. Satisfaction is quantified in this thesis by a threshold metric. A user's preference is used to calculate a rating for each possible solution. The score will be calculated using the average of a user's rating for each characteristic that is part of the solution. The result allows that a configuration can be compared to all other configurations and ranked according to the percentage of configurations that it beats. The threshold metric consists of two parameters. First the threshold center $tc$ and second the satisfaction distance $sd$. The threshold for a person being satisfied is $tc + sd$ and of a person being dissatisfied at $tc - sd$. If a recommendation lies in between these two thresholds the person is classified to neither by satisfied nor be unsatisfied with the solution. For this thesis $sd=5\%$ will be used. This choice is guided by the assumption that people switch from satisfied to unsatisfied rather quickly. Therefore the parameter considered in this thesis is the $tc$. An example is the choice of $tc = 60\%$. This results in a person being satisfied if recommendation is better than at lest $65\%$ of possible finished configurations. Moreover, a person is dissatisfied if the recommendation is only better than $55\%$ of possible finished configurations. A recommendation that is better than at least $55\%$ and not better than $65\%$ of possible solutions is considered neutral by the individual.
Different $tc$ values allow to model different situations. A situation where there is a low willingness to compromise is modelled by a high $tc$. A contrary situation where a group has a high willingness to compromise is modelled by a low $tc$.
\section{Questions to Answer During the Evaluation}
\label{sec:Evaluation:Questions}