From 85b4fc51fbebecdeda04f5db8450f06d442dd1ae Mon Sep 17 00:00:00 2001 From: "hannes.kuchelmeister" Date: Wed, 25 Mar 2020 12:24:50 +0100 Subject: [PATCH] changed metric used by evaluation --- 30_Thesis/sections/60_evaluation.tex | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/30_Thesis/sections/60_evaluation.tex b/30_Thesis/sections/60_evaluation.tex index a4792db..b7aff01 100644 --- a/30_Thesis/sections/60_evaluation.tex +++ b/30_Thesis/sections/60_evaluation.tex @@ -10,7 +10,8 @@ Generate groups with preferences (explicit preferences) and configuration state \section{Metric} \label{sec:Evaluation:Metrics} -For the evaluation a metric to evaluate by is needed. The proposed metric for usage is that of satisfactions. Satisfaction will be quantified by a threshold metric. A user's preference is used to calculate a rating for each possible solution. The score will be calculated using the average of a user's rating for each characteristic that is part of the solution. The result allows that a configuration can be compared to all other configurations and ranked according to the percentage of configurations that it beats. 50\% is used as base line and the parameter this metric accepts is something that will be called satisfaction mean distance $smd$. The users counting as satisfied with a solution find the solution to be better than $50\% + smd$ of all possible solutions. Respectively a solution that ranks among the lowest scored $50\% - smd$ is classified as dissatisfied. +For the evaluation a metric to evaluate by is needed. The proposed metric for usage is that of satisfactions. Satisfaction is quantified in this thesis by a threshold metric. A user's preference is used to calculate a rating for each possible solution. The score will be calculated using the average of a user's rating for each characteristic that is part of the solution. The result allows that a configuration can be compared to all other configurations and ranked according to the percentage of configurations that it beats. The threshold metric consists of two parameters. First the threshold center $tc$ and second the satisfaction distance $sd$. The threshold for a person being satisfied is $tc + sd$ and of a person being dissatisfied at $tc - sd$. If a recommendation lies in between these two thresholds the person is classified to neither by satisfied nor be unsatisfied with the solution. For this thesis $sd=5\%$ will be used. This choice is guided by the assumption that people switch from satisfied to unsatisfied rather quickly. Therefore the parameter considered in this thesis is the $tc$. An example is the choice of $tc = 60\%$. This results in a person being satisfied if recommendation is better than at lest $65\%$ of possible finished configurations. Moreover, a person is dissatisfied if the recommendation is only better than $55\%$ of possible finished configurations. A recommendation that is better than at least $55\%$ and not better than $65\%$ of possible solutions is considered neutral by the individual. +Different $tc$ values allow to model different situations. A situation where there is a low willingness to compromise is modelled by a high $tc$. A contrary situation where a group has a high willingness to compromise is modelled by a low $tc$. \section{Questions to Answer During the Evaluation} \label{sec:Evaluation:Questions}