13hannes11/bachelor_thesis
1
0
Fork 0
mirror of https://github.com/13hannes11/bachelor_thesis.git synced 2024-09-04 01:11:00 +02:00
Code Issues Projects Releases Wiki Activity

changed metric and qestions to answer during eval

Browse Source
This commit is contained in:
hannes.kuchelmeister
2020-03-27 10:56:40 +01:00
parent 050d118305
commit 7bb27bd206
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
30_Thesis/sections/60_evaluation.tex
View File

@@ -11,13 +11,16 @@ Generate groups with preferences (explicit preferences) and configuration state
\label{sec:Evaluation:Metrics} \label{sec:Evaluation:Metrics}
For the evaluation a metric to evaluate by is needed. The proposed metric for usage is that of satisfactions. Satisfaction is quantified in this thesis by a threshold metric. A user's preference is used to calculate a rating for each possible solution. The score will be calculated using the average of a user's rating for each characteristic that is part of the solution. The result allows that a configuration can be compared to all other configurations and ranked according to the percentage of configurations that it beats. The threshold metric consists of two parameters. First the threshold center $tc$ and second the satisfaction distance $sd$. The threshold for a person being satisfied is $tc + sd$ and of a person being dissatisfied at $tc - sd$. If a recommendation lies in between these two thresholds the person is classified to neither by satisfied nor be unsatisfied with the solution. For this thesis $sd=5\%$ will be used. This choice is guided by the assumption that people switch from satisfied to unsatisfied rather quickly. Therefore the parameter considered in this thesis is the $tc$. An example is the choice of $tc = 60\%$. This results in a person being satisfied if recommendation is better than at lest $65\%$ of possible finished configurations. Moreover, a person is dissatisfied if the recommendation is only better than $55\%$ of possible finished configurations. A recommendation that is better than at least $55\%$ and not better than $65\%$ of possible solutions is considered neutral by the individual. For the evaluation a metric to evaluate by is needed. The proposed metric for usage is that of satisfactions. Satisfaction is quantified in this thesis by a threshold metric. A user's preference is used to calculate a rating for each possible solution. The score will be calculated using the average of a user's rating for each characteristic that is part of the solution. The result allows that a configuration can be compared to all other configurations and ranked according to the percentage of configurations that it beats. The threshold metric consists of two parameters. First the threshold center $tc$ and second the satisfaction distance $sd$. The threshold for a person being satisfied is $tc + sd$ and of a person being dissatisfied at $tc - sd$. If a recommendation lies in between these two thresholds the person is classified to neither by satisfied nor be unsatisfied with the solution. For this thesis $sd=5\%$ will be used. This choice is guided by the assumption that people switch from satisfied to unsatisfied rather quickly. Therefore the parameter considered in this thesis is the $tc$. An example is the choice of $tc = 60\%$. This results in a person being satisfied if recommendation is better than at lest $65\%$ of possible finished configurations. Moreover, a person is dissatisfied if the recommendation is only better than $55\%$ of possible finished configurations. A recommendation that is better than at least $55\%$ and not better than $65\%$ of possible solutions is considered neutral by the individual.
Different $tc$ values allow to model different situations. A situation where there is a low willingness to compromise is modelled by a high $tc$. A contrary situation where a group has a high willingness to compromise is modelled by a low $tc$. Different $tc$ values allow to model different situations. A situation where there is a low willingness to compromise is modelled by a high $tc$. A contrary situation where a group has a high willingness to compromise is modelled by a low $tc$.
\todo[inline]{add disclaimer that this metric is a new one not found in literature because no fitting metric was found}
\section{Questions to Answer During the Evaluation} \section{Questions to Answer During the Evaluation}
\label{sec:Evaluation:Questions} \label{sec:Evaluation:Questions}
\begin{itemize} \begin{itemize}
\item Main question: How does the satisfaction with a group decision, guided by a the recommender, differ from the decision of a single decision maker, who does not take the other group member's opinion into account? \item Main question: How does the satisfaction with a group decision, guided by the recommender, differ from the decision of a single decision maker, the dictator, who does not take the other group member's opinion into account?
\item How many group members are satisfied with the group decision on average? \item How many group members are satisfied with the group decision on average?
%\item Is the recommender fair, i.e. no user type is always worse off than others? (Just uses groupe preferences) %\item Is the recommender fair, i.e. no user type is always worse off than others? (Just uses groupe preferences)
\item How does the amount of stored finished configurations relate to recommendation satisfaction? \item How does the amount of stored finished configurations relate to recommendation satisfaction?