From eeb94645b15e83eb0f9546ebca73f0f86a95cfee Mon Sep 17 00:00:00 2001 From: "hannes.kuchelmeister" Date: Wed, 25 Mar 2020 12:26:35 +0100 Subject: [PATCH] changed finding section --- 30_Thesis/sections/60_evaluation.tex | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/30_Thesis/sections/60_evaluation.tex b/30_Thesis/sections/60_evaluation.tex index a2cd9a3..c8830ff 100644 --- a/30_Thesis/sections/60_evaluation.tex +++ b/30_Thesis/sections/60_evaluation.tex @@ -131,21 +131,25 @@ These user profiles can be used to generate rather homogenous groups but also to \todo[inline]{explain preference profiles} -\section{Hypothesises} -\label{sec:Evaluation:Hypothesises} +\section{Hypotheses} +\label{sec:Evaluation:Hypotheses} Understanding data is made easier by first posing hypothesises. This section gives an overview over the hypothesis used during data analysis. \begin{enumerate}[font={\bfseries},label={H\arabic*}] + \item More homogeneous groups have more satisfied members with the recommender's decision but also with the dictator's decision compared to less homogeneous groups. + \item More heterogeneous groups see a bigger increase in satisfaction than less heterogeneous groups when comparing the dictator's decision with the recommender's decision. + \item A higher $tc$ value results in less satisfied people and more unsatisfied people. + \item There exists a $tc$ value which causes only one person to be satisfied with the dictator's decision and no one is satisfied with the group recommender's decision. + \item A higher amount of stored finished configurations results in a better recommendation result. + + \item \label{hyp:Evaluation:LowSMD} A low $smd$ results in more people being satisfied and in more people being dissatisfied. This is expected due to the increase of configurations that fall in the specified quantiles. \item \label{hyp:Evaluation:HighSMD} A high $smd$ results in less people being satisfied and in less people being dissatisfied. This is expected due to the decrease of configurations that fall in the specified quantiles. \item \label{hyp:Evaluation:MoreSatisfiedLessIncrease} More people being satisfied results in a lower increase of satisfaction due to most people being satisfied already. \item \label{hyp:Evaluation:OnePersonSatisfied} A too high $smd$ results in a negative satisfaction and therefore in a satisfaction change of minus one. This is caused because only one person, the person who made the individual decision, is satisfied with it. \item \label{hyp:Evaluation:NoOnedissatisfied} A too high $smd$ results in no decrease in dissatisfied people when comparing the group decision with the individual decision. - \item \label{hyp:Evaluation:IndivDecision} A higher number of people satisfied with the decision of the individual results in a lower increase in satisfaction. This is because if all people are already satisfied with the individual decision there cannot be an increase in people being additionally satisfied with the group decision. \item \label{hyp:Evaluation:NumberOfStored} More stored finished configurations results in a higher increase in satisfaction and a higher reduction in dissatisfied group members. - \item \label{hyp:Evaluation:HomogenousBenefit} The benefit for homogenous groups is small as individuals interest are so closely aligned. - \item \label{hyp:Evaluation:SatisfactionIndividualDifferenGroupTypes} The overall number of satisfied group members with the individual decision is higher in homogeneous groups compared to heterogeneous groups. \item \label{hyp:Evaluation:AggregationFunctions} Multiplication and best average aggregation strategies should perform better than least misery. These strategies are listed by \citeauthor{Masthoff2015} \cite[p. 755f]{Masthoff2015} and multiplication and best average came out as the best in most studies. Least misery was in some listed as performing worst. Therefore it fares worse than the other strategies here. \end{enumerate} @@ -159,7 +163,7 @@ The data confirms \hyporef{hyp:Evaluation:LowSMD} and \hyporef{hyp:Evaluation:Hi \hyporef{hyp:Evaluation:MoreSatisfiedLessIncrease} states that the amount of hapiness increase should be limited if we choose a low $smb$ because there is less people that can be changed to satisfied. This effect however is not observable in the data even when $smb = 5$. Neither can the adverse effect with a high $smb$ and dissatisfaction change. Yet it might be possible that this effect can be observed when choosing $smb > 35\%$. The effects of \hyporef{hyp:Evaluation:OnePersonSatisfied} can also not be seen in the data. Moreover the effect of \hyporef{hyp:Evaluation:NoOnedissatisfied} could also not be seen in the data but it was noticeable that a high $smb$ reduces unhappiness reduction. Nonetheless these effects still might occur when testing a higher $smb$ than $35\%$. -The $smd$ will be fixed from now on to $15\%$. This allows to still show the improvements of the recommender for rather hetereogenous groups but also prevents a too high reduction in dissatisfied group members thereby preventing any effect when just looking at dissatisfaction. As \autoref{fig:Evaluation:HappyUnhappySMD} shows homogenous groups are already too happy and have no dissatisfaction. This is why they will not undergo any more evaluation. +The $smd$ will be fixed from now on to $15\%$. This allows to still show the improvements of the recommender for rather heterogeneous groups but also prevents a too high reduction in dissatisfied group members thereby preventing any effect when just looking at dissatisfaction. As \autoref{fig:Evaluation:HappyUnhappySMD} shows homogenous groups are already too happy and have no dissatisfaction. This is why they will not undergo any more evaluation. \begin{figure} \centering @@ -172,13 +176,13 @@ The $smd$ will be fixed from now on to $15\%$. This allows to still show the imp In this section results for heterogeneous groups, random groups and homogenous groups, based on the forest use case, are shown. \autoref{fig:Evaluation:HeterogenousGroupIncrease} and \autoref{fig:Evaluation:HeterogenousGroupTotal} show results for heterogeneous groups. \autoref{fig:Evaluation:RandomGroupIncrease}, \autoref{fig:Evaluation:RandomGroupTotal} shows the results for random groups and \autoref{fig:Evaluation:HomogenousGroupIncrease}, \autoref{fig:Evaluation:HomogenousGroupTotal} show the results for homogenous groups. -The first thing that is noticed when analysing the data is that with homogenous groups the recommender does not have any benefit to an individual choosing based on their own preferences. This is most likely due to all individuals being already happy with the individual decisions. This is an effect that was noticed even when a higher $smd$ was chosen. Here we notice that the effect of not having many configurations in the store does decrease hapiness by a large amount. +The first thing that is noticed when analysing the data is that with homogenous groups the recommender does not have any benefit to an individual choosing based on their own preferences. This is most likely due to all individuals being already satisfied with the individual decisions. This is an effect that was noticed even when a higher $smd$ was chosen. Here we notice that the effect of not having many configurations in the store does decrease hapiness by a large amount. -When looking at results for random and heterogeneous groups the happiness level with an individual decision is much lower than individual decisions in homogenous groups. This finding is expected as random and homogenous groups are more diverse therefore opposing interest will be visible in these. +When looking at results for random and heterogeneous groups the satisfaction level with an individual decision is much lower than individual decisions in homogenous groups. This finding is expected as random and homogenous groups are more diverse therefore opposing interest will be visible in these. -All scoring functions are similarly good in decreasing unhappiness. However the results differ when looking at happiness, Here least misery performs abysmal compared to the other scoring functions. In \autoref{fig:Evaluation:HeterogenousGroupIncrease} it results even in a hapiness reduction whereby multiplication and best average increase it. Overall multiplication seems to perform the best in most scenarios. This confirms findings in expirments with real people as described by \citeauthor{Masthoff2015} \cite[p. 755f]{Masthoff2015}. +All scoring functions are similarly good in decreasing dissatisfaction. However the results differ when looking at satisfaction, Here least misery performs abysmal compared to the other scoring functions. In \autoref{fig:Evaluation:HeterogenousGroupIncrease} it results even in a hapiness reduction whereby multiplication and best average increase it. Overall multiplication seems to perform the best in most scenarios. This confirms findings in expirments with real people as described by \citeauthor{Masthoff2015} \cite[p. 755f]{Masthoff2015}. -The influence of stored configurations on performance can clearly be seen but the relationship does not seem to be linear. Therefore with already just a limited amount of stored finished configurations the recommender can increase happiness and decrease unhappiness. With 33\% of the stored configurations, the happiness increase is about 50\% to 75\% compared to using all stored finished configurations. +The influence of stored configurations on performance can clearly be seen but the relationship does not seem to be linear. Therefore with already just a limited amount of stored finished configurations the recommender can increase satisfaction and decrease dissatisfaction. With 33\% of the stored configurations, the satisfaction increase is about 50\% to 75\% compared to using all stored finished configurations. @@ -209,4 +213,3 @@ The influence of stored configurations on performance can clearly be seen but th \caption{The average happiness and unhappiness for \textbf{random} groups consisting of four members with $smd=15\%$.} \label{fig:Evaluation:RandomGroupTotal} \end{figure} -