13hannes11/bachelor_thesis
1
0
Fork 0
mirror of https://github.com/13hannes11/bachelor_thesis.git synced 2024-09-04 01:11:00 +02:00
Code Issues Projects Releases Wiki Activity

add feedback and start slight improvements

Browse Source
This commit is contained in:
hannes.kuchelmeister
2020-04-03 11:19:25 +02:00
parent 158be24330
commit 2c97ddd74a
1 changed files with 27 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
30_Thesis/sections/60_evaluation.tex
View File

@@ -12,10 +12,16 @@ Different $tc$ values allow to model different situations. A situation where the
A satisfaction and dissatisfaction classification allows groups to be measured by the amount of people that are satisfied and dissatisfied. Moreover, changes in satisfaction and dissatisfaction for different parameters can be compared. A reasonable $tc$ value has to be found for groups otherwise any derived metrics will not show any meaningful results.
\section{Evaluation Objective}
\label{sec:Evaluation:Questions}
An introduction to which questions lead the evaluation
\todo[inline]{fülle dieses Kapitel noch etwas mit Leben: Kurze Einleitung, dass die Evaluation von folgenden Fragen geleitet wird und im Anschluss an die Fragen jeweils noch eine Erläuterung, warum diese Fragen relevant sind / auf was das Beantworten dieser Fragen abzielt. zB Main Question -> herausfinden,ob der Recommender für die Gruppe tatsächlich von Vorteil ist
Und, dass zB die Frage bzgl Anzahl auf die technischen Eigenschaften des Recommenders abzielt}
\begin{itemize}
\item Main question: How does the satisfaction with a group decision, guided by the recommender, differ from the decision of a single decision maker, the dictator, who does not take the other group member's opinion into account?
\item Main question: How does the satisfaction with a group decision, guided by the recommender, differ from the decision of a single decision maker, the dictator, who does not take the other group member's opinions into account?
\item How many group members are satisfied with the group decision on average?
\item How does the amount of stored finished configurations relate to satisfaction with a recommendation?
\end{itemize}
@@ -23,17 +29,19 @@ A satisfaction and dissatisfaction classification allows groups to be measured b
\section{Effect of Stored Finished Configurations}
\label{sec:Evaluation:EffectFinishedConfiguration}
When evaluating just a subset of stored finished configurations it is important to avoid outliers. This is the reason why a process inspired by cross validation is used. The configuration database is randomly ordered and sliced into sub databases of the needed size. As an example, if the evaluated stored data size is 20, a configuration database containing 100 configurations is split into five sub databases of size 20. Now the evaluation is done on each of the sub databases and as a result the average is taken.
\todo[inline]{potentially move to data generation}
When evaluating a subset of stored finished configurations it is important to avoid outliers. This is the reason why a process inspired by \emph{cross validation} \todo{referenz hinzufügen} is used. The configuration database is randomly ordered and sliced into sub databases of the needed size. As an example, if the evaluated stored data size is 20, a configuration database containing 100 configurations is split into five sub databases of size 20. Now the evaluation is done on each of the sub databases and as a result the average is taken. This avoid that randomly a subset can be picked which either performs much better than most other possible combinations of databases or which performs much worse. This way the data is more aligned to the \emph{expected value} \todo{referenz}.
\section{Use Case}
\label{sec:Evaluation:UseCase}
To evaluate data, a given use case is needed. In this thesis a forestry use case is evaluated. This is a use case with four stakeholders. \autoref{fig:Concept:ForestExample} already presented the attributes and characteristics used in this use case but an extension is needed to fully show the whole use case. Namely rules of non valid configurations. The constraints for this use case are listed in \emph{not with} form in \autoref{tab:Evaluation:UseCase}.
To evaluate the recommender, a use case is needed. In this thesis, a forestry use case is evaluated. This is a use case with four stakeholders. \autoref{fig:Concept:ForestExample} presents the attributes and characteristics of this use case but an extension is needed to fully show the whole use case. Namely rules of non valid configurations. The constraints for this use case are listed in \emph{not with} form in \autoref{tab:Evaluation:UseCase}.
\begin{table}
\begin{center}
\begin{tabular}{r|l}
& not with (either of the listed) \\
\textbf{characteristic} & \textbf{not with (either of the listed) characteristics} \\
\hline
$(\textit{indigenous}, \text{moderate})$ & $(\textit{resilient}, \text{high})$ \\
\hline
@@ -62,32 +70,39 @@ To evaluate data, a given use case is needed. In this thesis a forestry use case
$(\textit{quantity}, \text{high})$ & $(\textit{accessibility}, \text{high}), (\textit{accessibility}, \text{moderate})$\\
\hline
\end{tabular}
\caption{Constrains in not with form for the forest use case.}
\caption{Constrains in \emph{not with} form for the forest use case.}
\label{tab:Evaluation:UseCase}
\end{center}
\todo[inline]{matrix anstelle von tabelle}
\end{table}
The stakeholders of this use case are: a forest owner, an athlete, an environmentalist and a consumer. The owner sees the forest as an investment, he is interested in a high long term profit. On the other hand the consumer is interested in reasonable wood price as she uses wood for furniture and also for her fireplace. In contrast, the environmentalist is interested in a healthy forest that is not impacted negatively by human activity. Last is the athlete who is interested in good accessibility of the forest and that there is some plant and animal life.
The stakeholders in this use case are: a forest owner, an athlete, an environmentalist, and a consumer. The owner sees the forest as an investment, he is interested in a high long term profit. On the other hand the consumer is interested in reasonable wood price as she uses wood for furniture and also for her fireplace. In contrast, the environmentalist is interested in a healthy forest that is not impacted negatively by human activity. Last is the athlete who is interested in good accessibility of the forest and that there is some plant and animal life.
\todo[inline]{Kapitel nochmal abschließen mit: hier liegen als sich widersprechende Präferenzen vor. Und: was sollen die Stakeholder jetzt entscheiden? in welcher Situation befinden sie sich? Wie setzt sich eine Gruppe zusammen? Aus 4 Personen von je einem Typ?}
\section{Generating Data}
\label{sec:Evaluation:GeneratingGroups}
The whole process explained in this section is visualized in \autoref{fig:Evaluation:GeneratingDataProcess}.
\todo[inline]{Dieses Kapitel ist für mich noch nicht konsistent. Auf der Abbildung fehlen Elemente (Präferenzen \& Gruppen generieren), im Text ist das Paaren von Präferenzen(?) und Konfigurationen nicht beschrieben. Und: was wird da wirklich gepaart: präferenzen oder Gruppen?}
The whole process explained in \todo[inline]{hier einen besseren Übergang schaffen: um den use case zu evaluieren, wurden basierend auf den vorherigen Informationen Daten generiert. Die Visualisierung...} this section is visualized in \autoref{fig:Evaluation:GeneratingDataProcess}.
\begin{figure}
\centering
\includegraphics[width=1\textwidth]{./figures/60_evaluation/bpmn_evaluation_input_data_generation.pdf}
\caption{The process used for generating data for the evaluation.}
\caption{Data generation process for the evaluation}
\label{fig:Evaluation:GeneratingDataProcess}
\end{figure}
\subsection{Generating Unfinished Configurations}
Unfinished configurations are generated using all finished configurations and taking a subset of the contained characteristics. This way all generated configurations will be valid and lead to valid solutions. For the results that are presented in this chapter around $\frac{1}{7} \approx 15\%$ of characteristics is kept.
Unfinished configurations are generated using all finished configurations and taking a subset of the contained characteristics. This way all generated configurations will be valid and lead to valid solutions. For the results that are presented in this chapter around $\frac{1}{7} \approx 15\%$ \todo{why this number} of characteristics is kept.
\subsection{Generating Preferences}
For the forest use case, the idea is that there are multiple types of user profiles. Each group profile is represented by a neutral, negative or positive attitude towards a characteristic. Now during data generation the attitude is converted to a preference using a normal distribution. \autoref{fig:Evaluation:DataGeneration} shows how the user profile can be converted to preferences. The actual group member profiles are shown in \autoref{tab:Evaluation:GroupMemberMappings}.
For the forest use case, the idea is that there are multiple types of user profiles. Each group profile is represented by a neutral, negative or positive attitude towards a characteristic. During data generation the attitude is converted to a preference \todo{hier evlt nochmal nennen, dass du Präferenzen zwischen 0 und 1 verwendest, steht aktuell nur in der Grafik} using a normal distribution. \autoref{fig:Evaluation:DataGeneration} shows how the user profile can be converted to preferences. The actual group member profiles are shown in \autoref{tab:Evaluation:GroupMemberMappings}.
\pgfplotsset{height=5cm,width=\textwidth,compat=1.8}
\pgfmathdeclarefunction{gauss}{2}{%
@@ -117,13 +132,14 @@ For the forest use case, the idea is that there are multiple types of user profi
\label{fig:Evaluation:DataGeneration}
\end{figure}
These user profiles can be used to generate rather homogenous groups but also to create groups that have interests that are more conflicting. The following group types are generated:
These user profiles can be used to generate rather homogenous groups but also to create groups that have interests that are more conflicting. The following group types are generated: \todo{wie genau sehen diese Gruppen aus? Aus wievielen Personen bestehen sie?}
\begin{itemize}
\item random groups (preferences are uniformly random)
\item heterogeneous groups (people adhere to one preference profile like forest owner, athlete, consumer, environmentalist)
\item homogeneous groups (only one preference profile for all group members which in this evaluation is the forest owner)
\end{itemize}
\todo[inline]{warum diese unterscheidungen}
\begin{table}
\begin{center}