01.intro.tex

\chapter{Introduction}\label{sec:intro}

% --------------------------------------------------------------------------------------------
% ------------------------------------------------     Motivation      
% --------------------------------------------------------------------------------------------
\section{Motivation}
    To find solutions to real-world engineering problems, it is often necessary to find and apply adequate parameters. The search for these parameters is a computationally expensive problem and requires optimization. This search is often achieved with the help of the parameter tuning or in other words, parameter optimization process.
    Traditionally engineers adhere to the manual parameter tuning; they put the effort in searching the optimal objectives guided by experience and intuition. Nevertheless, many of these optimization problems have vast search spaces that could be handled only with automatic tools. These tools can extrapolate and highlight the most perspective parameters from infinite space. At the same time, they struggles with multi-criteria decisions that are critical for engineering problems. For examples: architecture design, test generation, tuning machine-learning algorithms could be stated as multi-objective problems. To understand the space of possible solutions, they are represented on the Pareto front; i.e., the subset of solutions that could be not improved in some objective without degrading another.
    Multi-objective algorithms allow to find out some Pareto optimal solutions. Still, we require a massive amount of evaluations to obtain those solutions, and that is inappropriate for expensive problems. A common approach in the reduction of the final cost of the optimization algorithm is to replace some expensive estimations with cheaper ones with the help of surrogate models. The conventional algorithms to extrapolate available results are Bayesian Regression model (Kriging), neural networks, Support Vector Regression (SVR) or Tree regressions (Decision) estimators. However, almost all optimizations approaches use static models or aggregate several instances of one model type. These approaches lack variability and cannot be finely tuned.

    Ability to change the surrogate model strongly influences the optimization result. There is a demand for a strategy that allows us to combine multiple single-objective surrogate models to extrapolate multi-dimensional search spaces. This composition would make the surrogate model more versatile and capable of describing arbitrary optimization problems. Furthermore, it has been noted that the surrogate is domain-specific, and the same technique might perform differently on different problems. That is why extrapolation variability from the range of models improves final optimization results. However, only few researchers have addressed the solution of dynamic surrogate model selection. 

    Also, it is essential to obtain the necessary and sufficient number of samples to build an appropriate model. Unfortunately, to choose the optimum number of samples, it is required to have additional knowledge about a problem that is usually unknown. Moreover, arbitrary decisions on the sample size might be a reason that leads to inaccurate models and further wrong results.


% --------------------------------------------------------------------------------------------
% ------------------------------------------------     Objectives      
\section{Objectives}
For this thesis, we have chosen two broad objectives that we tried to achieve.
The first objective is to develop strategies that can dynamically compose the surrogate model from several single-objective models.
The second objective is to enhance parameter tuning with the best practices from multi-objective optimizations techniques. Successful fulfilment of those objectives means an overall improvement in the area that concerns with optimization of expensive black-box functions. Also, success implies the possibility of application of developed tools to the broader spectre of real-world problems. 

% --------------------------------------------------------------------------------------------
% ------------------------------------------------     Research Questions      
\section{Research questions}
\label{rq}
To achieve our objectives we defined three research questions, which we answer in this thesis.
Those research questions are:
\begin{description} 
    \item[RQ1:]\label{RQ1} Does the dynamic composition of different single-objective models improve the extrapolation of multi-objective problems?
    \item[RQ2:]\label{RQ2} Does a portfolio of surrogate models enhance optimization results?
    \item[RQ3:]\label{RQ3} Does a dynamic sampling plan help accelerate obtaining an optimal solution?
\end{description}

The purpose of this study is to provide a mechanism of a fined-grained models composition that allows making a superior multi-objective decision. Prerequisite for such solution is to reduce the number of actual evaluations while keeping the optimal quality of a decision.

% --------------------------------------------------------------------------------------------
% ------------------------------------------------     Overview
\section{Results overview}
    In this thesis, we introduce a modular structure for multi-objective parameter tuning that allows us to use various surrogate models and apply various optimization techniques. 
    The overall results were achieved in several stages: 1) In response to  RQ1, a composite model was implemented to combine several surrogate models. This solution made it possible to treat multi-objective extrapolation as a combination of single-objective models. 2) In response to RQ2 and RQ3, we developed a surrogate portfolio that enhances our strategy with the possibility to select surrogate models dynamically. Model selection is based on surrogate validation, which is also used as a criterion to check the sampling plan.
    An approach, combining all the aforementioned key features that answer the research questions was implemented under the name TutorM.

    The evaluation results from a wide range of problems showed excellent results and advantage of TutorM strategy over comparable approaches: NSGA2 and Hypermapper 2.0. TutorM also provides a possibility of scalable solutions for problems that demand that.

    The results can be used to improve the applicability of model-based optimization to a variety of expensive multi-objective parameter tuning problems.








    % Composition of surrogate models can be used to improve the applicability of model-based optimization to a verity of problems such as parameter tuning.

    % We have obtained that in the most majority of problems, the developed strategy provide significantly better results.

    % The approach we have developed in this study aims to excellent results in wide types of multi-objective problems. This approach
    % In numerous test problems, the portfolio with compositional-surrogates finds comparable solutions to standard MOEA (NSGA-II, MOEAD, MACO, NSPSO) doing considerably fewer evaluations (500 vs 10000). Dynamic sampling accelerates the start of the optimization process and prevents wasting resources. The surrogate portfolio allows adapting the optimization process to concrete domain problem and the number of available samples. 

% \section{Solution}

% \section{Organization of the Thesis}







    % Surrogate model or models based optimization is a common approach for a deal with expensive black-box function, but as far as the author is aware, there is no published research where the influence of heterogeneous portfolio of surrogate models was studied. The main target problem is an expensive multi-objective problem but the developed approach is also suitable for expensive single-objective optimization.
    % As black-box, we can not say what type of surface does the problem have. That is why it should be customized in the optimization process. The goal is to determine if the variability in extrapolation worth it. Introduce new surrogate-design-criteria for multi-objective hyperparameter optimization software.

    % It also provides backward compatibility for a single-objective problem. This optimization approach can significantly reduce expensive evaluations counts but torment from problems such as sampling size, type of surface and optimization techniques. We developed and adopted new technic in MBO such as portfolio surrogates, compositional model and surrogate validation. 

    % Multi-objective optimisation is an established parameter tuning technique. It is especially suited to solve complex, multidisciplinary design problems with an accent on system design.

    % When we talk about several objectives, the intention is to find good compromises rather than a single solution as in global optimization.
    % Since the solution for multi-objective optimization problems gives the appearance to a set of Pareto-optimal points, evolutionary optimization algorithms are ideal for handling multi-objective optimization problems.

    % General optimization methods could be classified into derivative and non-derivative methods. In this thesis focuses on non-derivative methods, as they are more suitable for parameter tuning. Therefore, they are also known as black-box methods and do not require any derivatives of the objective function to calculate the optimum.  Other benefits of these methods are that they are more likely to find a global optimum.