Multilevel Optimization
and Machine Learning
Kristin P. Bennett
Dept of Mathematical Sciences
Rensselaer Polytechnic Institute
Multilevel models are optimization
problems that require solution of other mathematical programming problems
as part of their constraints. Machine learning tasks such
as cross-validation can be naturally expressed as multilevel models.
Using cross-validation as an example, we will examine how multilevel
models are formulated. We will explore how bilevel models can
be transformed to alternative forms such as mathematical programs with equilibrium
constraints (MPEC), nonlinear programs, and integer programs. We will
discuss available algorithms for solving bilevel programs. We
will examine the versatility of bilevel framework for addressing different
machine learning tasks such as semi-supervised learning, variable selection,
variable scaling, and kernel selection. Finally, we will conclude
with a discussion of potential contributions and challenges of multilevel
optimization in machine learning.