Robust and efficient automated machine learning: systems, infrastructure and advances in hyperparameter optimization

Abstract: Automated Machine Learning (AutoML) is a new paradigm that democratizes machine learning and that will enable its widespread use. AutoML systems provide hands-free machine learning and automatically search for the best models, tune their hyperparameters, and ensemble them. Hyperparameter optimization (HPO) is a core part of AutoML research and also a research field on its own as models and datasets increase in size and complexity. However, current AutoML systems and HPO methods are resource-hungry and not yet robust enough for all the settings in which we would like to deploy them.

While machine learning automates programming by learning programs from data, AutoML goes one step further and allows us to leave such tasks entirely to the computer. It reduces the burden on the human expert and automates many steps required to build well-performing machine learning models. We are in dire need of such systems because of the increased usage of machine learning and the simultaneous shortage of machine learning practitioners and experts.

In this thesis, we make three contributions to increase the accessibility of machine learning by developing efficient and robust AutoML methods.

First, we address the problem of optimizing hyperparameters efficiently. Concretely, we survey the field of HPO, focusing on so-called multi-fidelity methods that efficiently optimize expensive machine learning algorithms by using cheaper approximations. Then, we develop a new method for transfer HPO, i.e., an HPO method that leverages knowledge gained on previous optimization tasks. It features strong empirical performance, worst-case performance guarantees, and is hyperparameter-free.

Second, we introduce our work on the efficient and robust AutoML system Auto-sklearn and its successor Auto-sklearn 2.0. We start by proposing Auto-sklearn, an extensible AutoML system that constructs linear machine learning pipelines from 15 classifiers, 16 preprocessing methods, and four data cleaning methods from scikit-learn. To improve over previous AutoML systems, we propose meta-learning and ensembling.

We extend on this with Auto-sklearn 2.0, in which we improve the meta-learning component and add new options to make Auto-sklearn suitable for a broader number of use cases. However, this opens up an HPO problem on the AutoML system level, and we introduce a meta-level meta-learning approach that adapts the AutoML system itself to the task at hand. Moreover, we describe how we used Auto-sklearn to win the 1st and 2nd ChaLearn AutoML competition.

Third, we enable AutoML research by contributing to the OpenML platform. Concretely, we create the OpenML-Python API, which gives us access to the OpenML platform with all its datasets. Then, to simplify access to datasets on OpenML and organize the thousands of datasets on OpenML, we develop OpenML benchmarking suites. These are curated datasets to standardize benchmarking practices in machine learning, and we have also used them in our works on HPO and AutoML.

Finally, we discuss how to improve the Auto-sklearn AutoML system further, pose several open questions to the field of AutoML to further increase the automation in machine learning practice and better understand the developed AutoML methods, and give suggestions on how to facilitate better benchmarking in AutoML using the OpenML platform.

In total, we demonstrate how machine learning can be made accessible by robust and efficient Automated Machine Learning and demonstrate substantial performance gains compared to previous AutoML systems

Object Details

• Location: Deutsche Nationalbibliothek Frankfurt am Main
• Extent: Online-Ressource
• Language: Englisch
• Notes: Universität Freiburg, Dissertation, 2022

Classification and Topics

• Keyword: Maschinelles Lernen ; Künstliche Intelligenz ; Maschinelles Lernen ; Überwachtes Lernen ; Benchmark ; API ; Python

Contributors, Places and Time

• Event: Veröffentlichung
• (where): Freiburg
• (who): Universität
• (when): 2022
• Creator: Feurer, Matthias
• Contributor: Hutter, Frank ; Vanschoren, Joaquin ; Albert-Ludwigs-Universität Freiburg. Institut für Informatik ; Maschinelles Lernen und Natürlichsprachliche Systeme, Professur Frank Hutter ; Albert-Ludwigs-Universität Freiburg. Fakultät für Angewandte Wissenschaften

Further information

• DOI: 10.6094/UNIFR/230516
• URN: urn:nbn:de:bsz:25-freidok-2305168
• Rights: Open Access; Der Zugriff auf das Objekt ist unbeschränkt möglich.
• Last update: 15.08.2025, 7:34 AM CEST