10 years of Auto-sklearn

With NeurIPS just around the corner let’s take some time to reflect on Auto-sklearn, a project that we published ten years ago at NeurIPS and that started my career as a machine learning researcher and brought me into my current position as assistant professor for “AutoML and Optimization” at TU Dortmund University and the Lamarr institute for machine learning and artificial intelligence. It was also highly influential for my co-authors, Aaron Klein is now group lead of the EuroLLM project at the ELLIS institute Tübingen, Katharina Eggensperger is also a professor for Machine Learning and Artificial Intelligence, also at TU Dortmund University and the Lamarr institute for machine learning and artificial intelligence and Frank Hutter is a fellow at the ELLIS institute Tübingen and professor at the University of Freiburg (although he is currently on leave to push his startup priorlabs).

With NeurIPS just around the corner, let’s take some time to reflect on Auto-sklearn, a project that we published ten years ago at NeurIPS. While it is not among the very top-cited papers from NeurIPS 2015 (which are [1][2][3]; we’re currently at 13#), but as far as we can tell, it is the most-cited NeurIPS 2015 paper that is not about (deep) neural networks.

How it started

We started developing Auto-sklearn in 2014 to participate in the ChaLearn Automated Machine Learning Challenge by Isabelle Guyon, ChaLearn and collaborators. The first version was very rough: it could only work with the competition data format and could do only supervised classification. But it already had the core ingredients: meta-learning, Bayesian optimization and post-hoc ensembling. 

A few months later we wrapped it in a scikit-learn interface for an ICML workshop paper. This turned out to be one of the most important decisions of the entire project. Suddenly, AutoML felt familiar. You could treat it like any other estimator, fit it, and go home early.

As the challenge tasks became more demanding, regression, missing data, larger datasets, we added features as fast as we could write them. Auto-sklearn ultimately won the AutoML challenge series [link more link], including the deep learning track where we introduced neural networks to Auto-sklearn for the first time. For the second AutoML challenge in 2018 we pushed the meta-learning system further, and these improvements later became part of Auto-sklearn 2.0, which finally included neural networks of its own (not too deep, but deep enough). 

The contributions. 

One of auto-sklearn’s biggest contributions is how it simplified standard machine learning tasks for the masses. You didn’t need to memorize the quirks of every scikit-learn estimator or hand-craft hyperparameter search spaces. For students, researchers in other fields, and developers without ML backgrounds, auto-sklearn became an invitation to experiment. Just as importantly, it helped experienced practitioners test baselines quickly, validate feature engineering ideas, and explore search spaces without manual babysitting. It earned a reputation as a trustworthy first line of attack on real-world problems. 

Where it struggled

 While it is an amazing project, Auto-sklearn turned out to have several limitations:

1. Maintenance: maintaining an open source project is time intensive. Auto-sklearn builds on several other open source projects that we maintained as well: OpenML-Python, ConfigSpace and SMAC3. Keeping these up-to-date with the evolving scikit-learn and python ecosystem proved to be too demanding.
2. Reliance only on scikit-learn models: over time, XGBoost, CatBoost and LightGBM showed great performance, and perform extremely well by themselves, reducing the need for solving a combined algorithm selection and hyperparameter optimization problem. Incorporating them would definitely have improved Auto-sklearn.
3. Black box nature: The system worked well automatically but left little room for users to guide or shape the optimization process.
4. Newer approaches: Multi-layer stacking and modern ensembling techniques—of which AutoGluon is a great example, often outperform classical hyperparameter optimization alone.

The legacy.

Even though we no longer actively develop Auto-sklearn, its fingerprints are everywhere:

1. Auto-sklearn popularized the field of AutoML. By packaging AutoML in the popular scikit-learn API, Auto-sklearn made AutoML available to all users of the most popular data science programming language. Today, automated machine learning has its own conference: the international conference on automated machine learning (automl), which will take place in Ljubljana next year, have a look at https://automl.cc. 
2. Many of the improvements of Auto-sklearn still have an impact in the state-of-the-art AutoML toolkit AutoGluon. AutoGluon uses posthoc ensembling introduced by Auto-sklearn and also uses Meta-Learning in a similar fashion as we introduced it in the Auto-sklearn 2.0 paper.
3. Auto-sklearn sparked the development of several other projects that now live as standalone Python projects and have their own community:
   1. Auto-sklearn was among the first large-scale users of OpenML.org [paper] and made Matthias start the development OpenML-Python [paper], the Python connector to OpenML.org. Today, the Python connector powers projects such as AutoML benchmark and TabArena.
   2. ConfigSpace allows handling hierarchical configuration spaces and serves as a backend of many other projects in the realm of automated machine learning.
   3. SMAC3 [paper] contains state-of-the-art Bayesian optimization and we ported the toolkit from Java to Python and substantially extended it. Today it still provides excellent out-of-the-box performance [paper], based on its improved configuration [paper].

A huge shoutout to all co-authors of the NeurIPS 2015 paper Aaron Klein, Katharina Eggensperger, Manuel Blum, Tobias Springenberg and Frank Hutter, the additional co-authors of the 2022 JMLR paper Stefan Falkner and Marius Lindauer, and all contributors to the whole Auto-sklearn ecosystem. 

Here’s to ten more successful years of developing automated machine learning systems building, pushing the boundaries of machine learning, and making ML easier to use. We will soon, together with more than a dozen more AutoML researchers, publish a paper describing what the future might look like, so stay tuned.