A Large-Scale Study of Overtuning Mitigation Strategies in Hyperparameter Optimization

Hyperparameter optimization aims to identify configurations that result in machine learning models that generalize well to unseen data, by repeatedly evaluating and selecting configurations using noisy validation estimates. When excessive optimization of these estimates leads the optimizer to select a configuration that performs worse on unseen data than an earlier incumbent, this is referred to as overtuning, a phenomenon observed widely across hyperparameter optimization benchmarks. Despite its practical relevance, there is little systematic evidence on how overtuning can be mitigated. To address this gap, we conduct a controlled benchmark of 11 overtuning mitigation strategies across 51 TabArena datasets, covering binary classification, multiclass classification, and regression. Our benchmark reveals three main findings. First, many strategies reduce overtuning but decrease final test performance, showing that overtuning mitigation must be evaluated jointly with generalization performance. Second, post-hoc ensembling consistently reduces overtuning while improving generalization. Third, mitigation is highly regime-dependent: small binary classification problems benefit most, and extensive resampling can further reduce overtuning when made feasible under a fixed budget, for example, through early stopping cross-validation. With these findings, we outline future directions for overtuning mitigation, highlighting overtuning-aware post-hoc ensembling, practical mechanisms for stronger resampling, and combinations of the two as the most promising approaches.