TL;DR
- We present ERAHBO, a method that improves RL hyperparameter optimization by adaptively managing run-to-run randomness. Compared to a fixed evaluation budget per configuration, ERAHBO uses an adaptive stopping rule to halt evaluation of bad configurations immediately. Supported by a new, large-scale benchmark dataset with 50 seeds per configuration, ERAHBO efficiently identifies reliable hyperparameters across 19 RL tasks.
Check out the GitHub repository: https://github.com/LUH-AI/Efficient-Risk-Averse-BO
Taming RL Randomness with Adaptive Hyperparameter Optimization
Deep Reinforcement Learning (RL) has a well-known replication problem: running the same algorithm with the same hyperparameter configuration across different random seeds can yield vastly different results. This high variance means that traditional Hyperparameter Optimization (HPO) methods, which focus only on average performance, can lead to hyperparameter configurations that are prone to failure.
In literature, Risk-averse Bayesian Optimization (BO) algorithms such as Risk-averse Heteroscedastic Bayesian Optimization (RAHBO) can be applied to extract reliable hyperparameter configurations by modeling both expected return and variance simultaneously by using Gaussian processes. However, for RL, RAHBO suffers from an efficiency bottleneck: it allocates a fixed repetition budget of training runs to each hyperparameter candidate it evaluates, throwing away compute on unviable configurations.
Introducing ERAHBO: Be Rather Sure Where it Matters
To be presented at Reinforcement Learning Conference (RLC) 2026, Efficient Risk-averse Heteroscedastic Bayesian Optimization (ERAHBO) solves this inefficiency by replacing fixed evaluation budgets with an adaptive, confidence-based stopping rule.
Instead of treating every hyperparameter candidate equally, ERAHBO actively monitors the uncertainty of a configuration’s performance. The algorithm focuses on when a candidate’s optimistic bound drops below the current best-known configuration with a finite-sample certificate of suboptimality. The moment a hyperparameter set is certified as suboptimal, ERAHBO cuts it off and moves on. More samples are allocated to promising candidates, especially where uncertainty could genuinely alter the decision, preserving the theoretical guarantees of sublinear regret while demonstrating superior sampling efficiency.
Figure 1: Adaptive re-sampling in ERHABO. Left: Example objective with two maxima at different noise levels, and the risk-averse Mean-Variance (MV) objective (dashed). Middle: Upper and lower confidence bounds for the MV objective (dotted) together with the incumbent LCB (blue dash-dot) after three configurations; the next query (chosen via UCB) is shown by the green vertical line. Right: After two additional samples at , the UCB falls below the incumbent LCB, so ERHABO does not need to re-sample.
# a brief showcase
Fixed-budget Risk-Averse BO:
[Candidate A] -> Run 20 times (Wasted compute on bad setup)
[Candidate B] -> Run 20 times (Good setup)
ERAHBO (Adaptive):
[Candidate A] -> Run 2 times -> Certified Suboptimal -> STOP
[Candidate B] -> Run 20 times -> Allocates budget to verify top qualityA New Dataset for Risk-Aware AutoRL
To support the study of risk in automated RL, we also released an offline dataset of RL learning outcomes. Previous datasets often included up to ten seeds per configuration, which is insufficient for reliable heteroscedastic variance estimation.
This new dataset maps hyperparameters to post-training policy returns across 50 independent runs (seeds). Covering 512 configurations for major algorithms (DQN, SAC, PPO) across 19 environments in Brax, XLand-Minigrid, and Classic Control, it enables standardized and reproducible benchmarking of risk-aware HPO methods.
Proven Across Diverse Environments
When tested on this dataset, the benchmarks showed that while fixed-budget methods struggled to balance efficiency across domains, ERAHBO ranked first overall. It successfully filters out high-variance configurations and pinpoints robust, reliable hyperparameters with reduced compute budgets.
Figure 2: Mean-variance (MV) regret ranks aggregated by domain with the solid line and shaded area showing the mean and standard error, respectively. ERAHBO performs best overall. We observe different optimal repetition values in RAHBO across domains.
Figure 3: Repetition numbers in dataset experiments. Left: Compared with RAHBO using fixed sampling repetition, ERAHBO rejects the majority of configurations without spending the full budget. Right: ERAHBO invests more repetition in promising configurations with higher MV scores, demonstrating adaptivity through the confidence-based resampling behavior.
For more details, please check the link below:
Full paper (preprint): https://arxiv.org/abs/2607.26680
GitHub: https://github.com/LUH-AI/Efficient-Risk-Averse-BO
Dataset on Hugging Face: https://huggingface.co/datasets/autorl-org/Risk-aware_ARLbench
Comments are closed