Abstract When running parallel optimization experiments using tabular or surrogate benchmarks, each evaluation must be ordered based on the runtime that each configuration would take in reality. However, the evaluation of tabular or surrogate benchmarks, by design, does not take long. For this reason, the timing of each configuration must be ordered as if we actually evaluated each configuration.
This package provides a simulator that automatically handles this problem by internally managing the order of hyperparameter configuration evaluations.
Abstract This visualization module enables users to plot the target value over time with standard error bands. This module is especially convenient when we use parallel optimization such as Asynchronous optimization simulation.
Class or Function Names plot_target_over_time Installation This module requires the following dependencies:
matplotlib numpy APIs plot_target_over_time(study_list, *, ax=None, states=None, target=None, target_direction=None, cumtime_func=None, log_time_scale=True, n_steps=100, color=None, **plot_kwargs) study_list: A list of optuna.Study objects. Each study is treated as one run, and results are averaged across them.
Class or Function Names plot_sampling_speed Installation This module requires the following dependencies:
matplotlib scipy Example A minimal example would be the following:
from collections import defaultdict import matplotlib.pyplot as plt import optuna import optunahub def objective(trial) -> float: return trial.suggest_float("x", -5, 5)**2 studies = defaultdict(lambda: []) for i in range(3): sampler = optuna.samplers.RandomSampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, timeout=1.0) studies["Random"].append(study) sampler = optuna.samplers.TPESampler() study = optuna.create_study(sampler=sampler) study.optimize(objective, timeout=3.0) studies["TPE"].append(study) plot_sampling_speed = optunahub.