« Back to top page

Benchmarks

Benchmarks

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

Abstract Hyperparameter optimization benchmark introduced in the paper HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO. The original benchmark is available here. Please note that this benchmark provides the results only at the last epoch of each configuration. APIs class Problem(dataset_id: int, seed: int | None = None, metric_names: list[str] | None = None) dataset_id: ID of the dataset to use. It must be in the range of [0, 7].

HPOLib: Tabular Benchmarks for Hyperparameter Optimization and Neural Architecture Search

Abstract Hyperparameter optimization benchmark introduced in the paper Tabular Benchmarks for Hyperparameter Optimization and Neural Architecture Search. The original benchmark is available here. Please note that this benchmark provides the results only at the last epoch of each configuration. APIs class Problem(dataset_id: int, seed: int | None = None, metric_names: list[str] | None = None) dataset_id: ID of the dataset to use. It must be in the range of [0, 3].

NATS-Bench (NAS-Bench-201): Benchmarking NAS Algorithms for Architecture Topology and Size

Abstract Neural architecture search benchmark introduced in the paper NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size. The original benchmark is available here. Please note that this benchmark provides the results only at the last epoch of each architecture. The preliminary version is the NAS-Bench-201, but since the widely used name is NAS-Bench-201, we stick to the name, NAS-Bench-201 APIs class Problem(dataset_id: int, seed: int | None = None, metric_names: list[str] | None = None) dataset_id: ID of the dataset to use.

The blackbox optimization benchmarking (bbob) test suite

Abstract The blackbox optimization benchmarking (bbob) test suite comprises 24 noiseless single-objective test functions. BBOB is one of the most widely used test suites to evaluate and compare the performance of blackbox optimization algorithms. Each benchmark function is provided in dimensions [2, 3, 5, 10, 20, 40] with 110 instances. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob benchmark function to use. It must be in the range of [1, 24].

The blackbox optimization benchmarking biobj (bbob-biobj) and biobj-ext (bbob-biobj-ext) test suites

Abstract The bbob-biobj test suite was created by combining existing 55 noiseless single-objective test functions. bbob-biobj (and its extension, bbob-biobj-ext) has in total of 92 (= original 55 + additional 37) bi-objective functions. Each benchmark function is provided in dimensions [2, 3, 5, 10, 20, 40] with 15 instances. In this package, all the 92 functions are available. Please refer to the paper for details about each benchmark function. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob benchmark function to use.

The blackbox optimization benchmarking biobj-mixed-integer (biobj-bbob-mixint) test suite

Abstract The blackbox optimization benchmarking biobj-mixed-integer (bbob-biobj-mixint) test suite comprises 92 noiseless mixed-integer bi-objective test functions. Each benchmark function is provided in dimensions [5, 10, 20, 40, 80, 160] with 15 instances. Please refer to the paper for details about each benchmark function. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob benchmark function to use. It must be in the range of [1, 92].

The blackbox optimization benchmarking largescale (bbob-largescale) test suite

Abstract The blackbox optimization benchmarking largescale (bbob-largescale) test suite comprises high-dimensional 24 noiseless single-objective test functions. Each benchmark function is provided in dimensions [20, 40, 80, 160, 320, 640] with 15 instances. Please refer to the paper for details about each benchmark function. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob benchmark function to use. It must be in the range of [1, 24].

The blackbox optimization benchmarking mixed-integer (bbob-mixint) test suite

Abstract The blackbox optimization benchmarking mixed-integer (bbob-mixint) test suite comprises 24 noiseless mixed-integer single-objective test functions. Each benchmark function is provided in dimensions [5, 10, 20, 40, 80, 160] with 15 instances. Please refer to the paper for details about each benchmark function. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob benchmark function to use. It must be in the range of [1, 24].

The blackbox optimization benchmarking noisy (bbob-noisy) test suite

Abstract The blackbox optimization benchmarking noisy (bbob-noisy) test suite comprises 30 noisy test functions. Each benchmark function is provided in dimensions [2, 3, 5, 10, 20, 40] with 15 instances. Please refer to the paper for details about each benchmark function. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob benchmark function to use. It must be in the range of [101, 130]. dimension: Dimension of the benchmark function.

The blackbox optimization benchmarking-constrained (bbob-constrained) test suite

Abstract This package provides a wrapper of the COCO experiments libarary’s bbob-constrained test suite. APIs class Problem(function_id: int, dimension: int, instance_id: int = 1) function_id: ID of the bbob-constrained benchmark function to use. It must be in the range of [1, 54]. dimension: Dimension of the benchmark function. It must be in [2, 3, 5, 10, 20, 40]. instance_id: ID of the instance of the benchmark function. It must be in the range of [1, 15].

The DTLZ Problem Collection

Abstract This package provides a wrapper of the optproblems library’s DTLZ test suite, which consists of 7 kinds of continuous problems with variadic objectives and variables. For the details of the benchmark problems, please take a look at the original paper (Deb et al., 2001) in the reference section. APIs class Problem(function_id: int, n_objectives: int, dimension: int, k: int, **kwargs: Any) function_id: Function ID of the WFG problem in [1, 9].

The WFG Problem Collection

Abstract This package provides a wrapper of the optproblems library’s WFG test suite, which consists of 9 kinds of continuous problems with variadic objectives and variables. For the details of the benchmark problems, please take a look at the original paper (Huband et al., 2006) in the reference section. APIs class Problem(function_id: int, n_objectives: int, dimension: int, k: int | None = None, **kwargs: Any) function_id: Function ID of the WFG problem in [1, 9].

The ZDT Problem Collection

Abstract This package provides a wrapper of the optproblems library’s ZDT test suite, which consists of 6 kinds of benchmark problems. For the details of the benchmark problems, please take a look at the original paper (Zitzler et al., 2000) in the reference section. APIs class Problem(function_id: int, **kwargs: Any) function_id: Function ID of the WFG problem in [1, 9]. kwargs: Arbitrary keyword arguments, please refer to the optproblems documentation for more details.