SVM Hyperparameter Optimization Benchmark

Abstract

This package provides a hyperparameter optimization (HPO) benchmark based on a Support Vector Machine classifier (sklearn.svm.SVC). Each evaluation trains the SVM with stratified k-fold cross-validation on a small dataset that ships with scikit-learn, and returns the mean classification error (1 - accuracy), which is minimized.

The benchmark is useful for testing samplers on a real machine learning task that runs in seconds on a laptop, without downloading any data or precomputed tables. Its search space mixes three kinds of parameters:

• log-scale continuous parameters: C (regularization strength) and gamma (kernel coefficient),
• a categorical parameter: kernel,
• a conditional integer parameter: degree, which is only sampled when kernel == "poly".

APIs

class Problem(dataset_name: str = "breast_cancer", n_splits: int = 5, seed: int | None = None)

• dataset_name: Name of the built-in scikit-learn dataset. One of "breast_cancer", "wine" and "digits". See Problem.available_dataset_names.
• n_splits: Number of folds for stratified k-fold cross-validation. Must be at least 2.
• seed: Seed that controls how the cross-validation folds are shuffled. The same seed always gives the same result for the same parameters.

Search Space

Methods and Properties

• search_space: Return the full search space (including the conditional degree).
  • Returns: dict[str, optuna.distributions.BaseDistribution]
• directions: Return the optimization direction, which is [StudyDirection.MINIMIZE].
  • Returns: list[optuna.study.StudyDirection]
• available_dataset_names: The names of the available datasets.
  • Returns: list[str]
• __call__(trial: optuna.Trial): Sample a configuration from the trial and return its cross-validated error. degree is suggested only for the polynomial kernel.
  • Returns: float
• evaluate(params: dict[str, int | float | str]): Return the cross-validated error for a dictionary of parameters. degree is optional and defaults to 3.
  • Returns: float
• reseed(seed: int | None = None): Change the seed that controls how the folds are shuffled.

Features are standardized with StandardScaler inside the cross-validation pipeline, so no statistics leak from the validation folds.

Installation

$ pip install scikit-learn

Example

import optuna
import optunahub


svm_hpo = optunahub.load_module("benchmarks/svm_hpo")
problem = svm_hpo.Problem(dataset_name="wine", seed=42)

study = optuna.create_study(directions=problem.directions)
study.optimize(problem, n_trials=30)
print(study.best_params, study.best_value)

See example.py for a comparison of TPESampler and RandomSampler over several seeds. On the wine dataset with 30 trials and 5 seeds, it produced the following results:

TPE finds slightly better configurations, and after its warm-up phase its typical trial is much better than a random one, because it concentrates on the promising region of the search space.

Others

Running the Tests

From the root of the repository:

$ pip install pytest scikit-learn
$ pytest package/benchmarks/svm_hpo/tests