Real-World Multi-Objective Optimization Benchmark Problems (RE Problem Suite)

Abstract

This package provides the real-world multi-objective optimization benchmark problems (the RE problem suite) introduced in An Easy-to-use Real-world Multi-objective Problem Suite. The original benchmark implementation is available here. This package serves as a wrapper for a re-implementation of the original benchmark, ported to Python from the reference C source code (reproblem.c).

Note that ConstrainedProblem relies on optuna.trial.Trial.set_constraint, which requires Optuna v5.0.0 or newer.

Disclaimer

This benchmark collection modified some parts of the original implementation:

• The constraint clipping can be disabled. The original implementation hard-codes the clipping.
• The CRE1X series is added by exposing the constraint sum from the RE2X series, so CRE1X does not belong to the contribution of the original paper.
• The search space of CRE21 and RE31 use the log scale for x0 and x1 because the feasiblity ratio is extremely low without this modification. You can also disable it by setting <instance name>._enable_modification = False.

APIs

• Problem(problem_name: str)

  • problem_name: The name of an unconstrained benchmark problem. Available names are RE21, RE22, RE23, RE24, RE25, RE31, RE32, RE33, RE34, RE35, RE36, RE37, RE41, RE42, RE61, and RE91. Note that some of these problems internally reformulate their original constraints into an additional penalty objective, so they remain unconstrained from Optuna’s perspective.
  • Attributes:
    • search_space: Return the search space.
      • Returns: dict[str, optuna.distributions.BaseDistribution]
    • reference_point: Return the reference point.
      • Returns: list[float]
    • directions: Return the optimization directions.
      • Returns: list[optuna.study.StudyDirection]
    • metric_names: Return the objective names in the order returned by evaluate.
      • Returns: list[str] of length self.n_objectives.
    • original_problem_name: Return the name of the original problem this problem is derived from, as in Table 1 of the paper, e.g. FourBarTruss for RE21.
      • Returns: str
    • evaluate(params: dict[str, float]): Evaluate the objective function given a dictionary of parameters.
      • Args:
        • params: A dictionary representing decision variables like {"x0": x0_value, "x1": x1_value, ..., "xn": xn_value}. The number of parameters must be equal to self.n_variables.
      • Returns: List of length self.n_objectives.

• ConstrainedProblem(problem_name: str, clip_constraints: bool = True)

  • problem_name: The name of a constrained benchmark problem. Available names are CRE12, CRE13, CRE14, CRE15, CRE21, CRE22, CRE23, CRE24, CRE25, CRE31, CRE32, and CRE51.
  • clip_constraints: Whether to clip constraints by max(0, constraint_violation). Defaults to True following the original implementation.
  • Attributes:
    • search_space: Return the search space.
      • Returns: dict[str, optuna.distributions.BaseDistribution]
    • reference_point: Return the reference point.
      • Returns: list[float]
    • directions: Return the optimization directions.
      • Returns: list[optuna.study.StudyDirection]
    • metric_names: Return the objective names in the order returned by evaluate.
      • Returns: list[str] of length self.n_objectives.
    • original_problem_name: Return the name of the original problem this problem is derived from, as in Table 1 of the paper, e.g. FourBarTruss for RE21.
      • Returns: str
    • constraint_names: Return the constraint names used as the keys of evaluate_constraints. This property is only available in ConstrainedProblem.
      • Returns: list[str] of length self.n_constraints.
    • evaluate(params: dict[str, float]): Evaluate the objective function given a dictionary of parameters.
      • Args:
        • params: A dictionary representing decision variables, with the same format as in Problem.evaluate.
      • Returns: List of length self.n_objectives.
    • evaluate_constraints(params: dict[str, float]): Evaluate the constraint functions and return the constraint values keyed by their names. A trial is considered feasible when all the values are zero or less.
      • Args:
        • params: A dictionary representing the decision variables, with the same format and value range as in evaluate.
      • Returns: dict[str, float] of length self.n_constraints.

The properties and functions of classes in reproblem.reproblem_original are also available, such as lbound and ubound.

Objective and Constraint Names

Each RE problem models a different engineering problem, so unlike a synthetic suite, every problem has its own objectives. The names below carry the f_i and g_j indices used by the supplementary file, which defines each problem, so metric_names and constraint_names can be read directly against it. Which names a problem uses can also be inspected at runtime via problem.metric_names and problem.constraint_names.

As Table 1 of the paper shows, an unconstrained problem and its constrained counterpart are derived from the same original problem, e.g. RE31 and CRE21 are both the TwoBarTruss problem, and the pair shares both the original problem name and the original objectives listed below. The two variants differ in how they treat the original constraints: Problem folds them into an extra aggregated objective, whereas ConstrainedProblem exposes them through evaluate_constraints. CRE12-CRE15 are not in the paper’s Table 1: RE22-RE25 already fold real constraints into their violation objective, so this package exposes those same constraints as a constrained counterpart, following the naming convention of the pairs the paper does define.

Objectives (all minimized)

The original names below are the ones listed in Table 1 of the paper and returned by original_problem_name. Since the paper maximizes the annual cargo transport capacity of the ConceptualMarine problem, the corresponding objective is negated and prefixed with negative_.

The second objective of the TwoBarTruss problem is the stress of the member AC, which is the quantity that g2 bounds, rather than a joint displacement. A displacement would not share the 1e5 kPa limit that g3 imposes on the stress of the member BC, and reference [27] of the paper, which is where this problem comes from, likewise gives the two objectives as the volume of the truss and the stress of the member AC.

In Problem, the original constraints are reformulated into one extra objective that sums their violations, named f{n}_total_constraint_violation where n is self.n_objectives. For example, Problem("RE31").metric_names is ["f1_structural_weight", "f2_member_ac_stress", "f3_total_constraint_violation"]. The four problems whose original formulation has no constraint, namely RE21, RE34, RE37, and RE91, expose only the objectives listed above. RE91 is a special case: instead of aggregating the folded constraints, it keeps each of them as its own objective, so its violation objectives are named individually after the constraint they come from.

Constraints (feasible when zero or less)

Neither the paper nor the supplementary file names a constraint, since the supplementary file defines each one only by its formula. The descriptive part of each name below is therefore read off the constraint formula together with the original reference that the paper cites for the corresponding problem.

CRE51 is the exception that keeps the bare g_j indices. Its seven constraints are regression surrogates that bound quantities which the original reference leaves unnamed, so there is nothing to read off them.

Note that evaluate_constraints preserves the order of the original implementation, which is not sorted by the constraint index for CRE32, as the table above shows. This is a discrepancy between the paper and its implementations rather than a deliberate ordering: the C, Matlab, and Python versions all return 500000 - DWT before DWT - 3000, whereas the supplementary file defines them the other way around as g6 and g7. The two are the halves of the single two-sided bound 3000 <= DWT <= 500000, so neither order is more correct than the other, and the names above follow the implementation because that is the order the values come back in.

Note also that the original implementation returns the violation of each constraint rather than the constraint function value itself. The supplementary file writes every constraint as g_j(x) >= 0, and the implementation converts it into max(-g_j(x), 0), which is zero exactly when the constraint is satisfied and positive otherwise. The returned values are therefore always zero or greater, and a trial is feasible when every value is zero, which is consistent with Optuna’s convention that a constraint is satisfied when its value is zero or less.

Constraints folded into f{n}_total_constraint_violation

Every problem listed in the table above that has original constraints exposes them through its constrained counterpart, so what Problem folds into the violation objective can always be looked up there. RE21, RE34, and RE37 have no original constraints at all, so their Problem has nothing folded in, and RE91 is the special case described above that keeps each folded constraint as its own objective instead.

These names are documentation only, since constraint_names exists on ConstrainedProblem alone.

Example

from __future__ import annotations

import optuna
import optunahub


reproblem = optunahub.load_module("benchmarks/reproblem")
problem = reproblem.ConstrainedProblem("CRE21")
study = optuna.create_study(directions=problem.directions)
study.optimize(problem, n_trials=10)

if len(problem.directions) == 1:
    print(study.best_trial)
else:
    print(study.best_trials)

Reference

@article{tanabe2020easy,
  title={An easy-to-use real-world multi-objective optimization problem suite},
  author={Tanabe, Ryoji and Ishibuchi, Hisao},
  journal={Applied Soft Computing},
  volume={89},
  pages={106078},
  year={2020},
  publisher={Elsevier}
}