What Types of Optimization are Suitable

Before considering the settings, let's first examine the strengths and weaknesses of the optimization methods available in Tunny.

Tunny supports many optimization methods, so you may be unsure which one to choose. Therefore, I will first provide the strengths and weaknesses of each method as described in the official documentation, along with a flowchart for selection.

Official characteristics of each sampler (excerpt)

	RandomSampler	TPESampler	CmaEsSampler	NSGAIISampler	GPSampler	BoTorchSampler
Float parameters	✓	✓	✓	▴	✓	✓
Integer parameters	✓	✓	✓	▴	✓	▴
Categorical parameters	✓	✓	▴	✓	✓	▴
Pruning	✓	✓	▴	×	▴	▴
Multivariate optimization	▴	✓	✓	▴	✓	✓
Conditional search space	✓	✓	▴	▴	▴	▴
Multi-objective optimization	✓	✓	×	✓ (▴ for single-objective)	×	✓
Batch optimization	✓	✓	✓	✓	▴	✓
Distributed optimization	✓	✓	✓	✓	▴	✓
Constrained optimization	×	✓	×	✓	×	✓
Recommended budgets (#trials)	as many as one likes	100 – 1000	1000 – 10000	100 – 10000	– 500	10 – 100

Selection Flowchart

The flowchart is more of a suggestion for what to try first, but it may not necessarily be suitable for the problem you are dealing with, so it's best to experiment with various methods in practice.

Now, let's take a look at the behavior when actually using it.

Optimization with Tunny's Default Settings

Unlike other optimization plugins, Tunny's default optimization method is TPE, not GA, and it supports various methods. Since many people may not be familiar with it, let's see what happens when we run it with the default settings.

When examining optimization methods, various benchmark functions are typically used, but let's not worry about that and instead solve a simple minimization problem of the sum of two NumberSliders.

The NumberSlider takes values in the range of 0 to 1, with a precision of up to 1/10,000.

Optimization using TPE

Now, let's run the optimization using Tunny's default settings without overthinking it.

The default settings are as follows:

• Method: TPE

• Number of trials: 100

The optimization uses randomness in some parts, so the results change each time it is executed. Therefore, we run the optimization 5 times.

It's also possible to directly specify the seed value for generating random numbers, which allows for the reproducibility of the optimization. We will use this feature in a later chapter.

The graph below shows the optimization history, with the vertical axis representing the objective function value (in this case, the sum of the two NumberSliders) and the horizontal axis representing the number of trials.

The lines shown indicate the change in the minimum value obtained in each optimization.

As we move to the right, the optimization progresses, and the minimum value decreases.

Note that the vertical axis is on a logarithmic scale.

In the table below, in addition to the results of the 5 optimizations, we also include the minimum value, mean value, and standard deviation among the 5 runs.

	1	2	3	4	5	minimum	Average	SD
TPE	0.0108	0.0017	0.0004	0.0211	0.0039	0.0004	0.0076	0.0077

You're right.

Even for such a simple problem where a human can easily see at a glance that the combination of variables X=Y=0 gives the minimum value, when there are many combinations of variables (\(10^4 \times 10^4=10^8\)), it doesn't quite reach 0.

Optimization using GA

Let's try the same thing with NSGAII, which corresponds to GA, a commonly used optimization method in Grasshopper. The results are as follows. Even with GA, it didn't reach 0.

	1	2	3	4	5	minimum	Average	SD
NSGAII	0.1799	0.1048	0.204	0.0445	0.0857	0.0445	0.1238	0.0595

It's worth noting that Tunny's default settings for GA are 50 individuals per generation and a total of 2 generations (100 individuals in total), which are not particularly suitable for optimization using GA as is. The results shown are based on these default settings.

Comparison of Multiple Methods

Tunny offers other optimization methods as well, so I've summarized the results of trying each of them below.

	1	2	3	4	5	minimum	Average	SD
TPE	0.0108	0.0017	0.0004	0.0211	0.0039	0.0004	0.0076	0.0077
NSGAII	0.1799	0.1048	0.2040	0.0445	0.0857	0.0445	0.1238	0.0595
Random	0.1315	0.1750	0.1266	0.0974	0.2091	0.0974	0.1479	0.0394
CMA-ES	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
GP-Botorch	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
GP-Optuna	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

GP and CMA-ES yielded results that solidly reached 0 even with 100 trials, but the others did not reach 0.

Although TPE did not reach 0, it had the smallest mean value and standard deviation among TPE, NSGAII, and Random in the 5 optimizations performed, making it the 4th best method.

The method with the largest mean value was Random at 0.1479, but this result was not significantly different from NSGAII's 0.1238.

Comparison of Optimization Time

Looking at the above results, you might think it's best to use the 3 methods that reached 0.

The drawback of these methods is that they take a considerable amount of time, even with 100 trials.

Below is a summary of the time it took to perform 100 trials.

sec	1	2	3	4	5	minimum	Average	SD
TPE	3.00	3.04	3.04	3.11	3.10	3.00	3.06	0.04
NSGAII	2.70	2.77	2.82	2.82	2.71	2.70	2.77	0.05
CMA-ES	41.09	39.87	39.31	40.54	39.99	39.31	40.16	0.61
GP-Botorch	19.80	23.49	20.83	21.65	19.54	19.54	21.06	1.43
GP-Optuna	19.97	19.55	19.80	20.09	24.09	19.55	20.70	1.70
Random	2.32	2.28	2.31	2.34	2.32	2.28	2.32	0.02

When considering the expected optimal value per unit time for 100 trials, although TPE does not reach the true optimal solution of 0, it is evident that it is a relatively cost-effective method.

Notes

• The 2 GP methods reach 0 in about 20 trials, so the actual time to obtain a value of 0 is faster.

• CMA-ES is generally a faster method than GP, but in this example, it is slower than GP due to the influence of the withMargin option, which is enabled by default in Tunny.

• The above results are based on optimization using Tunny's default settings and do not indicate the superiority or inferiority of the 6 methods in all cases.

In the next chapter, we will examine how adjusting the settings can lead to better results.

Trying to Improve Results by Adjusting Settings

Performing More Trials

It may simply be that there were not enough trials. Without changing the optimization method settings, let's see what happens when we increase the number of trials from 100 to 1000, which is 10 times more.

The targets are TPE, NSGAII, and Random, which did not reach 0 in 100 trials. The results are summarized in the table below.

Results Table

	1	2	3	4	5	minimum	Average	SD
TPE	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
NSGAII	0.0092	0.0180	0.0041	0.0040	0.0034	0.0034	0.0077	0.0055
Random	0.0640	0.0331	0.0465	0.0174	0.0200	0.0174	0.0362	0.0173

TPE history

GA history

Random history

TPE reached 0 in all cases, but GA did not reach 0.

However, compared to the average of 0.1238 with 100 trials, this time it is 0.0077, which is certainly a much smaller value.

We also checked how many trials it took for TPE to reach 0.

The results are as follows:

On average, over 5 runs, it reached 0 at 596 trials.

trial	1	2	3	4	5	minimum	Average	SD
TPE	935	556	392	759	337	337	595.8	224.4

It's simple, but it's clear that both TPE and GA will yield better values as the number of trials increases.

Changing Optimization Settings

Next, let's try changing the settings within the optimization methods.

Here, we fix the seed value used internally for random number generation to compare the optimization settings. This allows us to consider only the differences in settings.

TPE Section

Approach

Since TPE reaches 0 in about 600 trials even with the default settings, we will try changing the settings to see if we can obtain 0 with fewer trials.

The default settings are as follows:

Here, we will target the following three settings for modification.

The values in parentheses are the default values.

• Number of startup trials (10)
• Number of EI candidates (24)
• Consider Endpoints (False)

TPE and other methods classified as Bayesian optimization create a surrogate model and use it to determine the next point to explore.

The creation of the surrogate model is influenced by the number of initial samplings performed.

In Tunny, the Number of startup trials determines the number of initial samplings, and this many samplings are performed randomly instead of being optimized.

The Number of EI candidates is the number of samplings when evaluating the acquisition function.

According to the documentation [1], each of these is described as follows:

It is good to set Number of startup trials to about 1/10 of the total number of trials.
Increasing Number of EI candidates can be expected to improve the accuracy of the acquisition function evaluation.
When you want to carefully examine both ends of the variables, it is good to check Consider Endpoints.

Searching for Good Settings

Here are the results of performing 1000 trials while varying each setting.

The tables have Number of EI Candidates on the horizontal axis and Number of startup trials on the vertical axis.

In the "Number of trials to reach 0" table, cases where 0 was not reached are uniformly set to 1000.

Consider Endpoints	result	Number of trials to reach 0
False
True

When using the default settings, a value of 0 was reached on the 597th trial.

Let's check if the results improve when we perform the optimization 5 times without fixing the seed value, as we did at the beginning, using the following combinations that yielded good results above.

	case 1	case 2	case3
Number of startup trials	50	50	100
Number of EI candidates	30	50	50
Consider Endpoints	True	False	True

When Number of EI candidates is 20, the results show that 0 is reached with the fewest number of trials.

However, there is a large variation in the number of trials required to reach 0 depending on the settings.

Therefore, when selecting the above combination of settings, I chose from the results with Number of EI candidates of 30 or higher, which have less variability.

Optimization with Determined Settings

The following are the results of optimization performed using the three settings determined in the previous section.

Since the optimal value reached 0 in all cases, only the number of trials required to reach 0 is summarized.

	1	2	3	4	5	minimum	Average	SD
case 1	243	498	220	581	828	220	474	226
case 2	71	320	679	244	437	71	350	203
case 3	641	652	799	558	568	558	644	86
default	935	556	392	759	337	337	596	224

The optimization history is as follows:

Due to the logarithmic display, when the value reaches 0, the line extends outside the frame.

Case 1 and 2 show improvements in the minimum and average number of trials among the 5 runs compared to the default settings.

Case 3 has a slightly worse minimum and average number of trials than the default, but the standard deviation is smaller, and a value of 0 was obtained with less variability.

GA Section

Next, let's try changing the GA settings.

Trying Even More Iterations

We tried 1000 trials, but here are the results of trying 10000 trials, which is 10 times more.

By doing many iterations, the value gradually becomes smaller, but it did not reach 0.

	1	2	3	4	5	minimum	Average	SD
NSGAII	0.0011	0.0008	0.0005	0.0003	0.0015	0.0003	0.0008	0.0004

Changing the Crossover Method

Unlike other methods, if GA cannot obtain 0, it is likely that the settings are poor, so let's consider the settings.

In GA, good parent individuals are crossed to explore better individuals, so we will focus on the crossover method and try changing the settings.

In Tunny, you can choose from the following 6 types of crossover methods:

1. BLXApha Crossover
2. SBX Crossover
3. SPX Crossover
4. UNDX Crossover
5. Uniform Crossover ( default )
6. VSBX Crossover

The default is uniform crossover, but according to reference [2], uniform crossover performs relatively poorly in optimizations with many variables.

Therefore, changing this method is expected to improve the results.

Here is a summary of the results when we actually changed the method and performed the optimization.

For each method, we performed optimization 5 times, and we set the random number seed value to the same value for each run to ensure the same conditions.

Optimization Results

Number of Trials

History of each method for Seed 1

SBX, UNDX, and Uniform did not yield any results that reached 0, but BLXAlpha, SPX, and VSBX were able to obtain 0.

In particular, BLXAlpha obtained 0 in all cases with an average number of trials of 530, which is a significantly better result than the default Uniform.

The average number of trials of 530 is comparable to the results of TPE after adjusting the settings.

From these results, we can see that GA is not inferior to TPE as a method, but rather that the default optimization settings were not appropriate for the optimization problem at hand.

Summary

In this article, we used Tunny to examine how optimization settings affect the results.

What we confirmed in this article is only the case of optimizing a simple objective function A+B using Tunny's implementation.

When considering actual optimization problems, various cases can be considered, such as:

• What about multi-objective optimization?
• What if Wallacei or Galapagos is used instead of Tunny?
• What if the objective function is more complex?
• What if the number of variable combinations increases further?
• What if constraints are taken into account?

Depending on each case, the plugin, settings, and methods that can be used to obtain good results quickly will vary.

Always consider whether optimization is being performed with the required accuracy under your current settings.

If you want to know more about the meaning of each setting after reading this article, I recommend reading the book [1] written by the developers of Optuna, the optimization library used by Tunny, which is referenced as a resource in the article.

Let's optimize optimization and lead a more optimal optimization life.

References

• [1] : Optuna によるブラックボックス最適化
• [2] : Stjepan Picek and Martin Golub. Comparison of a Crossover Operator in Binary-coded Genetic Algorithms. WSEAS Transactions on Computers archive, pages 1064-1073, 2010.