C. Audet and W. Hare, Derivative-free and blackbox optimization, 2017.

A. Auger, D. Brockhoff, and N. Hansen, GECCO (Companion) workshop on Black-Box Optimization Benchmarking (BBOB'2013), pp.1225-1232, 2013.

A. Auger, D. Brockhoff, N. Hansen, D. Tu?ar, T. Tu?ar et al., Benchmarking MATLAB's gamultiobj (NSGA-II) on the Bi-objective BBOB-2016 Test Suite, pp.1233-1239, 2016.

A. Auger, D. Brockhoff, N. Hansen, D. Tu?ar, T. Tu?ar et al., GECCO (Companion) workshop on Black-Box Optimization Benchmarking (BBOB'2016), pp.1241-1247, 2016.

A. Auger, D. Brockhoff, N. Hansen, D. Tu?ar, T. Tu?ar et al., The Impact of Search Volume on the Performance of RANDOMSEARCH on the Bi-objective BBOB-2016 Test Suite, GECCO (Companion) workshop on Black-Box Optimization Benchmarking (BBOB'2016), pp.1257-1264, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01435453

A. Auger, D. Brockhoff, N. Hansen, D. Tu?ar, T. Tu?ar et al., The Impact of Variation Operators on the Performance of SMS-EMOA on the Bi-objective BBOB-2016
URL : https://hal.archives-ouvertes.fr/hal-01435456

, GECCO (Companion) workshop on Black-Box Optimization Benchmarking (BBOB'2016), pp.1225-1232, 2016.

A. Auger and N. Hansen, Performance Evaluation of an Advanced Local Search Evolutionary Algorithm, Proceedings of the IEEE Congress on Evolutionary Computation, pp.1777-1784, 2005.

A. Auger and R. Ros, Benchmarking the pure random search on the BBOB-2009 testbed, pp.2479-2484, 2009.
URL : https://hal.archives-ouvertes.fr/inria-00430532

R. S. Barr, B. L. Golden, J. P. Kelly, M. G. Resende, and W. R. Stewart, Designing and reporting on computational experiments with heuristic methods, Journal of heuristics, vol.1, pp.9-32, 1995.

V. Beiranvand, W. Hare, and Y. Lucet, Best practices for comparing optimization algorithms, Optimization and Engineering, vol.18, pp.815-848, 2017.

N. Beume, B. Naujoks, and M. Emmerich, SMS-EMOA: Multiobjective Selection Based on Dominated Hypervolume, European Journal of Operational Research, vol.181, pp.1653-1669, 2007.

A. Blelly, M. Felipe-gomes, A. Auger, and D. Brockhoff, Stopping Criteria, Initialization, and Implementations of BFGS and their Effect on the BBOB Test Suite, GECCO (Companion) workshop on Black-Box Optimization Benchmarking (BBOB'2009), 2018.
URL : https://hal.archives-ouvertes.fr/hal-01811588

S. Bleuler, M. Laumanns, L. Thiele, and E. Zitzler, PISA-A Platform and Programming Language Independent Interface for Search Algorithms, Conference on Evolutionary Multi-Criterion Optimization, vol.2632, pp.494-508, 2003.

J. Bossek, Performance assessment of multi-objective evolutionary algorithms with the R package ecr, Companion Proceedings of the Genetic and Evolutionary Computation Conference, pp.1350-1356, 2018.

D. Brockhoff, T. D. Tran, and N. Hansen, Benchmarking Numerical Multiobjective Optimizers Revisited, Genetic and Evolutionary Computation Conference, 2015.
URL : https://hal.archives-ouvertes.fr/hal-01146741

D. Brockhoff, T. Tu?ar, A. Auger, and N. Hansen, Using well-understood singleobjective functions in multiobjective black-box optimization test suites, 2019.
URL : https://hal.archives-ouvertes.fr/hal-01296987

D. Brockhoff, T. Tu?ar, D. Tu?ar, T. Wagner, N. Hansen et al., Biobjective performance assessment with the COCO platform, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01315317

C. G. Broyden, The convergence of a class of double-rank minimization algorithms, Journal of the Institute of Mathematics and Its Applications, vol.6, pp.76-90, 1970.

S. Bubeck, Convex optimization: Algorithms and complexity, 2014.

M. R. Bussieck, S. P. Dirkse, and S. Vigerske, Paver 2.0: an open source environment for automated performance analysis of benchmarking data, Journal of Global Optimization, vol.59, pp.259-275, 2014.

K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II, IEEE Transactions on Evolutionary Computation, vol.6, pp.182-197, 2002.

B. Doerr, M. Fouz, M. Schmidt, and M. Wahlström, BBOB: Nelder-Mead with resize and halfruns, in Rothlauf, pp.2239-2246, 2009.

C. Doerr, H. Wang, F. Ye, S. Van-rijn, and T. Bäck, IOHprofiler: A benchmarking and profiling tool for iterative optimization heuristics, 2018.

E. D. Dolan and J. J. Moré, Benchmarking optimization software with performance profiles, Mathematical programming, vol.91, pp.201-213, 2002.

J. J. Durillo and A. J. Nebro, jMetal: a Java Framework for Multi-Objective Optimization, Advances in Engineering Software, vol.42, pp.760-771, 2011.

B. Efron and R. J. Tibshirani, An introduction to the bootstrap, 1994.

T. A. El-mihoub, A. A. Hopgood, L. Nolle, and A. Battersby, Hybrid genetic algorithms: A review, Engineering Letters, vol.13, pp.124-137, 2006.

S. Finck, N. Hansen, R. Ros, and A. Auger, Real-parameter black-box optimization benchmarking 2009: Presentation of the noiseless functions, Research Center PPE, 2009.

R. Fletcher, A new approach to variable metric algorithms, Computer journal, vol.13, pp.317-322, 1970.

M. Gaviano, D. Kvasov, D. Lera, and Y. D. Sergeyev, Software for generation of classes of test functions with known local and global minima for global optimization, ACM Transactions on Mathematical Software, pp.469-480, 2003.

A. Georges, A. Gleixner, G. Gojic, R. L. Gottwald, D. Haley et al., Feature-based algorithm selection for mixed integer programming, ZIB, Takustr, vol.7, p.14195, 2018.

D. Goldfarb, A family of variable metric updates derived by variational means, Mathematics of Computation, vol.24, pp.23-26, 1970.

N. Gould and J. Scott, A note on performance profiles for benchmarking software, ACM Transactions on Mathematical Software (TOMS), p.43, 2016.

N. Hansen, A. Auger, D. Brockhoff, D. Tu?ar, and T. Tu?ar, COCO: Performance assessment, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01315318

N. Hansen, A. Auger, S. Finck, and R. Ros, Real-parameter black-box optimization benchmarking 2009: Experimental setup, INRIA, 2009.
URL : https://hal.archives-ouvertes.fr/inria-00362649

N. Hansen, A. Auger, S. Finck, and R. Ros, Real-parameter black-box optimization benchmarking 2010: Experimental setup, INRIA, 2010.
URL : https://hal.archives-ouvertes.fr/inria-00462481

N. Hansen, A. Auger, O. Mersmann, T. Tu?ar, and D. Brockhoff, COCO: A platform for comparing continuous optimizers in a black-box setting, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01294124

N. Hansen, S. Finck, R. Ros, and A. Auger, Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions, INRIA, 2009.
URL : https://hal.archives-ouvertes.fr/inria-00362633

N. Hansen, S. Finck, R. Ros, and A. Auger, Real-parameter black-box optimization benchmarking 2009: Noisy functions definitions, INRIA, 2009.
URL : https://hal.archives-ouvertes.fr/inria-00369466

N. Hansen and A. Ostermeier, Completely Derandomized Self-Adaptation in Evolution Strategies, Evolutionary Computation, vol.9, pp.159-195, 2001.

N. Hansen, T. Tu?ar, O. Mersmann, A. Auger, and D. Brockhoff, COCO: The experimental procedure, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01294167

N. Hansen, A. Auger, R. Ros, S. Finck, and P. Po?ík, Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009, GECCO '10: Proceedings of the 12th annual conference comp on Genetic and evolutionary computation, pp.1689-1696, 2010.
URL : https://hal.archives-ouvertes.fr/hal-00545727

G. Harik and F. Lobo, A parameter-less genetic algorithm, Genetic and Evolutionary Computation Conference (GECCO 1999), vol.1, pp.258-265, 1999.

W. Hock and K. Schittkowski, Test Examples for Nonlinear Programming Codes, Lecture Notes in Economics and Mathematical Systems, vol.187, 1981.

A. Hoffman, M. Mannos, D. Sokolowsky, and N. Wiegmann, Computational experience in solving linear programs, Journal of the Society for Industrial and Applied Mathematics, vol.1, pp.17-33, 1953.

J. N. Hooker, Testing heuristics: We have it all wrong, Journal of heuristics, vol.1, pp.33-42, 1995.

H. Hoos and T. Stützle, Evaluating Las Vegas Algorithms-Pitfalls and Remedies, Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (UAI-98, pp.238-245, 1998.

J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in science & engineering, vol.9, p.90, 2007.

W. Huyer and A. Neumaier, Global optimization by multilevel coordinate search, J. of Global Optimization, vol.14, pp.331-355, 1999.

W. Huyer and A. Neumaier, Benchmarking of MCS on the noiseless function testbed, 2009.

D. S. Johnson, A theoretician's guide to the experimental analysis of algorithms, Data structures, near neighbor searches, and methodology: fifth and sixth DIMACS implementation challenges, vol.59, pp.215-250, 2002.

I. Loshchilov, M. Schoenauer, and M. Sebag, Black-Box Optimization Benchmarking of NIPOP-aCMA-ES and NBIPOP-aCMA-ES on the BBOB-2012 Noiseless Testbed, Companion Proceedings of the Genetic and Evolutionary Computation Conference, pp.269-276, 2012.
URL : https://hal.archives-ouvertes.fr/hal-00737409

O. Mersmann, M. Preuss, H. Trautmann, B. Bischl, and C. Weihs, Analyzing the bbob results by means of benchmarking concepts, Evolutionary computation, vol.23, pp.161-185, 2015.

J. J. Moré, B. S. Garbow, and K. E. Hillstrom, Testing unconstrained optimization software, ACM Transactions on Mathematical Software (TOMS), vol.7, pp.17-41, 1981.

J. J. Moré and S. M. Wild, Benchmarking derivative-free optimization algorithms, SIAM Journal on Optimization, vol.20, pp.172-191, 2009.

J. Nelder and R. Mead, The downhill simplex method, Computer Journal, vol.7, pp.308-313, 1965.

A. Nemirovski, Information-based complexity of convex programming, Lecture Notes, 1995.

Y. Nesterov, Lectures on convex optimization, vol.137, 2018.

M. J. Powell, The NEWUOA software for unconstrained optimization without derivatives, pp.255-297, 2006.

K. Price, Differential evolution vs. the functions of the second ICEO, Proceedings of the IEEE International Congress on Evolutionary Computation, pp.153-157, 1997.

T. Robi? and B. Filipi?, Differential evolution for multiobjective optimization, Evolutionary Multi-Criterion Optimization, pp.520-533, 2005.

R. Ros, Benchmarking the NEWUOA on the BBOB-2009 function testbed, pp.2421-2428, 2009.
URL : https://hal.archives-ouvertes.fr/inria-00377082

, Genetic and Evolutionary Computation Conference, 2009.

K. Schittkowski, More test examples for nonlinear programming codes, Lecture Nots in Economics and Mathematical Systems, vol.282, 1987.

D. F. Shanno, Conditioning of quasi-newton methods for function minimization, Mathematics of Computation, vol.24, pp.647-656, 1970.

S. S. Stevens, On the theory of scales of measurement, Science, pp.677-680, 1946.

Y. Tian, R. Cheng, X. Zhang, and Y. Jin, Platemo: A matlab platform for evolutionary multi-objective optimization, IEEE Computational Intelligence Magazine, vol.12, pp.73-87, 2017.

T. Tu?ar and B. Filipi?, Performance of the DEMO Algorithm on the Bi-objective BBOB Test Suite, Companion Proceedings of the Genetic and Evolutionary Computation Conference, pp.1249-1256, 2016.

T. Tu?ar, D. Brockhoff, and N. Hansen, Mixed-integer benchmark problems for single-and bi-objective optimization, Genetic and Evolutionary Computation Conference, pp.718-726, 2019.

T. Tu?ar, N. Hansen, and D. Brockhoff, Anytime Benchmarking of Budget-Dependent Algorithms with the COCO Platform, International Multiconference Information Society, pp.47-50, 2017.

K. Varelas, A. Auger, D. Brockhoff, N. Hansen, O. A. Elhara et al., A comparative study of large-scale variants of CMA-ES, International Conference on Parallel Problem Solving from Nature, pp.3-15, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01881454

V. Volz, B. Naujoks, P. Kerschke, and T. Tu?ar, Single-and multi-objective gamebenchmark for evolutionary algorithms, in Genetic and Evolutionary Computation Conference (GECCO, pp.647-655, 2019.

D. Whitley, S. Rana, J. Dzubera, and K. E. Mathias, Evaluating evolutionary algorithms, Artificial intelligence, vol.85, pp.245-276, 1996.

Q. Zhang, A. Zhou, Y. Jin, and . Rm-meda, A regularity model-based multiobjective estimation of distribution algorithm, IEEE Transactions on Evolutionary Computation, vol.12, pp.41-63, 2008.

E. Zitzler and L. Thiele, Multiobjective Optimization Using Evolutionary Algorithms -A Comparative Case Study, Conference on Parallel Problem Solving from Nature (PPSN V), vol.1498, pp.292-301, 1998.