A. Agarwal, O. Dekel, and L. Xiao, Optimal algorithms for online convex optimization with multi-point bandit feedback, COLT '10: Proceedings of the 23rd Annual Conference on Learning Theory, 2010.

S. Arora, E. Hazan, and S. Kale, The multiplicative weights update method: A meta-algorithm and applications, Theory of Computing, vol.8, issue.1, pp.121-164, 2012.

P. Auer, N. Cesa-bianchi, Y. Freund, and R. E. Schapire, Gambling in a rigged casino: The adversarial multi-armed bandit problem, Proceedings of the 36th Annual Symposium on Foundations of Computer Science, 1995.

H. H. Bauschke, L. Patrick, and . Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01517477

M. Benaïm, Dynamics of stochastic approximation algorithms, Séminaire de Probabilités XXXIII, vol.1709, pp.1-68, 1999.

S. Bervoets, M. Bravo, and M. Faure, Learning with minimal information in continuous games, 2018.

G. Chen and M. Teboulle, Convergence analysis of a proximal-like minimization algorithm using Bregman functions, SIAM Journal on Optimization, vol.3, issue.3, pp.538-543, 1993.
DOI : 10.1137/0803026

K. Chung, On a stochastic approximation method, The Annals of Mathematical Statistics, vol.25, issue.3, pp.463-483, 1954.

J. Cohen, A. Héliou, and P. Mertikopoulos, Learning with bandit feedback in potential games. NIPS '17: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01643352

P. L. Combettes, Quasi-Fejérian analysis of some optimization algorithms, Inherently Parallel Algorithms in Feasibility and Optimization and Their Applications, pp.115-152, 2001.
DOI : 10.1016/s1570-579x(01)80010-0
URL : http://www.ann.jussieu.fr/~plc/ipa.pdf

P. L. Combettes and J. Pesquet, Stochastic quasi-Fejér block-coordinate fixed point iterations with random sweeping, SIAM Journal on Optimization, vol.25, issue.2, pp.1221-1248, 2015.
DOI : 10.1137/140971233
URL : http://arxiv.org/pdf/1404.7536

G. Debreu, A social equilibrium existence theorem, Proceedings of the National Academy of Sciences of the USA, vol.38, issue.10, pp.886-893, 1952.
DOI : 10.1017/ccol052123736x.003
URL : http://europepmc.org/articles/pmc1063675?pdf=render

A. D. Flaxman, H. B. Adam-tauman-kalai, and . Mcmahan, Online convex optimization in the bandit setting: gradient descent without a gradient, SODA '05: Proceedings of the 16th annual ACM-SIAM Symposium on Discrete Algorithms, pp.385-394, 2005.

D. J. Foster, T. Lykouris, K. Sridharan, and É. Tardos, Learning in games: Robustness of fast convergence, NIPS '16: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp.4727-4735, 2016.

Y. Freund and R. E. Schapire, Adaptive game playing using multiplicative weights, Games and Economic Behavior, vol.29, pp.79-103, 1999.
DOI : 10.1006/game.1999.0738
URL : http://www.cs.princeton.edu/~schapire/papers/FreundScYY.pdf

S. Ghadimi and G. Lan, Stochastic first-and zeroth-order methods for nonconvex stochastic programming, SIAM Journal on Optimization, vol.23, issue.4, pp.2341-2368, 2013.
DOI : 10.1137/120880811
URL : http://arxiv.org/pdf/1309.5549

J. C. Goodman, Note on existence and uniqueness of equilibrium points for concave N-person games, Econometrica, vol.48, issue.1, p.251, 1980.
DOI : 10.2307/1912028

P. Hall and C. C. Heyde, Martingale Limit Theory and Its Application. Probability and Mathematical Statistics, 1980.

R. D. Kleinberg, Nearly tight bounds for the continuum-armed bandit problem, NIPS' 04: Proceedings of the 18th Annual Conference on Neural Information Processing Systems, 2004.

J. M. Lee, Introduction to Smooth Manifolds, Graduate Texts in Mathematics, 2003.
DOI : 10.1007/978-0-387-21752-9

P. Mertikopoulos, E. V. Belmega, R. Negrel, and L. Sanguinetti, Distributed stochastic optimization via matrix exponential learning, IEEE Trans. Signal Process, vol.65, issue.9, pp.2277-2290, 2017.
DOI : 10.1109/tsp.2017.2656847
URL : https://hal.archives-ouvertes.fr/hal-01382285

P. Mertikopoulos, B. Lecouat, H. Zenati, C. Foo, V. Chandrasekhar et al., Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01891551

. Mertikopoulos, C. H. Panayotis, and . Papadimitriou, Georgios Piliouras. 2018b. Cycles in adversarial regularized learning. SODA '18: Proceedings of the 29th annual ACM-SIAM Symposium on Discrete Algorithms

P. Mertikopoulos and Z. Zhou, Learning in games with continuous action sets and unknown payoff functions, 2018.
DOI : 10.1007/s10107-018-1254-8
URL : http://arxiv.org/pdf/1608.07310

A. Nemirovski, A. Semen, . Juditsky, (. Guangui, A. George)-lan et al., Robust stochastic approximation approach to stochastic programming, SIAM Journal on Optimization, vol.19, issue.4, pp.1574-1609, 2009.
DOI : 10.1137/070704277
URL : https://hal.archives-ouvertes.fr/hal-00976649

A. Nemirovski, D. B. Semen, and . Yudin, Problem Complexity and Method Efficiency in Optimization, 1983.

Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course. No. 87 in Applied Optimization, 2004.
DOI : 10.1007/978-1-4419-8853-9

Y. Nesterov, Primal-dual subgradient methods for convex problems, Mathematical Programming, vol.120, issue.1, pp.221-259, 2009.
DOI : 10.2139/ssrn.912637
URL : http://dial.uclouvain.be/downloader/downloader.php?pid=boreal:4663&datastream=PDF_01&disclaimer=e3017c8ee412106bd1e8ee14d48ac71e1dd7a203f4355c9da4fe45261166af93

A. Orda, R. Rom, and N. Shimkin, Competitive routing in multi-user communication networks, IEEE/ACM Trans. Netw, vol.1, issue.5, pp.614-627, 1993.
DOI : 10.1109/infcom.1993.253270
URL : http://www.comnet.technion.ac.il/rom/Pubs/comp-rte.ps.gz

G. Palaiopanos, I. Panageas, and G. Piliouras, Multiplicative weights update with constant step-size in congestion games: Convergence, limit cycles and chaos, NIPS '17: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.

. Perkins, D. S. Steven, and . Leslie, Stochastic fictitious play with continuous action sets, Journal of Economic Theory, vol.152, pp.179-213, 2014.
DOI : 10.1016/j.jet.2014.04.008
URL : https://doi.org/10.1016/j.jet.2014.04.008

S. Perkins, P. Mertikopoulos, and D. S. Leslie, Mixed-strategy learning with continuous action sets, IEEE Trans. Autom. Control, vol.62, issue.1, pp.379-384, 2017.
DOI : 10.1109/tac.2015.2511930
URL : https://hal.archives-ouvertes.fr/hal-01382280

R. Rockafellar and . Tyrrell, Convex Analysis, 1970.

J. B. Rosen, Existence and uniqueness of equilibrium points for concave N-person games, Econometrica, vol.33, issue.3, pp.520-534, 1965.
DOI : 10.2307/1911749
URL : http://hdl.handle.net/2060/19650010164

O. Shamir, On the complexity of bandit and derivative-free stochastic convex optimization, COLT '13: Proceedings of the 26th Annual Conference on Learning Theory, 2013.

. Sorin, C. Sylvain, and . Wan, Finite composite games: Equilibria and dynamics, Journal of Dynamics and Games, vol.3, issue.1, pp.101-120, 2016.

J. C. Spall, A one-measurement form of simultaneous perturbation stochastic approximation, Automatica, vol.33, issue.1, pp.109-112, 1997.
DOI : 10.1016/s0005-1098(96)00149-5

V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire, Fast convergence of regularized learning in games. NIPS '15: Proceedings of the 29th International Conference on Neural Information Processing Systems, pp.2989-2997, 2015.

Y. Viossat and A. Zapechelnyuk, No-regret dynamics and fictitious play, Journal of Economic Theory, vol.148, issue.2, pp.825-842, 2013.
DOI : 10.1016/j.jet.2012.07.003
URL : https://hal.archives-ouvertes.fr/hal-00713871

M. Zinkevich, Online convex programming and generalized infinitesimal gradient ascent, ICML '03: Proceedings of the 20th International Conference on Machine Learning, pp.928-936, 2003.