Reward Function Design for Crowd Simulation via Reinforcement Learning
Authors: 
Ariel Kwiatkowski, Vicky Kalogeiton, 
Julien Pettré, and Marie-Paule CaniAuthors Info & Claims
Ariel Kwiatkowski, Vicky Kalogeiton, Julien Pettré, and Marie-Paule CaniAuthors Info & ClaimsNovember 2023
Article No.: 4, Pages 1 - 7
Abstract
Crowd simulation is important for video-games design, since it enables to populate virtual worlds with autonomous avatars that navigate in a human-like manner. Reinforcement learning has shown great potential in simulating virtual crowds, but the design of the reward function is critical to achieving effective and efficient results. In this work, we explore the design of reward functions for reinforcement learning-based crowd simulation. We provide theoretical insights on the validity of certain reward functions according to their analytical properties, and evaluate them empirically using a range of scenarios, using the energy efficiency as the metric. Our experiments show that directly minimizing the energy usage is a viable strategy as long as it is paired with an appropriately scaled guiding potential, and enable us to study the impact of the different reward components on the behavior of the simulated crowd. Our findings can inform the development of new crowd simulation techniques, and contribute to the wider study of human-like navigation.
Supplementary Material
Appendix (NavRew-supp.pdf)
- Download
- 1.02 MB
References
[1]
Pieter Abbeel and Andrew Y. Ng. 2004. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning(ICML ’04). Association for Computing Machinery, New York, NY, USA, 1. https://doi.org/10.1145/1015330.1015430
[2]
Julien Bruneau, Anne-Hélène Olivier, and Julien Pettré. 2015. Going Through, Going Around: A Study on Individual Avoidance of Groups. IEEE Transactions on Visualization and Computer Graphics 21, 4 (April 2015), 9. https://doi.org/10.1109/TVCG.2015.2391862
[3]
William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, and Hugo Larochelle. 2019. Hyperbolic Discounting and Learning over Multiple Horizons. arXiv:1902.06865 [cs, stat] (Feb. 2019). arXiv:1902.06865.
[4]
Stephen J. Guy, Jatin Chhugani, Sean Curtis, Pradeep Dubey, Ming Lin, and Dinesh Manocha. 2010. PLEdestrians: A Least-Effort Approach to Crowd Simulation. Eurographics/ ACM SIGGRAPH Symposium on Computer Animation (2010), 10 pages. https://doi.org/10.2312/SCA/SCA10/119-128 Artwork Size: 10 pages ISBN: 9783905674279 Publisher: The Eurographics Association.
[5]
Kaidong Hu, Michael Brandon Haworth, Glen Berseth, Vladimir Pavlovic, Petros Faloutsos, and Mubbasir Kapadia. 2022. Heterogeneous Crowd Simulation using Parametric Reinforcement Learning. IEEE Transactions on Visualization and Computer Graphics (2022), 1–1. https://doi.org/10.1109/TVCG.2021.3139031
[6]
Ariel Kwiatkowski, Eduardo Alvarado, Vicky Kalogeiton, C. Karen Liu, Julien Pettré, Michiel van de Panne, and Marie‐Paule Cani. 2022. A Survey on Reinforcement Learning Methods in Character Animation. Computer Graphics Forum 41, 2 (May 2022), 613–639. https://doi.org/10.1111/cgf.14504
[7]
Ariel Kwiatkowski, Vicky Kalogeiton, Julien Pettré, and Marie-Paule Cani. 2023a. UGAE: A Novel Approach to Non-exponential Discounting. https://doi.org/10.48550/arXiv.2302.05740 arXiv:2302.05740 [cs].
[8]
Ariel Kwiatkowski, Vicky Kalogeiton, Julien Pettré, and Marie-Paule Cani. 2023b. Understanding reinforcement learned crowds. Computers & Graphics 110 (Feb. 2023), 28–37. https://doi.org/10.1016/j.cag.2022.11.007
[9]
Jaedong Lee, Jungdam Won, and Jehee Lee. 2018. Crowd simulation by deep reinforcement learning. In Proceedings of the 11th Annual International Conference on Motion, Interaction, and Games. ACM, Limassol Cyprus, 1–7. https://doi.org/10.1145/3274247.3274510
[10]
Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. 2017. Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems(AAMAS ’17). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 464–473. event-place: São Paulo, Brazil.
[11]
Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. 2018. Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning. arXiv:1709.10082 [cs] (May 2018). arXiv:1709.10082.
[12]
Pei Lv, Qingqing Yu, Boya Xu, Chaochao Li, Bing Zhou, and Mingliang Xu. 2022. Emotional Contagion-Aware Deep Reinforcement Learning for Antagonistic Crowd Simulation. https://doi.org/10.48550/arXiv.2105.00854 arXiv:2105.00854 [physics].
[13]
Abhishek Naik, Roshan Shariff, Niko Yasui, Hengshuai Yao, and Richard S. Sutton. 2019. Discounted Reinforcement Learning Is Not an Optimization Problem. arXiv:1910.02140 [cs] (Nov. 2019). arXiv:1910.02140.
[14]
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning(ICML ’99). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 278–287.
[15]
Andrew Y. Ng and Stuart J. Russell. 2000. Algorithms for Inverse Reinforcement Learning. In Proceedings of the Seventeenth International Conference on Machine Learning(ICML ’00). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 663–670.
[16]
Andreas Panayiotou, Theodoros Kyriakou, Marilena Lemonari, Yiorgos Chrysanthou, and Panayiotis Charalambous. 2022. CCP: Configurable Crowd Profiles. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings. ACM, Vancouver BC Canada, 1–10. https://doi.org/10.1145/3528233.3530712
[17]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs] (Aug. 2017).
[18]
L. Sun, J. Zhai, and W. Qin. 2019. Crowd Navigation in an Unknown and Dynamic Environment Based on Deep Reinforcement Learning. IEEE Access 7 (2019), 109544–109554. https://doi.org/10.1109/ACCESS.2019.2933492 Conference Name: IEEE Access.
[19]
Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA.
[20]
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems(NIPS’99). MIT Press, Cambridge, MA, USA, 1057–1063.
[21]
W. Toll and J. Pettré. 2021. Algorithms for Microscopic Crowd Simulation: Advancements in the 2010s. Computer Graphics Forum 40, 2 (May 2021), 731–754. https://doi.org/10.1111/cgf.142664
[22]
Michael W. Whittle. 2008. Gait analysis: an introduction (4th ed., reprinted ed.). Butterworth-Heinemann, Elsevier, Edinburgh.
[23]
Dong Xu, Xiao Huang, Zhenlong Li, and Xiang Li. 2020. Local motion simulation using deep reinforcement learning. Transactions in GIS 24, 3 (2020), 756–779. https://doi.org/10.1111/tgis.12620 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/tgis.12620.
[24]
Pei Xu and Ioannis Karamouzas. 2021. Human-Inspired Multi-Agent Navigation using Knowledge Distillation. arXiv:2103.10000 [cs] (March 2021). http://arxiv.org/abs/2103.10000 arXiv:2103.10000.
[25]
S. Zheng and H. Liu. 2019. Improved Multi-Agent Deep Deterministic Policy Gradient for Path Planning-Based Crowd Simulation. IEEE Access 7 (2019), 147755–147770. https://doi.org/10.1109/ACCESS.2019.2946659 Conference Name: IEEE Access.
[26]
George K. Zipf. 1949. Human Behaviour and the Principle of Least Effort. Addison-Wesley.
Index Terms
- Reward Function Design for Crowd Simulation via Reinforcement Learning
Recommendations
Crowd simulation by deep reinforcement learning
MIG '18: Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and GamesSimulating believable virtual crowds has been an important research topic in many research fields such as industry films, computer games, urban engineering, and behavioral science. One of the key capabilities agents should have is navigation, which is ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent SystemsRecent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Analysis and Design of an Improved R-learning
ICIS '09: Proceedings of the 2009 Eigth IEEE/ACIS International Conference on Computer and Information ScienceThis paper presents a modified R-learning according to the traditional average reward reinforcement learning algorithm. Reinforcement learning problems constitute an important class of learning and control problems faced by artificial intelligence ...
Comments
Information & Contributors
Information
Published In
November 2023
224 pages
ISBN:9798400703935
DOI:10.1145/3623264
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 November 2023
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Funding Sources
Conference
MIG '23
Sponsor:
MIG '23: The 16th ACM SIGGRAPH Conference on Motion, Interaction and Games
November 15 - 17, 2023
Rennes, France
Acceptance Rates
Overall Acceptance Rate -9 of -9 submissions, 100%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 100Total Downloads
- Downloads (Last 12 months)100
- Downloads (Last 6 weeks)8
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format