skip to main content
research-article
Open access

Generative Expressive Robot Behaviors using Large Language Models

Published: 11 March 2024 Publication History
  • Get Citation Alerts
  • Abstract

    People employ expressive behaviors to effectively communicate and coordinate their actions with others, such as nodding to acknowledge a person glancing at them or saying "excuse me" to pass people in a busy corridor. We would like robots to also demonstrate expressive behaviors in human-robot interaction. Prior work proposes rule-based methods that struggle to scale to new communication modalities or social situations, while data-driven methods require specialized datasets for each social situation the robot is used in. We propose to leverage the rich social context available from large language models (LLMs) and their ability to generate motion based on instructions or user preferences, to generate expressive robot motion that is adaptable and composable, building upon each other. Our approach utilizes few-shot chain-of-thought prompting to translate human language instructions into parametrized control code using the robot's available and learned skills. Through user studies and simulation experiments, we demonstrate that our approach produces behaviors that users found to be competent and easy to understand. Supplementary material can be found at https://generative-expressive-motion.github.io/.

    Supplemental Material

    ZIP File

    References

    [1]
    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2023. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. In Conference on Robot Learning. PMLR, 287--318.
    [2]
    Amir Aly and Adriana Tapus. 2013. A model for synthesizing a combined verbal and nonverbal behavior based on personality traits in human-robot interaction. In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 325--332.
    [3]
    Niklas Bergstrom, Takayuki Kanda, Takahiro Miyashita, Hiroshi Ishiguro, and Norihiro Hagita. 2008. Modeling of natural human-robot encounters. In 2008 ieee/rsj international conference on intelligent robots and systems. IEEE, 2623--2629.
    [4]
    Nina Buchina, Sherin Kamel, and Emilia Barakova. 2016. Design and evaluation of an end-user friendly tool for robot programming. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 185--191.
    [5]
    Michael Jae-Yoon Chung, Justin Huang, Leila Takayama, Tessa Lau, and Maya Cakmak. 2016. Iterative design of a system for programming socially interactive service robots. In Social Robotics: 8th International Conference, ICSR 2016, Kansas City, MO, USA, November 1--3, 2016 Proceedings 8. Springer, 919--929.
    [6]
    Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. 2023. No, to the Right: Online Language Corrections for Robotic Manipulation via Shared Autonomy. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 93--101.
    [7]
    Porfirio David, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2022. Interaction Templates: A Data-Driven Approach for Authoring Robot Programs. In PLATEAU: 12th Annual Workshop at theIntersection of PL and HCI.
    [8]
    Ruta Desai, Fraser Anderson, Justin Matejka, Stelian Coros, James McCann, George Fitzmaurice, and Tovi Grossman. 2019. Geppetto: Enabling semantic design of expressive robot behaviors. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--14.
    [9]
    Malcolm Doering, Dylan F Glas, and Hiroshi Ishiguro. 2019. Modeling interaction structure for robot imitation learning of human social behavior. IEEE Transactions on Human-Machine Systems 49, 3 (2019), 219--231.
    [10]
    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022).
    [11]
    Paola Ferrarelli, María T Lázaro, and Luca Iocchi. 2018. Design of robot teaching assistants through multi-modal human-robot interactions. In Robotics in Education: Latest Results and Developments. Springer, 274--286.
    [12]
    Guy Hoffman and Wendy Ju. 2014. Designing robots with movement in mind. Journal of Human-Robot Interaction 3, 1 (2014), 91--122.
    [13]
    Chien-Ming Huang and Bilge Mutlu. 2012. Robot behavior toolkit: generating effective social behaviors for robots. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. 25--32.
    [14]
    Chien-Ming Huang and Bilge Mutlu. 2013. The repertoire of robot behavior: Enabling robots to achieve interaction goals through social behavior. Journal of Human-Robot Interaction 2, 2 (2013), 80--102.
    [15]
    Chien-Ming Huang and Bilge Mutlu. 2014. Learning-based modeling of multimodal behaviors for humanlike robots. In Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction. 57--64.
    [16]
    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2023. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Conference on Robot Learning. PMLR, 1769--1782.
    [17]
    Nusrah Hussain, Engin Erzin, T Metin Sezgin, and Yücel Yemez. 2022. Training socially engaging robots: modeling backchannel behaviors with batch reinforcement learning. IEEE Transactions on Affective Computing 13, 4 (2022), 1840--1853.
    [18]
    Yusuke Kato, Takayuki Kanda, and Hiroshi Ishiguro. 2015. May I help you? Design of human-like polite approaching behavior. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction. 35--42.
    [19]
    Alyssa Kubota, Emma IC Peterson, Vaishali Rajendren, Hadas Kress-Gazit, and Laurel D Riek. 2020. Jessie: Synthesizing social robot behaviors for personalized neurorehabilitation and beyond. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction. 121--130.
    [20]
    Minae Kwon, Hengyuan Hu, Vivek Myers, Siddharth Karamcheti, Anca Dragan, and Dorsa Sadigh. 2023. Toward Grounded Social Reasoning. arXiv preprint arXiv:2306.08651 (2023).
    [21]
    Minae Kwon, Sandy H Huang, and Anca D Dragan. 2018. Expressing robot incapability. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. 87--95.
    [22]
    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward Design with Language Models. In International Conference on Learning Representations (ICLR).
    [23]
    Nicola Leonardi, Marco Manca, Fabio Paternò, and Carmen Santoro. 2019. Triggeraction programming for personalising humanoid robot behaviour. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--13.
    [24]
    Zhongyu Li, Christine Cummings, and Koushil Sreenath. 2020. Animated cassie: A dynamic relatable robotic character. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 3739--3746.
    [25]
    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 9493--9500.
    [26]
    Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. 2023. Text2Motion: From Natural Language Instructions to Feasible Plans. Auton. Robots 47, 8 (Nov 2023), 1345--1365. https://doi.org/10.1007/s10514-023--10131--7
    [27]
    Phoebe Liu, Dylan F Glas, Takayuki Kanda, and Hiroshi Ishiguro. 2016. Datadriven HRI: Learning social behaviors by example from human--human interaction. IEEE Transactions on Robotics 32, 4 (2016), 988--1008.
    [28]
    Mina Marmpena, Angelica Lim, Torbjørn S Dahl, and Nikolas Hemion. 2019. Generating robotic emotional body language with variational autoencoders. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 545--551.
    [29]
    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 (2022).
    [30]
    Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. 2023. Large Language Models as General Pattern Machines. In Proceedings of the 7th Conference on Robot Learning (CoRL).
    [31]
    Michael Murray, Nick Walker, Amal Nanavati, Patricia Alves-Oliveira, Nikita Filippov, Allison Sauppe, Bilge Mutlu, and Maya Cakmak. 2022. Learning backchanneling behaviors for a social robot via data augmentation from human-human conversations. In Conference on robot learning. PMLR, 513--525.
    [32]
    OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
    [33]
    Nurziya Oralbayeva, Amir Aly, Anara Sandygulova, and Tony Belpaeme. 2023. Data-Driven Communicative Behaviour Generation: A Survey. ACM Transactions on Human-Robot Interaction (2023).
    [34]
    David Porfirio, Evan Fisher, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2019. Bodystorming human-robot interactions. In proceedings of the 32nd annual ACM symposium on user Interface software and technology. 479--491.
    [35]
    David Porfirio, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2018. Authoring and verifying human-robot interactions. In Proceedings of the 31st annual acm symposium on user interface software and technology. 75--86.
    [36]
    David Porfirio, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2020. Transforming robot programs based on social context. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1--12.
    [37]
    David Porfirio, Laura Stegner, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2023. Sketching Robot Programs On the Fly. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 584--593.
    [38]
    David J Porfirio, Laura Stegner, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2021. Figaro: A tabletop authoring environment for humanrobot interaction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.
    [39]
    Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 11523--11530.
    [40]
    Arjun Sripathy, Andreea Bobu, Zhongyu Li, Koushil Sreenath, Daniel S Brown, and Anca D Dragan. 2022. Teaching robots to span the space of functional expressive motion. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 13406--13413.
    [41]
    Michael Suguitan, Mason Bretan, and Guy Hoffman. 2019. Affective robot movement generation using cyclegans. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 534--535.
    [42]
    Michael Suguitan, Randy Gomez, and Guy Hoffman. 2020. MoveAE: modifying affective robot movements using classifying variational autoencoders. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction. 481--489.
    [43]
    Leila Takayama, Doug Dooley, and Wendy Ju. 2011. Expressing thought: improving robot readability with animation principles. In Proceedings of the 6th international conference on Human-robot interaction. 69--76.
    [44]
    GuanzhiWang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023).
    [45]
    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824--24837.
    [46]
    Sarah Woods, Michael Walters, Kheng Lee Koay, and Kerstin Dautenhahn. 2006. Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In 9th IEEE InternationalWorkshop on Advanced Motion Control, 2006. IEEE, 750--755.
    [47]
    Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. 2023. TidyBot: Personalized Robot Assistance with Large Language Models. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 3546--3553. https://doi.org/10.1109/IROS55552.2023.10341577
    [48]
    Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. 2023. Language to Rewards for Robotic Skill Synthesis. In Proceedings of the 7th Conference on Robot Learning (CoRL).
    [49]
    Allan Zhou and Anca D Dragan. 2018. Cost functions for robot motion style. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 3632--3639.

    Cited By

    View all
    • (2024)Using Large Language Models for Robot-Assisted Therapeutic Role-Play: Factuality is not enough!Proceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665886(1-6)Online publication date: 8-Jul-2024

    Index Terms

    1. Generative Expressive Robot Behaviors using Large Language Models

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      HRI '24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction
      March 2024
      982 pages
      ISBN:9798400703225
      DOI:10.1145/3610977
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 11 March 2024

      Check for updates

      Badges

      • Best Paper

      Author Tags

      1. generative expressive robot behaviors
      2. in-context learning
      3. language corrections

      Qualifiers

      • Research-article

      Data Availability

      Conference

      HRI '24
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 268 of 1,124 submissions, 24%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)657
      • Downloads (Last 6 weeks)217

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Using Large Language Models for Robot-Assisted Therapeutic Role-Play: Factuality is not enough!Proceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665886(1-6)Online publication date: 8-Jul-2024

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Get Access

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media