Showing 1–50 of 62 results for author: Mutlu, B

Robots in Family Routines: Development of and Initial Insights from the Family-Robot Routines Inventory

Authors: Michael F. Xu, Bengisu Cagiltay, Joseph Michaelis, Sarah Sebo, Bilge Mutlu

Abstract: Despite advances in areas such as the personalization of robots, sustaining adoption of robots for long-term use in families remains a challenge. Recent studies have identified integrating robots into families' routines and rituals as a promising approach to support long-term adoption. However, few studies explored the integration of robots into family routines and there is a gap in systematic mea… ▽ More Despite advances in areas such as the personalization of robots, sustaining adoption of robots for long-term use in families remains a challenge. Recent studies have identified integrating robots into families' routines and rituals as a promising approach to support long-term adoption. However, few studies explored the integration of robots into family routines and there is a gap in systematic measures to capture family preferences for robot integration. Building upon existing routine inventories, we developed Family-Robot Routines Inventory (FRRI), with 24 family routines and 24 child routine items, to capture parents' attitudes toward and expectations from the integration of robotic technology into their family routines. Using this inventory, we collected data from 150 parents through an online survey. Our analysis indicates that parents had varying perceptions for the utility of integrating robots into their routines. For example, parents found robot integration to be more helpful in children's individual routines, than to the collective routines of their families. We discuss the design implications of these preliminary findings, and how they may serve as a first step toward understanding the diverse challenges and demands of designing and integrating household robots for families. △ Less

Submitted 16 June, 2024; originally announced June 2024.

CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithms

Authors: Arda Sarp Yenicesu, Furkan B. Mutlu, Suleyman S. Kozat, Ozgur S. Oguz

Abstract: The utilization of the experience replay mechanism enables agents to effectively leverage their experiences on several occasions. In previous studies, the sampling probability of the transitions was modified based on their relative significance. The process of reassigning sample probabilities for every transition in the replay buffer after each iteration is considered extremely inefficient. Hence,… ▽ More The utilization of the experience replay mechanism enables agents to effectively leverage their experiences on several occasions. In previous studies, the sampling probability of the transitions was modified based on their relative significance. The process of reassigning sample probabilities for every transition in the replay buffer after each iteration is considered extremely inefficient. Hence, in order to enhance computing efficiency, experience replay prioritization algorithms reassess the importance of a transition as it is sampled. However, the relative importance of the transitions undergoes dynamic adjustments when the agent's policy and value function are iteratively updated. Furthermore, experience replay is a mechanism that retains the transitions generated by the agent's past policies, which could potentially diverge significantly from the agent's most recent policy. An increased deviation from the agent's most recent policy results in a greater frequency of off-policy updates, which has a negative impact on the agent's performance. In this paper, we develop a novel algorithm, Corrected Uniform Experience Replay (CUER), which stochastically samples the stored experience while considering the fairness among all other experiences without ignoring the dynamic nature of the transition importance by making sampled state distribution more on-policy. CUER provides promising improvements for off-policy continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training. △ Less

Submitted 13 June, 2024; originally announced June 2024.

Understanding On-the-Fly End-User Robot Programming

Authors: Laura Stegner, Yuna Hwang, David Porfirio, Bilge Mutlu

Abstract: Novel end-user programming (EUP) tools enable on-the-fly (i.e., spontaneous, easy, and rapid) creation of interactions with robotic systems. These tools are expected to empower users in determining system behavior, although very little is understood about how end users perceive, experience, and use these systems. In this paper, we seek to address this gap by investigating end-user experience with… ▽ More Novel end-user programming (EUP) tools enable on-the-fly (i.e., spontaneous, easy, and rapid) creation of interactions with robotic systems. These tools are expected to empower users in determining system behavior, although very little is understood about how end users perceive, experience, and use these systems. In this paper, we seek to address this gap by investigating end-user experience with on-the-fly robot EUP. We trained 21 end users to use an existing on-the-fly EUP tool, asked them to create robot interactions for four scenarios, and assessed their overall experience. Our findings provide insight into how these systems should be designed to better support end-user experience with on-the-fly EUP, focusing on user interaction with an automatic program synthesizer that resolves imprecise user input, the use of multimodal inputs to express user intent, and the general process of programming a robot. △ Less

Submitted 2 June, 2024; originally announced June 2024.

Comments: To appear at DIS'24. Stegner and Hwang contributed equally to this research

Tangible Scenography as a Holistic Design Method for Human-Robot Interaction

Authors: Amy Koike, Bengisu Cagiltay, Bilge Mutlu

Abstract: Traditional approaches to human-robot interaction design typically examine robot behaviors in controlled environments and narrow tasks. These methods are impractical for designing robots that interact with diverse user groups in complex human environments. Drawing from the field of theater, we present the construct of scenes -- individual environments consisting of specific people, objects, spatia… ▽ More Traditional approaches to human-robot interaction design typically examine robot behaviors in controlled environments and narrow tasks. These methods are impractical for designing robots that interact with diverse user groups in complex human environments. Drawing from the field of theater, we present the construct of scenes -- individual environments consisting of specific people, objects, spatial arrangements, and social norms -- and tangible scenography, as a holistic design approach for human-robot interactions. We created a design tool, Tangible Scenography Kit (TaSK), with physical props to aid in design brainstorming. We conducted design sessions with eight professional designers to generate exploratory designs. Designers used tangible scenography and TaSK components to create multiple scenes with specific interaction goals, characterize each scene's social environment, and design scene-specific robot behaviors. From these sessions, we found that this method can encourage designers to think beyond a robot's narrow capabilities and consider how they can facilitate complex social interactions. △ Less

Submitted 1 June, 2024; v1 submitted 29 May, 2024; originally announced May 2024.

The AI-DEC: A Card-based Design Method for User-centered AI Explanations

Authors: Christine P Lee, Min Kyung Lee, Bilge Mutlu

Abstract: Increasing evidence suggests that many deployed AI systems do not sufficiently support end-user interaction and information needs. Engaging end-users in the design of these systems can reveal user needs and expectations, yet effective ways of engaging end-users in the AI explanation design remain under-explored. To address this gap, we developed a design method, called AI-DEC, that defines four di… ▽ More Increasing evidence suggests that many deployed AI systems do not sufficiently support end-user interaction and information needs. Engaging end-users in the design of these systems can reveal user needs and expectations, yet effective ways of engaging end-users in the AI explanation design remain under-explored. To address this gap, we developed a design method, called AI-DEC, that defines four dimensions of AI explanations that are critical for the integration of AI systems -- communication content, modality, frequency, and direction -- and offers design examples for end-users to design AI explanations that meet their needs. We evaluated this method through co-design sessions with workers in healthcare, finance, and management industries who regularly use AI systems in their daily work. Findings indicate that the AI-DEC effectively supported workers in designing explanations that accommodated diverse levels of performance and autonomy needs, which varied depending on the AI system's workplace role and worker values. We discuss the implications of using the AI-DEC for the user-centered design of AI explanations in real-world systems. △ Less

Submitted 26 May, 2024; originally announced May 2024.

Journal ref: Designing Interactive Systems Conference, 2024, (DIS '24)

REX: Designing User-centered Repair and Explanations to Address Robot Failures

Authors: Christine P Lee, Pragathi Praveena, Bilge Mutlu

Abstract: Robots in real-world environments continuously engage with multiple users and encounter changes that lead to unexpected conflicts in fulfilling user requests. Recent technical advancements (e.g., large-language models (LLMs), program synthesis) offer various methods for automatically generating repair plans that address such conflicts. In this work, we understand how automated repair and explanati… ▽ More Robots in real-world environments continuously engage with multiple users and encounter changes that lead to unexpected conflicts in fulfilling user requests. Recent technical advancements (e.g., large-language models (LLMs), program synthesis) offer various methods for automatically generating repair plans that address such conflicts. In this work, we understand how automated repair and explanations can be designed to improve user experience with robot failures through two user studies. In our first, online study ($n=162$), users expressed increased trust, satisfaction, and utility with the robot performing automated repair and explanations. However, we also identified risk factors -- safety, privacy, and complexity -- that require adaptive repair strategies. The second, in-person study ($n=24$) elucidated distinct repair and explanation strategies depending on the level of risk severity and type. Using a design-based approach, we explore automated repair with explanations as a solution for robots to handle conflicts and failures, complemented by adaptive strategies for risk factors. Finally, we discuss the implications of incorporating such strategies into robot designs to achieve seamless operation among changing user needs and environments. △ Less

Submitted 26 May, 2024; originally announced May 2024.

Journal ref: Designing Interactive Systems Conference, 2024, (DIS '24)

"This really lets us see the entire world:" Designing a conversational telepresence robot for homebound older adults

Authors: Yaxin Hu, Laura Stegner, Yasmine Kotturi, Caroline Zhang, Yi-Hao Peng, Faria Huq, Yuhang Zhao, Jeffrey P. Bigham, Bilge Mutlu

Abstract: In this paper, we explore the design and use of conversational telepresence robots to help homebound older adults interact with the external world. An initial needfinding study (N=8) using video vignettes revealed older adults' experiential needs for robot-mediated remote experiences such as exploration, reminiscence and social participation. We then designed a prototype system to support these go… ▽ More In this paper, we explore the design and use of conversational telepresence robots to help homebound older adults interact with the external world. An initial needfinding study (N=8) using video vignettes revealed older adults' experiential needs for robot-mediated remote experiences such as exploration, reminiscence and social participation. We then designed a prototype system to support these goals and conducted a technology probe study (N=11) to garner a deeper understanding of user preferences for remote experiences. The study revealed user interactive patterns in each desired experience, highlighting the need of robot guidance, social engagements with the robot and the remote bystanders. Our work identifies a novel design space where conversational telepresence robots can be used to foster meaningful interactions in the remote physical environment. We offer design insights into the robot's proactive role in providing guidance and using dialogue to create personalized, contextualized and meaningful experiences. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: In proceedings of ACM Designing Interactive Systems (DIS) 2024

MSC Class: 68-06

"It Is Easy Using My Apps:" Understanding Technology Use and Needs of Adults with Down Syndrome

Authors: Hailey L. Johnson, Audra Sterling, Bilge Mutlu

Abstract: Assistive technologies for adults with Down syndrome (DS) need designs tailored to their specific technology requirements. While prior research has explored technology design for individuals with intellectual disabilities, little is understood about the needs and expectations of adults with DS. Assistive technologies should leverage the abilities and interests of the population, while incorporatin… ▽ More Assistive technologies for adults with Down syndrome (DS) need designs tailored to their specific technology requirements. While prior research has explored technology design for individuals with intellectual disabilities, little is understood about the needs and expectations of adults with DS. Assistive technologies should leverage the abilities and interests of the population, while incorporating age- and context-considerate content. In this work, we interviewed six adults with DS, seven parents of adults with DS, and three experts in speech-language pathology, special education, and occupational therapy to determine how technology could support adults with DS. In our thematic analysis, four main themes emerged, including (1) community vs. home social involvement; (2) misalignment of skill expectations between adults with DS and parents; (3) family limitations in technology support; and (4) considerations for technology development. Our findings extend prior literature by including the voices of adults with DS in how and when they use technology. △ Less

Submitted 24 March, 2024; originally announced March 2024.

Comments: 17 pages, 4 figures, to be published in ACM CHI 2024

"It's Not a Replacement:" Enabling Parent-Robot Collaboration to Support In-Home Learning Experiences of Young Children

Authors: Hui-Ru Ho, Edward Hubbard, Bilge Mutlu

Abstract: Learning companion robots for young children are increasingly adopted in informal learning environments. Although parents play a pivotal role in their children's learning, very little is known about how parents prefer to incorporate robots into their children's learning activities. We developed prototype capabilities for a learning companion robot to deliver educational prompts and responses to pa… ▽ More Learning companion robots for young children are increasingly adopted in informal learning environments. Although parents play a pivotal role in their children's learning, very little is known about how parents prefer to incorporate robots into their children's learning activities. We developed prototype capabilities for a learning companion robot to deliver educational prompts and responses to parent-child pairs during reading sessions and conducted in-home user studies involving 10 families with children aged 3-5. Our data indicates that parents want to work with robots as collaborators to augment parental activities to foster children's learning, introducing the notion of parent-robot collaboration. Our findings offer an empirical understanding of the needs and challenges of parent-child interaction in informal learning scenarios and design opportunities for integrating a companion robot into these interactions. We offer insights into how robots might be designed to facilitate parent-robot collaboration, including parenting policies, collaboration patterns, and interaction paradigms. △ Less

Submitted 20 March, 2024; originally announced March 2024.

Crowdsourcing Task Traces for Service Robotics

Authors: David Porfirio, Allison Sauppé, Maya Cakmak, Aws Albarghouthi, Bilge Mutlu

Abstract: Demonstration is an effective end-user development paradigm for teaching robots how to perform new tasks. In this paper, we posit that demonstration is useful not only as a teaching tool, but also as a way to understand and assist end-user developers in thinking about a task at hand. As a first step toward gaining this understanding, we constructed a lightweight web interface to crowdsource step-b… ▽ More Demonstration is an effective end-user development paradigm for teaching robots how to perform new tasks. In this paper, we posit that demonstration is useful not only as a teaching tool, but also as a way to understand and assist end-user developers in thinking about a task at hand. As a first step toward gaining this understanding, we constructed a lightweight web interface to crowdsource step-by-step instructions of common household tasks, leveraging the imaginations and past experiences of potential end-user developers. As evidence of the utility of our interface, we deployed the interface on Amazon Mechanical Turk and collected 207 task traces that span 18 different task categories. We describe our vision for how these task traces can be operationalized as task models within end-user development tools and provide a roadmap for future work. △ Less

Submitted 20 March, 2024; originally announced March 2024.

Comments: Published in the companion proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction

End-User Development for Human-Robot Interaction

Authors: Laura Stegner, David Porfirio, Laura M. Hiatt, Séverin Lemaignan, Ross Mead, Bilge Mutlu

Abstract: End-user development (EUD) represents a key step towards making robotics accessible for experts and nonexperts alike. Within academia, researchers investigate novel ways that EUD tools can capture, represent, visualize, analyze, and test developer intent. At the same time, industry researchers increasingly build and ship programming tools that enable customers to interact with their robots. Howeve… ▽ More End-user development (EUD) represents a key step towards making robotics accessible for experts and nonexperts alike. Within academia, researchers investigate novel ways that EUD tools can capture, represent, visualize, analyze, and test developer intent. At the same time, industry researchers increasingly build and ship programming tools that enable customers to interact with their robots. However, despite this growing interest, the role of EUD within HRI is not well defined. EUD struggles to situate itself within a growing array of alternative approaches to application development, such as robot learning and teleoperation. EUD further struggles due to the wide range of individuals who can be considered end users, such as independent third-party application developers, consumers, hobbyists, or even employees of the robot manufacturer. Key questions remain such as how EUD is justified over alternate approaches to application development, which contexts EUD is most suited for, who the target users of an EUD system are, and where interaction between a human and a robot takes place, amongst many other questions. We seek to address these challenges and questions by organizing the first End-User Development for Human-Robot Interaction (EUD4HRI) workshop at the 2024 International Conference of Human-Robot Interaction. The workshop will bring together researchers with a wide range of expertise across academia and industry, spanning perspectives from multiple subfields of robotics, with the primary goal being a consensus of perspectives about the role that EUD must play within human-robot interaction. △ Less

Submitted 27 February, 2024; originally announced February 2024.

Comments: Accepted at Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24 Companion), March 11--14, 2024, Boulder, CO, USA

Factors that Affect Personalization of Robots for Older Adults

Authors: Laura Stegner, Emmanuel Senft, Bilge Mutlu

Abstract: We introduce a taxonomy of important factors to consider when designing interactions with an assistive robot in a senior living facility. These factors are derived from our reflection on two field studies and are grouped into the following high-level categories: primary user (residents), care partners, robot, facility and external circumstances. We outline how multiple factors in these categories… ▽ More We introduce a taxonomy of important factors to consider when designing interactions with an assistive robot in a senior living facility. These factors are derived from our reflection on two field studies and are grouped into the following high-level categories: primary user (residents), care partners, robot, facility and external circumstances. We outline how multiple factors in these categories impact different aspects of personalization, such as adjusting interactions based on the unique needs of a resident or modifying alerts about the robot's status for different care partners. This preliminary taxonomy serves as a framework for considering how to deploy personalized assistive robots in the complex caregiving ecosystem. △ Less

Submitted 27 February, 2024; originally announced February 2024.

Comments: Presented at CONCATENATE Workshop at HRI 2023 in Stockholm, Sweden

Report number: CONCATENATE/2023/04

Toward Family-Robot Interactions: A Family-Centered Framework in HRI

Authors: Bengisu Cagiltay, Bilge Mutlu

Abstract: As robotic products become more integrated into daily life, there is a greater need to understand authentic and real-world human-robot interactions to inform product design. Across many domestic, educational, and public settings, robots interact with not only individuals and groups of users, but also families, including children, parents, relatives, and even pets. However, products developed to da… ▽ More As robotic products become more integrated into daily life, there is a greater need to understand authentic and real-world human-robot interactions to inform product design. Across many domestic, educational, and public settings, robots interact with not only individuals and groups of users, but also families, including children, parents, relatives, and even pets. However, products developed to date and research in human-robot and child-robot interactions have focused on the interaction with their primary users, neglecting the complex and multifaceted interactions between family members and with the robot. There is a significant gap in knowledge, methods, and theories for how to design robots to support these interactions. To inform the design of robots that can support and enhance family life, this paper provides (1) a narrative review exemplifying the research gap and opportunities for family-robot interactions and (2) an actionable family-centered framework for research and practices in human-robot and child-robot interaction. △ Less

Submitted 25 January, 2024; originally announced January 2024.

A System for Human-Robot Teaming through End-User Programming and Shared Autonomy

Authors: Michael Hagenow, Emmanuel Senft, Robert Radwin, Michael Gleicher, Michael Zinn, Bilge Mutlu

Abstract: Many industrial tasks-such as sanding, installing fasteners, and wire harnessing-are difficult to automate due to task complexity and variability. We instead investigate deploying robots in an assistive role for these tasks, where the robot assumes the physical task burden and the skilled worker provides both the high-level task planning and low-level feedback necessary to effectively complete the… ▽ More Many industrial tasks-such as sanding, installing fasteners, and wire harnessing-are difficult to automate due to task complexity and variability. We instead investigate deploying robots in an assistive role for these tasks, where the robot assumes the physical task burden and the skilled worker provides both the high-level task planning and low-level feedback necessary to effectively complete the task. In this article, we describe the development of a system for flexible human-robot teaming that combines state-of-the-art methods in end-user programming and shared autonomy and its implementation in sanding applications. We demonstrate the use of the system in two types of sanding tasks, situated in aircraft manufacturing, that highlight two potential workflows within the human-robot teaming setup. We conclude by discussing challenges and opportunities in human-robot teaming identified during the development, application, and demonstration of our system. △ Less

Submitted 22 January, 2024; originally announced January 2024.

Comments: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24), March 11 - 14, 2024, Boulder, CO, USA

Sprout: Designing Expressivity for Robots Using Fiber-Embedded Actuator

Authors: Amy Koike, Michael Wehner, Bilge Mutlu

Abstract: In this paper, we explore how techniques from soft robotics can help create a new form of robot expression. We present Sprout, a soft expressive robot that conveys its internal states by changing its body shape. Sprout can extend, bend, twist, and expand using fiber-embedded actuators integrated into its construction. These deformations enable Sprout to express its internal states, for example, by… ▽ More In this paper, we explore how techniques from soft robotics can help create a new form of robot expression. We present Sprout, a soft expressive robot that conveys its internal states by changing its body shape. Sprout can extend, bend, twist, and expand using fiber-embedded actuators integrated into its construction. These deformations enable Sprout to express its internal states, for example, by expanding to express anger and bending its body sideways to express curiosity. Through two user studies, we investigated how users interpreted Sprout's expressions, their perceptions of Sprout, and their expectations from future iterations of Sprout's design. We argue that the use of soft actuators opens a novel design space for robot expressions to convey internal states, emotions, and intent. △ Less

Submitted 11 January, 2024; originally announced January 2024.

Comments: 10 pages, 5 figures

Making Informed Decisions: Supporting Cobot Integration Considering Business and Worker Preferences

Authors: Dakota Sullivan, Nathan Thomas White, Andrew Schoen, Bilge Mutlu

Abstract: Robots are ubiquitous in small-to-large-scale manufacturers. While collaborative robots (cobots) have significant potential in these settings due to their flexibility and ease of use, proper integration is critical to realize their full potential. Specifically, cobots need to be integrated in ways that utilize their strengths, improve manufacturing performance, and facilitate use in concert with h… ▽ More Robots are ubiquitous in small-to-large-scale manufacturers. While collaborative robots (cobots) have significant potential in these settings due to their flexibility and ease of use, proper integration is critical to realize their full potential. Specifically, cobots need to be integrated in ways that utilize their strengths, improve manufacturing performance, and facilitate use in concert with human workers. Effective integration requires careful consideration and the knowledge of roboticists, manufacturing engineers, and business administrators. We propose an approach involving the stages of planning, analysis, development, and presentation, to inform manufacturers about cobot integration within their facilities prior to the integration process. We contextualize our approach in a case study with an SME collaborator and discuss insights learned. △ Less

Submitted 10 January, 2024; originally announced January 2024.

Comments: 9 pages, 9 figures. To be published in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24)

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

Authors: Callie Y. Kim, Christine P. Lee, Bilge Mutlu

Abstract: Large-language models (LLMs) hold significant promise in improving human-robot interaction, offering advanced conversational skills and versatility in managing diverse, open-ended user requests in various tasks and domains. Despite the potential to transform human-robot interaction, very little is known about the distinctive design requirements for utilizing LLMs in robots, which may differ from t… ▽ More Large-language models (LLMs) hold significant promise in improving human-robot interaction, offering advanced conversational skills and versatility in managing diverse, open-ended user requests in various tasks and domains. Despite the potential to transform human-robot interaction, very little is known about the distinctive design requirements for utilizing LLMs in robots, which may differ from text and voice interaction and vary by task and context. To better understand these requirements, we conducted a user study (n = 32) comparing an LLM-powered social robot against text- and voice-based agents, analyzing task-based requirements in conversational tasks, including choose, generate, execute, and negotiate. Our findings show that LLM-powered robots elevate expectations for sophisticated non-verbal cues and excel in connection-building and deliberation, but fall short in logical communication and may induce anxiety. We provide design implications both for robots integrating LLMs and for fine-tuning LLMs for use with robots. △ Less

Submitted 6 January, 2024; originally announced January 2024.

Comments: 10 pages, 4 figures. Callie Y. Kim and Christine P. Lee contributed equally to the work. To be published in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24), March 11--14, 2024, Boulder, CO, USA

Characterizing Barriers and Technology Needs in the Kitchen for Blind and Low Vision People

Authors: Ru Wang, Nihan Zhou, Tam Nguyen, Sanbrita Mondal, Bilge Mutlu, Yuhang Zhao

Abstract: Cooking is a vital yet challenging activity for people with visual impairments (PVI). It involves tasks that can be dangerous or difficult without vision, such as handling a knife or adding a suitable amount of salt. A better understanding of these challenges can inform the design of technologies that mitigate safety hazards and improve the quality of the lives of PVI. Furthermore, there is a need… ▽ More Cooking is a vital yet challenging activity for people with visual impairments (PVI). It involves tasks that can be dangerous or difficult without vision, such as handling a knife or adding a suitable amount of salt. A better understanding of these challenges can inform the design of technologies that mitigate safety hazards and improve the quality of the lives of PVI. Furthermore, there is a need to understand the effects of different visual abilities, including low vision and blindness, and the role of rehabilitation training where PVI learn cooking skills and assistive technologies. In this paper, we aim to comprehensively characterize PVI's challenges, strategies, and needs in the kitchen from the perspectives of both PVI and rehabilitation professionals. Through a contextual inquiry study, we observed 10 PVI, including six low vision and four blind participants, when they cooked dishes of their choices in their own kitchens. We then interviewed six rehabilitation professionals to explore their training strategies and technology recommendations. Our findings revealed the differences between low vision and blind people during cooking as well as the gaps between training and reality. We suggest improvements for rehabilitation training and distill design considerations for future assistive technology in the kitchen. △ Less

Submitted 9 October, 2023; originally announced October 2023.

Investigating the Day-to-Day Experiences of Users with Traumatic Brain Injury with Conversational Agents

Authors: Yaxin Hu, Hajin Lim, Hailey L. Johnson, Josephine M. O'Shaughnessy, Lisa Kakonge, Lyn S. Turkstra, Melissa C. Duff, Catalina L. Toma, Bilge Mutlu

Abstract: Traumatic brain injury (TBI) can cause cognitive, communication, and psychological challenges that profoundly limit independence in everyday life. Conversational Agents (CAs) can provide individuals with TBI with cognitive and communication support, although little is known about how they make use of CAs to address injury-related needs. In this study, we gave nine adults with TBI an at-home CA for… ▽ More Traumatic brain injury (TBI) can cause cognitive, communication, and psychological challenges that profoundly limit independence in everyday life. Conversational Agents (CAs) can provide individuals with TBI with cognitive and communication support, although little is known about how they make use of CAs to address injury-related needs. In this study, we gave nine adults with TBI an at-home CA for four weeks to investigate use patterns, challenges, and design requirements, focusing particularly on injury-related use. The findings revealed significant gaps between the current capabilities of CAs and accessibility challenges faced by TBI users. We also identified 14 TBI-related activities that participants engaged in with CAs. We categorized those activities into four groups: mental health, cognitive activities, healthcare and rehabilitation, and routine activities. Design implications focus on accessibility improvements and functional designs of CAs that can better support the day-to-day needs of people with TBI. △ Less

Submitted 10 October, 2023; v1 submitted 2 September, 2023; originally announced September 2023.

Comments: In Proceedings The 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS'23)

MSC Class: 68-06

Exploring the Design Space of Extra-Linguistic Expression for Robots

Authors: Amy Koike, Bilge Mutlu

Abstract: In this paper, we explore the new design space of extra-linguistic cues inspired by graphical tropes used in graphic novels and animation to enhance the expressiveness of social robots. To achieve this, we identified a set of cues that can be used to generate expressions, including smoke/steam/fog, water droplets, and bubbles. We prototyped devices that can generate these fluid expressions for a r… ▽ More In this paper, we explore the new design space of extra-linguistic cues inspired by graphical tropes used in graphic novels and animation to enhance the expressiveness of social robots. To achieve this, we identified a set of cues that can be used to generate expressions, including smoke/steam/fog, water droplets, and bubbles. We prototyped devices that can generate these fluid expressions for a robot and conducted design sessions where eight designers explored the use and utility of the cues in conveying the robot's internal states in various design scenarios. Our analysis of the 22 designs, the associated design justifications, and the interviews with designers revealed patterns in how each cue was used, how they were combined with nonverbal cues, and where the participants drew their inspiration from. These findings informed the design of an integrated module called EmoPack, which can be used to augment the expressive capabilities of any robot platform. △ Less

Submitted 27 June, 2023; originally announced June 2023.

Handheld Haptic Device with Coupled Bidirectional Input

Authors: Megh Vipul Doshi, Michael Hagenow, Robert Radwin, Michael Gleicher, Bilge Mutlu, Michael Zinn

Abstract: Handheld kinesthetic haptic interfaces can provide greater mobility and richer tactile information as compared to traditional grounded devices. In this paper, we introduce a new handheld haptic interface which takes input using bidirectional coupled finger flexion. We present the device design motivation and design details and experimentally evaluate its performance in terms of transparency and re… ▽ More Handheld kinesthetic haptic interfaces can provide greater mobility and richer tactile information as compared to traditional grounded devices. In this paper, we introduce a new handheld haptic interface which takes input using bidirectional coupled finger flexion. We present the device design motivation and design details and experimentally evaluate its performance in terms of transparency and rendering bandwidth using a handheld prototype device. In addition, we assess the device's functional performance through a user study comparing the proposed device to a commonly used grounded input device in a set of targeting and tracking tasks. △ Less

Submitted 30 May, 2023; originally announced May 2023.

Comments: 7 pages, 7 figures, World Haptics Conference 2023 Preprint

Periscope: A Robotic Camera System to Support Remote Physical Collaboration

Authors: Pragathi Praveena, Yeping Wang, Emmanuel Senft, Michael Gleicher, Bilge Mutlu

Abstract: We investigate how robotic camera systems can offer new capabilities to computer-supported cooperative work through the design, development, and evaluation of a prototype system called Periscope. With Periscope, a local worker completes manipulation tasks with guidance from a remote helper who observes the workspace through a camera mounted on a semi-autonomous robotic arm that is co-located with… ▽ More We investigate how robotic camera systems can offer new capabilities to computer-supported cooperative work through the design, development, and evaluation of a prototype system called Periscope. With Periscope, a local worker completes manipulation tasks with guidance from a remote helper who observes the workspace through a camera mounted on a semi-autonomous robotic arm that is co-located with the worker. Our key insight is that the helper, the worker, and the robot should all share responsibility of the camera view--an approach we call shared camera control. Using this approach, we present a set of modes that distribute the control of the camera between the human collaborators and the autonomous robot depending on task needs. We demonstrate the system's utility and the promise of shared camera control through a preliminary study where 12 dyads collaboratively worked on assembly tasks. Finally, we discuss design and research implications of our work for future robotic camera systems that facilitate remote collaboration. △ Less

Submitted 25 September, 2023; v1 submitted 12 May, 2023; originally announced May 2023.

Comments: This is a pre-print of the article accepted for publication in PACM HCI and will be presented at CSCW 2023

Knowing Who Knows What: Designing Socially Assistive Robots with Transactive Memory System

Authors: Yaxin Hu, Laura Stegner, Bilge Mutlu

Abstract: Transactive Memory System (TMS) is a group theory that describes how communication can enable the combination of individual minds into a group. While this theory has been extensively studied in human-human groups, it has not yet been formally applied to socially assistive robot design. We demonstrate how the three-phase TMS group communication process-which involves encoding, storage, and retrieva… ▽ More Transactive Memory System (TMS) is a group theory that describes how communication can enable the combination of individual minds into a group. While this theory has been extensively studied in human-human groups, it has not yet been formally applied to socially assistive robot design. We demonstrate how the three-phase TMS group communication process-which involves encoding, storage, and retrieval-can be leveraged to improve decision making in socially assistive robots with multiple stakeholders. By clearly defining how the robot is gaining information, storing and updating its memory, and retrieving information from its memory, we believe that socially assistive robots can make better decisions and provide more transparency behind their actions in the group context. Bringing communication theory to robot design can provide a clear framework to help robots integrate better into human-human group dynamics and thus improve their acceptance and use. △ Less

Submitted 8 May, 2023; originally announced May 2023.

Comments: Paper Accepted to Workshop "Socially Assistive Robots as Decision Makers: Transparency, Motivations, and Intentions" (SARs: TMI - 2023) at CHI 2023

Report number: SARTMI/2023/2

Family Theories in Child-Robot Interactions: Understanding Families as a Whole for Child-Robot Interaction Design

Authors: Bengisu Cagiltay, Bilge Mutlu, Margaret Kerr

Abstract: In this work, we discuss a theoretically motivated family-centered design approach for child-robot interactions, adapted by Family Systems Theory (FST) and Family Ecological Model (FEM). Long-term engagement and acceptance of robots in the home is influenced by factors that surround the child and the family, such as child-sibling-parent relationships and family routines, rituals, and values. A fam… ▽ More In this work, we discuss a theoretically motivated family-centered design approach for child-robot interactions, adapted by Family Systems Theory (FST) and Family Ecological Model (FEM). Long-term engagement and acceptance of robots in the home is influenced by factors that surround the child and the family, such as child-sibling-parent relationships and family routines, rituals, and values. A family-centered approach to interaction design is essential when developing in-home technology for children, especially for social agents like robots with which they can form connections and relationships. We review related literature in family theories and connect it with child-robot interaction and child-computer interaction research. We present two case studies that exemplify how family theories, FST and FEM, can inform the integration of robots into homes, particularly research into child-robot and family-robot interaction. Finally, we pose five overarching recommendations for a family-centered design approach in child-robot interactions. △ Less

Submitted 4 May, 2023; originally announced May 2023.

"My Unconditional Homework Buddy:'' Exploring Children's Preferences for a Homework Companion Robot

Authors: Bengisu Cagiltay, Bilge Mutlu, Joseph E Michaelis

Abstract: We aim to design robotic educational support systems that can promote socially and intellectually meaningful learning experiences for students while they complete school work outside of class. To pursue this goal, we conducted participatory design studies with 10 children (aged 10--12) to explore their design needs for robot-assisted homework. We investigated children's current ways of doing homew… ▽ More We aim to design robotic educational support systems that can promote socially and intellectually meaningful learning experiences for students while they complete school work outside of class. To pursue this goal, we conducted participatory design studies with 10 children (aged 10--12) to explore their design needs for robot-assisted homework. We investigated children's current ways of doing homework, the type of support they receive while doing homework, and co-designed the speech and expressiveness of a homework companion robot. Children and parents attending our design sessions explained that an emotionally expressive social robot as a homework aid can support students' motivation and engagement, as well as their affective state. Children primarily perceived the robot as a dedicated assistant at home, capable of forming meaningful friendships, or a shared classroom learning resource. We present key design recommendations to support students' homework experiences with a learning companion robot. △ Less

Submitted 4 May, 2023; originally announced May 2023.

Designing Parent-child-robot Interactions to Facilitate In-Home Parental Math Talk with Young Children

Authors: Hui-Ru Ho, Nathan White, Edward Hubbard, Bilge Mutlu

Abstract: Parent-child interaction is critical for child development, yet parents may need guidance in some aspects of their engagement with their children. Current research on educational math robots focuses on child-robot interactions but falls short of including the parents and integrating the critical role they play in children's learning. We explore how educational robots can be designed to facilitate… ▽ More Parent-child interaction is critical for child development, yet parents may need guidance in some aspects of their engagement with their children. Current research on educational math robots focuses on child-robot interactions but falls short of including the parents and integrating the critical role they play in children's learning. We explore how educational robots can be designed to facilitate parent-child conversations, focusing on math talk, a predictor of later math ability in children. We prototyped capabilities for a social robot to support math talk via reading and play activities and conducted an exploratory Wizard-of-Oz in-home study for parent-child interactions facilitated by a robot. Our findings yield insights into how parents were inspired by the robot's prompts, their desired interaction styles and methods for the robot, and how they wanted to include the robot in the activities, leading to guidelines for the design of parent-child-robot interaction in educational contexts. △ Less

Submitted 3 May, 2023; originally announced May 2023.

Comments: 15 pages, Accepted to IDC'23

Exploring the Use of Collaborative Robots in Cinematography

Authors: Pragathi Praveena, Bengisu Cagiltay, Michael Gleicher, Bilge Mutlu

Abstract: Robotic technology can support the creation of new tools that improve the creative process of cinematography. It is crucial to consider the specific requirements and perspectives of industry professionals when designing and developing these tools. In this paper, we present the results from exploratory interviews with three cinematography practitioners, which included a demonstration of a prototype… ▽ More Robotic technology can support the creation of new tools that improve the creative process of cinematography. It is crucial to consider the specific requirements and perspectives of industry professionals when designing and developing these tools. In this paper, we present the results from exploratory interviews with three cinematography practitioners, which included a demonstration of a prototype robotic system. We identified many factors that can impact the design, adoption, and use of robotic support for cinematography, including: (1) the ability to meet requirements for cost, quality, mobility, creativity, and reliability; (2) the compatibility and integration of tools with existing workflows, equipment, and software; and (3) the potential for new creative opportunities that robotic technology can open up. Our findings provide a starting point for future co-design projects that aim to support the work of cinematographers with collaborative robots. △ Less

Submitted 16 April, 2023; originally announced April 2023.

Comments: Article accepted at CHI LBW 2023. Pragathi Praveena and Bengisu Cagiltay contributed equally to the work

So, I Can Feel Normal: Participatory Design for Accessible Social Media Sites for Individuals with Traumatic Brain Injury

Authors: Hajin Lim, Lisa Kakonge, Yaxin Hu, Lyn S. Turkstra, Melissa C. Duff, Catalina L. Toma, Bilge Mutlu

Abstract: Traumatic brain injury (TBI) can result in chronic sensorimotor, cognitive, psychosocial, and communication challenges that can limit social participation. Social media can be a useful outlet for social participation for individuals with TBI, but there are barriers to access. While research has drawn attention to the nature of access barriers, few studies have investigated technological solutions… ▽ More Traumatic brain injury (TBI) can result in chronic sensorimotor, cognitive, psychosocial, and communication challenges that can limit social participation. Social media can be a useful outlet for social participation for individuals with TBI, but there are barriers to access. While research has drawn attention to the nature of access barriers, few studies have investigated technological solutions to address these barriers, particularly considering the perspectives of individuals with TBI. To address this gap in knowledge, we used a participatory approach to engage 10 adults with TBI in conceptualizing tools to address their challenges accessing Facebook. Participants described multifaceted challenges in using social media, including interface overload, social comparisons, and anxiety over self-presentation and communication after injury. They discussed their needs and preferences and generated ideas for design solutions. Our work contributes to designing assistive and accessibility technology to facilitate an equal access to the benefits of social media for individuals with TBI. △ Less

Submitted 4 April, 2023; originally announced April 2023.

Comments: Accepted at CHI '23, April 23-28, 2023, Hamburg, Germany

Coordinated Multi-Robot Shared Autonomy Based on Scheduling and Demonstrations

Authors: Michael Hagenow, Emmanuel Senft, Nitzan Orr, Robert Radwin, Michael Gleicher, Bilge Mutlu, Dylan P. Losey, Michael Zinn

Abstract: Shared autonomy methods, where a human operator and a robot arm work together, have enabled robots to complete a range of complex and highly variable tasks. Existing work primarily focuses on one human sharing autonomy with a single robot. By contrast, in this paper we present an approach for multi-robot shared autonomy that enables one operator to provide real-time corrections across two coordina… ▽ More Shared autonomy methods, where a human operator and a robot arm work together, have enabled robots to complete a range of complex and highly variable tasks. Existing work primarily focuses on one human sharing autonomy with a single robot. By contrast, in this paper we present an approach for multi-robot shared autonomy that enables one operator to provide real-time corrections across two coordinated robots completing the same task in parallel. Sharing autonomy with multiple robots presents fundamental challenges. The human can only correct one robot at a time, and without coordination, the human may be left idle for long periods of time. Accordingly, we develop an approach that aligns the robot's learned motions to best utilize the human's expertise. Our key idea is to leverage Learning from Demonstration (LfD) and time warping to schedule the motions of the robots based on when they may require assistance. Our method uses variability in operator demonstrations to identify the types of corrections an operator might apply during shared autonomy, leverages flexibility in how quickly the task was performed in demonstrations to aid in scheduling, and iteratively estimates the likelihood of when corrections may be needed to ensure that only one robot is completing an action requiring assistance. Through a preliminary study, we show that our method can decrease the scheduled time spent sanding by iteratively estimating the times when each robot could need assistance and generating an optimized schedule that allows the operator to provide corrections to each robot during these times. △ Less

Submitted 25 October, 2023; v1 submitted 28 March, 2023; originally announced March 2023.

Comments: IEEE Robotics and Automation Letters (RA-L)

Sketching Robot Programs On the Fly

Authors: David Porfirio, Laura Stegner, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, Bilge Mutlu

Abstract: Service robots for personal use in the home and the workplace require end-user development solutions for swiftly scripting robot tasks as the need arises. Many existing solutions preserve ease, efficiency, and convenience through simple programming interfaces or by restricting task complexity. Others facilitate meticulous task design but often do so at the expense of simplicity and efficiency. The… ▽ More Service robots for personal use in the home and the workplace require end-user development solutions for swiftly scripting robot tasks as the need arises. Many existing solutions preserve ease, efficiency, and convenience through simple programming interfaces or by restricting task complexity. Others facilitate meticulous task design but often do so at the expense of simplicity and efficiency. There is a need for robot programming solutions that reconcile the complexity of robotics with the on-the-fly goals of end-user development. In response to this need, we present a novel, multimodal, and on-the-fly development system, Tabula. Inspired by a formative design study with a prototype, Tabula leverages a combination of spoken language for specifying the core of a robot task and sketching for contextualizing the core. The result is that developers can script partial, sloppy versions of robot programs to be completed and refined by a program synthesizer. Lastly, we demonstrate our anticipated use cases of Tabula via a set of application scenarios. △ Less

Submitted 6 February, 2023; originally announced February 2023.

Comments: Accepted at HRI '23, March 13-16, 2023, Stockholm, Sweden

Situated Participatory Design: A Method for In Situ Design of Robotic Interaction with Older Adults

Authors: Laura Stegner, Emmanuel Senft, Bilge Mutlu

Abstract: We present a participatory design method to design human-robot interactions with older adults and its application through a case study of designing an assistive robot for a senior living facility. The method, called Situated Participatory Design (sPD), was designed considering the challenges of working with older adults and involves three phases that enable designing and testing use scenarios thro… ▽ More We present a participatory design method to design human-robot interactions with older adults and its application through a case study of designing an assistive robot for a senior living facility. The method, called Situated Participatory Design (sPD), was designed considering the challenges of working with older adults and involves three phases that enable designing and testing use scenarios through realistic, iterative interactions with the robot. In design sessions with nine residents and three caregivers, we uncovered a number of insights about sPD that help us understand its benefits and limitations. For example, we observed how designs evolved through iterative interactions and how early exposure to the robot helped participants consider using the robot in their daily life. With sPD, we aim to help future researchers to increase and deepen the participation of older adults in designing assistive technologies. △ Less

Submitted 21 February, 2023; v1 submitted 1 February, 2023; originally announced February 2023.

Comments: Accepted at CHI '23, April 23-28, 2023, Hamburg, Germany

Actor Prioritized Experience Replay

Authors: Baturay Saglam, Furkan B. Mutlu, Dogan C. Cicek, Suleyman S. Kozat

Abstract: A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical… ▽ More A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical studies indicate that it considerably underperforms actor-critic algorithms in continuous control. We theoretically show that actor networks cannot be effectively trained with transitions that have large TD errors. As a result, the approximate policy gradient computed under the Q-network diverges from the actual gradient computed under the optimal Q-function. Motivated by this, we introduce a novel experience replay sampling framework for actor-critic methods, which also regards issues with stability and recent findings behind the poor empirical performance of PER. The introduced algorithm suggests a new branch of improvements to PER and schedules effective and efficient training for both actor and critic networks. An extensive set of experiments verifies our theoretical claims and demonstrates that the introduced method significantly outperforms the competing approaches and obtains state-of-the-art results over the standard off-policy actor-critic algorithms. △ Less

Submitted 1 September, 2022; originally announced September 2022.

Comments: 21 pages, 5 figures, 4 tables

A Method For Automated Drone Viewpoints to Support Remote Robot Manipulation

Authors: Emmanuel Senft, Michael Hagenow, Pragathi Praveena, Robert Radwin, Michael Zinn, Michael Gleicher, Bilge Mutlu

Abstract: Drones can provide a minimally-constrained adapting camera view to support robot telemanipulation. Furthermore, the drone view can be automated to reduce the burden on the operator during teleoperation. However, existing approaches do not focus on two important aspects of using a drone as an automated view provider. The first is how the drone should select from a range of quality viewpoints within… ▽ More Drones can provide a minimally-constrained adapting camera view to support robot telemanipulation. Furthermore, the drone view can be automated to reduce the burden on the operator during teleoperation. However, existing approaches do not focus on two important aspects of using a drone as an automated view provider. The first is how the drone should select from a range of quality viewpoints within the workspace (e.g., opposite sides of an object). The second is how to compensate for unavoidable drone pose uncertainty in determining the viewpoint. In this paper, we provide a nonlinear optimization method that yields effective and adaptive drone viewpoints for telemanipulation with an articulated manipulator. Our first key idea is to use sparse human-in-the-loop input to toggle between multiple automatically-generated drone viewpoints. Our second key idea is to introduce optimization objectives that maintain a view of the manipulator while considering drone uncertainty and the impact on viewpoint occlusion and environment collisions. We provide an instantiation of our drone viewpoint method within a drone-manipulator remote teleoperation system. Finally, we provide an initial validation of our method in tasks where we complete common household and industrial manipulations. △ Less

Submitted 8 August, 2022; originally announced August 2022.

Comments: Article accepted at IROS 2022. Emmanuel Senft and Michael Hagenow contributed equally to the work, and consequently should share first authorship

Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach

Authors: Baturay Saglam, Dogan C. Cicek, Furkan B. Mutlu, Suleyman S. Kozat

Abstract: Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when the discrepancy between the underlying distributions of the agent's policy and collected data increases. Although the well-studied importance sampling and off-policy policy gradient… ▽ More Compared to on-policy counterparts, off-policy model-free deep reinforcement learning can improve data efficiency by repeatedly using the previously gathered data. However, off-policy learning becomes challenging when the discrepancy between the underlying distributions of the agent's policy and collected data increases. Although the well-studied importance sampling and off-policy policy gradient techniques were proposed to compensate for this discrepancy, they usually require a collection of long trajectories and induce additional problems such as vanishing/exploding gradients or discarding many useful experiences, which eventually increases the computational complexity. Moreover, their generalization to either continuous action domains or policies approximated by deterministic deep neural networks is strictly limited. To overcome these limitations, we introduce a novel policy similarity measure to mitigate the effects of such discrepancy in continuous control. Our method offers an adequate single-step off-policy correction that is applicable to deterministic policy networks. Theoretical and empirical studies demonstrate that it can achieve a "safe" off-policy learning and substantially improve the state-of-the-art by attaining higher returns in fewer steps than the competing methods through an effective schedule of the learning rate in Q-learning and policy optimization. △ Less

Submitted 25 September, 2023; v1 submitted 1 August, 2022; originally announced August 2022.

Safe and Robust Experience Sharing for Deterministic Policy Gradient Algorithms

Authors: Baturay Saglam, Dogan C. Cicek, Furkan B. Mutlu, Suleyman S. Kozat

Abstract: Learning in high dimensional continuous tasks is challenging, mainly when the experience replay memory is very limited. We introduce a simple yet effective experience sharing mechanism for deterministic policies in continuous action domains for the future off-policy deep reinforcement learning applications in which the allocated memory for the experience replay buffer is limited. To overcome the e… ▽ More Learning in high dimensional continuous tasks is challenging, mainly when the experience replay memory is very limited. We introduce a simple yet effective experience sharing mechanism for deterministic policies in continuous action domains for the future off-policy deep reinforcement learning applications in which the allocated memory for the experience replay buffer is limited. To overcome the extrapolation error induced by learning from other agents' experiences, we facilitate our algorithm with a novel off-policy correction technique without any action probability estimates. We test the effectiveness of our method in challenging OpenAI Gym continuous control tasks and conclude that it can achieve a safe experience sharing across multiple agents and exhibits a robust performance when the replay memory is strictly limited. △ Less

Submitted 27 July, 2022; originally announced July 2022.

Comments: ICML 2022 Workshop on Responsible Decision Making in Dynamic Environments (poster: http://responsibledecisionmaking.github.io/assets/poster/19.pdf , presentation: http://drive.google.com/file/d/1vjjMh_z51xdOjsQCcGfU5ojAcrrf3dOS/view?usp=sharing )

Exploring Children's Preferences for Taking Care of a Social Robot

Authors: Bengisu Cagiltay, Joseph Michaelis, Sarah Sebo, Bilge Mutlu

Abstract: Research in child-robot interactions suggests that engaging in "care-taking" of a social robot, such as tucking the robot in at night, can strengthen relationships formed between children and robots. In this work, we aim to better understand and explore the design space of caretaking activities with 10 children, aged 8--12 from eight families, involving an exploratory design session followed by a… ▽ More Research in child-robot interactions suggests that engaging in "care-taking" of a social robot, such as tucking the robot in at night, can strengthen relationships formed between children and robots. In this work, we aim to better understand and explore the design space of caretaking activities with 10 children, aged 8--12 from eight families, involving an exploratory design session followed by a preliminary feasibility testing of robot caretaking activities. The design sessions provided insight into children's current caretaking tasks, how they would take care of a social robot, and how these new caretaking activities could be integrated into their daily routines. The feasibility study tested two different types of robot caretaking tasks, which we call connection and utility, and measured their short term effects on children's perceptions of and closeness to the social robot. We discuss the themes and present interaction design guidelines of robot caretaking activities for children. △ Less

Submitted 18 May, 2022; originally announced May 2022.

Comments: Interaction Design and Children Conference, 2022

Journal ref: ACM Interaction Design and Children 2022 382-388

Understanding Factors that Shape Children's Long Term Engagement with an In-Home Learning Companion Robot

Authors: Bengisu Cagiltay, Nathan White, Rabia Ibtasar, Bilge Mutlu, Joseph Michaelis

Abstract: Social robots are emerging as learning companions for children, and research shows that they facilitate the development of interest and learning even through brief interactions. However, little is known about how such technologies might support these goals in authentic environments over long-term periods of use and interaction. We designed a learning companion robot capable of supporting children… ▽ More Social robots are emerging as learning companions for children, and research shows that they facilitate the development of interest and learning even through brief interactions. However, little is known about how such technologies might support these goals in authentic environments over long-term periods of use and interaction. We designed a learning companion robot capable of supporting children reading popular-science books by expressing social and informational commentaries. We deployed the robot in homes of 14 families with children aged 10-12 for four weeks during the summer. Our analysis revealed critical factors that affected children's long-term engagement and adoption of the robot, including external factors such as vacations, family visits, and extracurricular activities; family/parental involvement; and children's individual interests. We present four in-depth cases that illustrate these factors and demonstrate their impact on children's reading experiences and discuss the implications of our findings for robot design. △ Less

Submitted 18 May, 2022; originally announced May 2022.

Comments: Interaction Design and Children Conference, 2022

Journal ref: ACM Interaction Design and Children 2022 362-373

Designing for Caregiving: Integrating Robotic Assistance in Senior Living Communities

Authors: Laura Stegner, Bilge Mutlu

Abstract: Robots hold significant promise to assist with providing care to an aging population and to help overcome increasing caregiver demands. Although a large body of research has explored robotic assistance for individuals with disabilities and age-related challenges, this past work focuses primarily on building robotic capabilities for assistance and has not yet fully considered how these capabilities… ▽ More Robots hold significant promise to assist with providing care to an aging population and to help overcome increasing caregiver demands. Although a large body of research has explored robotic assistance for individuals with disabilities and age-related challenges, this past work focuses primarily on building robotic capabilities for assistance and has not yet fully considered how these capabilities could be used by professional caregivers. To better understand the workflows and practices of caregivers who support aging populations and to determine how robotic assistance can be integrated into their work, we conducted a field study using ethnographic and co-design methods in a senior living community. From our results, we created a set of design opportunities for robotic assistance, which we organized into three different parts: supporting caregiver workflows, adapting to resident abilities, and providing feedback to all stakeholders of the interaction. △ Less

Submitted 18 May, 2022; originally announced May 2022.

Comments: 14 pages, 9 figures

Journal ref: Designing Interactive Systems Conference (DIS '22), June 13-17, 2022, Virtual Event, Australia

Manually Acquiring Targets from Multiple Viewpoints Using Video Feedback

Authors: Bailey Ramesh, Anna Konstant, Pragathi Praveena, Emmanuel Senft, Michael Gleicher, Bilge Mutlu, Michael Zinn, Robert G. Radwin

Abstract: Objective: The effect of camera viewpoint was studied when performing visually obstructed psychomotor targeting tasks. Background: Previous research in laparoscopy and robotic teleoperation found that complex perceptual-motor adaptations associated with misaligned viewpoints corresponded to degraded performance in manipulation. Because optimal camera positioning is often unavailable in restricted… ▽ More Objective: The effect of camera viewpoint was studied when performing visually obstructed psychomotor targeting tasks. Background: Previous research in laparoscopy and robotic teleoperation found that complex perceptual-motor adaptations associated with misaligned viewpoints corresponded to degraded performance in manipulation. Because optimal camera positioning is often unavailable in restricted environments, alternative viewpoints that might mitigate performance effects are not obvious. Methods: A virtual keyboard-controlled targeting task was remotely distributed to workers of Amazon Mechanical Turk. The experiment was performed by 192 subjects for a static viewpoint with independent parameters of target direction, Fitts' law index of difficulty, viewpoint azimuthal angle (AA), and viewpoint polar angle (PA). A dynamic viewpoint experiment was also performed by 112 subjects in which the viewpoint AA changed after every trial. Results: AA and target direction had significant effects on performance for the static viewpoint experiment. Movement time and travel distance increased while AA increased until there was a discrete improvement in performance for 180°. Increasing AA from 225° to 315° linearly decreased movement time and distance. There were significant main effects of current AA and magnitude of transition for the dynamic viewpoint experiment. Orthogonal direction and no-change viewpoint transitions least affected performance. Conclusions: Viewpoint selection should aim to minimize associated rotations within the manipulation plane when performing targeting tasks whether implementing a static or dynamic viewing solution. Because PA rotations had negligible performance effects, PA adjustments may extend the space of viable viewpoints. Applications: These results can inform viewpoint-selection for visual feedback during psychomotor tasks. △ Less

Submitted 15 April, 2022; v1 submitted 14 April, 2022; originally announced April 2022.

Comments: Accepted for publication in Human Factors

Polite or Direct? Conversation Design of a Smart Display for Older Adults Based on Politeness Theory

Authors: Yaxin Hu, Yuxiao Qu, Adam Maus, Bilge Mutlu

Abstract: Conversational interfaces increasingly rely on human-like dialogue to offer a natural experience. However, relying on dialogue involving multiple exchanges for even simple tasks can overburden users, particularly older adults. In this paper, we explored the use of politeness theory in conversation design to alleviate this burden and improve user experience. To achieve this goal, we categorized the… ▽ More Conversational interfaces increasingly rely on human-like dialogue to offer a natural experience. However, relying on dialogue involving multiple exchanges for even simple tasks can overburden users, particularly older adults. In this paper, we explored the use of politeness theory in conversation design to alleviate this burden and improve user experience. To achieve this goal, we categorized the voice interaction offered by a smart display application designed for older adults into seven major speech acts: request, suggest, instruct, comment, welcome, farewell, and repair. We identified face needs for each speech act, applied politeness strategies that best address these needs, and tested the ability of these strategies to shape the perceived politeness of a voice assistant in an online study ($n=64$). Based on the findings of this study, we designed direct and polite versions of the system and conducted a field study ($n=15$) in which participants used each of the versions for five days at their homes. Based on five factors merged from our qualitative findings, we identified four distinctive user personas$\unicode{x2013}$socially oriented follower, socially oriented leader, utility oriented follower, and utility oriented leader$\unicode{x2013}$that can inform personalized design of smart displays. △ Less

Submitted 29 March, 2022; originally announced March 2022.

Comments: To be published in 2022 CHI Conference on Human Factors in Computing Systems (CHI '22)

Registering Articulated Objects With Human-in-the-loop Corrections

Authors: Michael Hagenow, Emmanuel Senft, Evan Laske, Kimberly Hambuchen, Terrence Fong, Robert Radwin, Michael Gleicher, Bilge Mutlu, Michael Zinn

Abstract: Remotely programming robots to execute tasks often relies on registering objects of interest in the robot's environment. Frequently, these tasks involve articulating objects such as opening or closing a valve. However, existing human-in-the-loop methods for registering objects do not consider articulations and the corresponding impact to the geometry of the object, which can cause the methods to f… ▽ More Remotely programming robots to execute tasks often relies on registering objects of interest in the robot's environment. Frequently, these tasks involve articulating objects such as opening or closing a valve. However, existing human-in-the-loop methods for registering objects do not consider articulations and the corresponding impact to the geometry of the object, which can cause the methods to fail. In this work, we present an approach where the registration system attempts to automatically determine the object model, pose, and articulation for user-selected points using nonlinear fitting and the iterative closest point algorithm. When the fitting is incorrect, the operator can iteratively intervene with corrections after which the system will refit the object. We present an implementation of our fitting procedure for one degree-of-freedom (DOF) objects with revolute joints and evaluate it with a user study that shows that it can improve user performance, in measures of time on task and task load, ease of use, and usefulness compared to a manual registration approach. We also present a situated example that integrates our method into an end-to-end system for articulating a remote valve. △ Less

Submitted 16 August, 2022; v1 submitted 11 March, 2022; originally announced March 2022.

Comments: Accepted as a contributed paper at the 2022 International Conference on Intelligent Robots and Systems (IROS)

The Unboxing Experience: Exploration and Design of Initial Interactions Between Children and Social Robots

Authors: Christine P Lee, Bengisu Cagiltay, Bilge Mutlu

Abstract: Social robots are increasingly introduced into children's lives as educational and social companions, yet little is known about how these products might best be introduced to their environments. The emergence of the "unboxing" phenomenon in media suggests that introduction is key to technology adoption where initial impressions are made. To better understand this phenomenon toward designing a posi… ▽ More Social robots are increasingly introduced into children's lives as educational and social companions, yet little is known about how these products might best be introduced to their environments. The emergence of the "unboxing" phenomenon in media suggests that introduction is key to technology adoption where initial impressions are made. To better understand this phenomenon toward designing a positive unboxing experience in the context of social robots for children, we conducted three field studies with families of children aged 8 to 13: (1) an exploratory free-play activity ($n=12$); (2) a co-design session ($n=11$) that informed the development of a prototype box and a curated unboxing experience; and (3) a user study ($n=9$) that evaluated children's experiences. Our findings suggest the unboxing experience of social robots can be improved through the design of a creative aesthetic experience that engages the child socially to guide initial interactions and foster a positive child-robot relationship. △ Less

Submitted 16 February, 2022; originally announced February 2022.

Comments: To be published in 2022 CHI Conference on Human Factors in Computing Systems (CHI '22)

Journal ref: CHI Conference on Human Factors in Computing Systems (CHI '22), April 29-May 5, 2022, New Orleans, LA, USA

CONFIDANT: A Privacy Controller for Social Robots

Authors: Brian Tang, Dakota Sullivan, Bengisu Cagiltay, Varun Chandrasekaran, Kassem Fawaz, Bilge Mutlu

Abstract: As social robots become increasingly prevalent in day-to-day environments, they will participate in conversations and appropriately manage the information shared with them. However, little is known about how robots might appropriately discern the sensitivity of information, which has major implications for human-robot trust. As a first step to address a part of this issue, we designed a privacy co… ▽ More As social robots become increasingly prevalent in day-to-day environments, they will participate in conversations and appropriately manage the information shared with them. However, little is known about how robots might appropriately discern the sensitivity of information, which has major implications for human-robot trust. As a first step to address a part of this issue, we designed a privacy controller, CONFIDANT, for conversational social robots, capable of using contextual metadata (e.g., sentiment, relationships, topic) from conversations to model privacy boundaries. Afterwards, we conducted two crowdsourced user studies. The first study (n=174) focused on whether a variety of human-human interaction scenarios were perceived as either private/sensitive or non-private/non-sensitive. The findings from our first study were used to generate association rules. Our second study (n=95) evaluated the effectiveness and accuracy of the privacy controller in human-robot interaction scenarios by comparing a robot that used our privacy controller against a baseline robot with no privacy controls. Our results demonstrate that the robot with the privacy controller outperforms the robot without the privacy controller in privacy-awareness, trustworthiness, and social-awareness. We conclude that the integration of privacy controllers in authentic human-robot conversations can allow for more trustworthy robots. This initial privacy controller will serve as a foundation for more complex solutions. △ Less

Submitted 7 January, 2022; originally announced January 2022.

Comments: HRI 2022

AWD3: Dynamic Reduction of the Estimation Bias

Authors: Dogan C. Cicek, Enes Duran, Baturay Saglam, Kagan Kaya, Furkan B. Mutlu, Suleyman S. Kozat

Abstract: Value-based deep Reinforcement Learning (RL) algorithms suffer from the estimation bias primarily caused by function approximation and temporal difference (TD) learning. This problem induces faulty state-action value estimates and therefore harms the performance and robustness of the learning algorithms. Although several techniques were proposed to tackle, learning algorithms still suffer from thi… ▽ More Value-based deep Reinforcement Learning (RL) algorithms suffer from the estimation bias primarily caused by function approximation and temporal difference (TD) learning. This problem induces faulty state-action value estimates and therefore harms the performance and robustness of the learning algorithms. Although several techniques were proposed to tackle, learning algorithms still suffer from this bias. Here, we introduce a technique that eliminates the estimation bias in off-policy continuous control algorithms using the experience replay mechanism. We adaptively learn the weighting hyper-parameter beta in the Weighted Twin Delayed Deep Deterministic Policy Gradient algorithm. Our method is named Adaptive-WD3 (AWD3). We show through continuous control environments of OpenAI gym that our algorithm matches or outperforms the state-of-the-art off-policy policy gradient learning algorithms. △ Less

Submitted 12 November, 2021; originally announced November 2021.

Comments: Accepted at The 33rd IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2021)

Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replay

Authors: Dogan C. Cicek, Enes Duran, Baturay Saglam, Furkan B. Mutlu, Suleyman S. Kozat

Abstract: The experience replay mechanism allows agents to use the experiences multiple times. In prior works, the sampling probability of the transitions was adjusted according to their importance. Reassigning sampling probabilities for every transition in the replay buffer after each iteration is highly inefficient. Therefore, experience replay prioritization algorithms recalculate the significance of a t… ▽ More The experience replay mechanism allows agents to use the experiences multiple times. In prior works, the sampling probability of the transitions was adjusted according to their importance. Reassigning sampling probabilities for every transition in the replay buffer after each iteration is highly inefficient. Therefore, experience replay prioritization algorithms recalculate the significance of a transition when the corresponding transition is sampled to gain computational efficiency. However, the importance level of the transitions changes dynamically as the policy and the value function of the agent are updated. In addition, experience replay stores the transitions are generated by the previous policies of the agent that may significantly deviate from the most recent policy of the agent. Higher deviation from the most recent policy of the agent leads to more off-policy updates, which is detrimental for the agent. In this paper, we develop a novel algorithm, Batch Prioritizing Experience Replay via KL Divergence (KLPER), which prioritizes batch of transitions rather than directly prioritizing each transition. Moreover, to reduce the off-policyness of the updates, our algorithm selects one batch among a certain number of batches and forces the agent to learn through the batch that is most likely generated by the most recent policy of the agent. We combine our algorithm with Deep Deterministic Policy Gradient and Twin Delayed Deep Deterministic Policy Gradient and evaluate it on various continuous control tasks. KLPER provides promising improvements for deep deterministic continuous control algorithms in terms of sample efficiency, final performance, and stability of the policy during the training. △ Less

Submitted 12 November, 2021; v1 submitted 2 November, 2021; originally announced November 2021.

Comments: Accepted at The 33rd IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2021)

Appearance

Authors: Rachel McDonnell, Bilge Mutlu

Abstract: Socially interactive agents (SIAs) are no longer mere visions for future user interfaces, as 20 years of research and technology development has enabled the use of virtual and physical agents in day-to-day interfaces and environments. This chapter of the ACM "The Handbook on Socially Interactive Agents" reviews research on and technologies involving socially interactive agents, including virtually… ▽ More Socially interactive agents (SIAs) are no longer mere visions for future user interfaces, as 20 years of research and technology development has enabled the use of virtual and physical agents in day-to-day interfaces and environments. This chapter of the ACM "The Handbook on Socially Interactive Agents" reviews research on and technologies involving socially interactive agents, including virtually embodied agents and physically embodied robots, focusing particularly on the appearance of socially interactive agents. It covers the history of the development of these technologies; outlines the design space for the appearance of agents, including what appearance comprises, modalities in which agents are presented, and how agents are constructed; and the features that agents use to support social interaction, including facial and bodily features, those that express demographic characteristics, and issues surrounding realism, appeal, and the uncanny valley. The chapter concludes with a brief discussion of open questions surrounding the appearance of socially interactive agents. △ Less

Submitted 7 October, 2021; originally announced October 2021.

Comments: 38 pages, 16 figures, appears in the "The Handbook on Socially Interactive Agents: 20 Years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social Robotics Volume 1: Methods, Behavior, Cognition," published by the ACM

Journal ref: In B. Lugrin, C. Pelachaud, D. Traum (Eds.), The Handbook on Socially Interactive Agents, Volume 1 (pp. 107-146). ACM books (2021)

Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients

Authors: Baturay Saglam, Furkan Burak Mutlu, Dogan Can Cicek, Suleyman Serdar Kozat

Abstract: Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a high variance, deep actor-critic approaches that overcome the overestimation bias lead to a substantial underestimation bias. We first address the detrimental issues in the existing a… ▽ More Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a high variance, deep actor-critic approaches that overcome the overestimation bias lead to a substantial underestimation bias. We first address the detrimental issues in the existing approaches that aim to overcome such underestimation error. Then, through extensive statistical analysis, we introduce a novel, parameter-free Deep Q-learning variant to reduce this underestimation bias in deterministic policy gradients. By sampling the weights of a linear combination of two approximate critics from a highly shrunk estimation bias interval, our Q-value update rule is not affected by the variance of the rewards received by the agents throughout learning. We test the performance of the introduced improvement on a set of MuJoCo and Box2D continuous control tasks and demonstrate that it considerably outperforms the existing approaches and improves the state-of-the-art by a significant margin. △ Less

Submitted 19 May, 2022; v1 submitted 24 September, 2021; originally announced September 2021.

Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods

Authors: Baturay Saglam, Enes Duran, Dogan C. Cicek, Furkan B. Mutlu, Suleyman S. Kozat

Abstract: In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overestimation bias, if the reinforcement signals received by the agent have a high variance, a significant underestimation bias arises. To minimize the underestimation, we introduce a p… ▽ More In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overestimation bias, if the reinforcement signals received by the agent have a high variance, a significant underestimation bias arises. To minimize the underestimation, we introduce a parameter-free, novel deep Q-learning variant. Our Q-value update rule combines the notions behind Clipped Double Q-learning and Maxmin Q-learning by computing the critic objective through the nested combination of maximum and minimum operators to bound the approximate value estimates. We evaluate our modification on the suite of several OpenAI Gym continuous control tasks, improving the state-of-the-art in every environment tested. △ Less

Submitted 23 September, 2021; v1 submitted 22 September, 2021; originally announced September 2021.

Task-Level Authoring for Remote Robot Teleoperation

Authors: Emmanuel Senft, Michael Hagenow, Kevin Welsh, Robert Radwin, Michael Zinn, Michael Gleicher, Bilge Mutlu

Abstract: Remote teleoperation of robots can broaden the reach of domain specialists across a wide range of industries such as home maintenance, health care, light manufacturing, and construction. However, current direct control methods are impractical, and existing tools for programming robot remotely have focused on users with significant robotic experience. Extending robot remote programming to end users… ▽ More Remote teleoperation of robots can broaden the reach of domain specialists across a wide range of industries such as home maintenance, health care, light manufacturing, and construction. However, current direct control methods are impractical, and existing tools for programming robot remotely have focused on users with significant robotic experience. Extending robot remote programming to end users, i.e., users who are experts in a domain but novices in robotics, requires tools that balance the rich features necessary for complex teleoperation tasks with ease of use. The primary challenge to usability is that novice users are unable to specify complete and robust task plans to allow a robot to perform duties autonomously, particularly in highly variable environments. Our solution is to allow operators to specify shorter sequences of high-level commands, which we call task-level authoring, to create periods of variable robot autonomy. This approach allows inexperienced users to create robot behaviors in uncertain environments by interleaving exploration, specification of behaviors, and execution as separate steps. End users are able to break down the specification of tasks and adapt to the current needs of the interaction and environments, combining the reactivity of direct control to asynchronous operation. In this paper, we describe a prototype system contextualized in light manufacturing and its empirical validation in a user study where 18 participants with some programming experience were able to perform a variety of complex telemanipulation tasks with little training. Our results show that our approach allowed users to create flexible periods of autonomy and solve rich manipulation tasks. Furthermore, participants significantly preferred our system over comparative more direct interfaces, demonstrating the potential of our approach. △ Less

Submitted 6 September, 2021; originally announced September 2021.

Comments: Provisionally accepted in Frontiers in Robotics and AI on the 09/03/2021. Abstract has been shortened due to arxiv abstract requirements

Recognizing Orientation Slip in Human Demonstrations

Authors: Michael Hagenow, Bolun Zhang, Bilge Mutlu, Michael Gleicher, Michael Zinn

Abstract: Manipulations of a constrained object often use a non-rigid grasp that allows the object to rotate relative to the end effector. This orientation slip strategy is often present in natural human demonstrations, yet it is generally overlooked in methods to identify constraints from such demonstrations. In this paper, we present a method to model and recognize prehensile orientation slip in human dem… ▽ More Manipulations of a constrained object often use a non-rigid grasp that allows the object to rotate relative to the end effector. This orientation slip strategy is often present in natural human demonstrations, yet it is generally overlooked in methods to identify constraints from such demonstrations. In this paper, we present a method to model and recognize prehensile orientation slip in human demonstrations of constrained interactions. Using only observations of an end effector, we can detect the type of constraint, parameters of the constraint, and orientation slip properties. Our method uses a novel hierarchical model selection method that is informed by multiple origins of physics-based evidence. A study with eight participants shows that orientation slip occurs in natural demonstrations and confirms that it can be detected by our method. △ Less

Submitted 10 August, 2021; originally announced August 2021.

Comments: 2021 International Conference on Robotics and Automation (ICRA 2021)