From Agent Autonomy to Casual Collaboration: A Design Investigation on Help-Seeking Urban Robots

With the advancements in artificial intelligence, interactive products and systems have transitioned from performing programmed tasks under human supervision, to achieving higher level of autonomy, emphasising self-governance, adaptability, and collaborative interactions with humans [19, 64, 98]. These artefacts – commonly referred to as agents – include smart devices, robots, virtual agents, and voice-activated personal assistants. To support the shift towards more efficient human-agent collaboration [7], the evolving dynamics between humans and agents have become a topic of enduring interest across different HCI communities [10, 18, 49, 66]. Often inspired by theories from the social sciences and drawing on human-human interaction and behaviour studies, researchers developed frameworks to inform the design of interfaces and agent behaviour for human-agent teamwork settings. For example, Johnson et al. [48] introduced the coactive design approach, which is centred on the idea of mutual interdependence, underlining the essential principles of mutual observability, predictability, and directability for effective collaboration between humans and agents. Cila [18] drew insights from the Shared Cooperative Activity (SCA) framework, a model of human-human collaboration introduced by Bratman [11]. By reviewing its core tenet, the study underscored the importance of mutual support and pointed towards the need to investigate effective means for agents to request help during collaborations.

In human-robot collaboration (HRC) settings, research has explored various strategies to equip robots with the capability to seek assistance from human collaborators to complete a joint task. These methods include verbal cues [13, 52, 87] and non-verbal signals such as movement [56], light, and sound [15]. Due to the shared objectives and mutual understanding between humans and robots in these human-robot teaming contexts, such methods often enable efficient communication and prompt assistive behaviours from human collaborators.

In contrast to human-robot team settings where both parties share a mutual goal and have knowledge of the task, the dynamics of help-seeking become more complicated when robots interact with unassociated individuals such as casual bystanders. Contextual factors like specific task scenarios and the bystander’s current activity [46], combined with individual factors, such as the bystander’s trust towards and perceived competence of robots [14], collectively shape assistive behaviour. Despite the misaligned task objectives and the added complexities of various contextual factors, there is a noticeable absence of tailored design strategies or investigations for robot help-seeking from bystanders. Both academic research [50, 60, 75, 97] and commercially deployed robots [9] predominantly resort to verbal help-seeking strategies from bystanders in public environments. While validated to be efficient in human-robot teaming scenarios, their effectiveness in casual collaborations between robots and bystanders in public urban environments may be compromised by factors like cultural and linguistic differences, cognitive overload, and ambient noise and distortion. To our knowledge, the only study that expanded exploration on communication modalities is  [45]. They investigated the interplay of movement with auditory cues such as beeps and synthesised speech. Notably, their findings suggest that non-verbal expressions may elicit higher empathy from bystanders compared to verbal requests.

Transcending initial static and semi-controlled configurations (e.g., laboratories [78], factories [40], domestic environments [16]), robots have expanded their presence to public urban spaces, reshaping our cities. Urban robots serve various sectors, including transportation, infrastructure maintenance, cleaning, and surveillance [79].

Despite technological advancements, questions remain about how well these robots can operate in urban environments that are populated by and designed for humans [72]. Given the diverse infrastructure and unpredictability of urban settings, it’s challenging to fully anticipate the feasibility of different operational scenarios for robots during their development process. A vivid demonstration of this challenge emerged from several viral videos on social media in 2021, depicting delivery robots struggling in Estonia’s heavy snowfall [70]. Beyond the evident operational difficulties, a fascinating aspect of these incidents was the spontaneous assistance offered by passersby to these commercially deployed machines to help them resume moving [28]. The prosocial behaviour from bystanders was also observed in a recent field observation study [93], where pedestrians voluntarily assisted immobilised robots by removing obstacles. These observations underscore the potential of leveraging bystander assistance to enhance the operation of urban robots. This aligns not only with exploratory projects involving human-dependent robots [51, 85, 94] but also with emerging viewpoints in human-agent interaction that emphasise re-envisioning robot design through a relational lens when addressing operational challenges [61].

In summary, as agents become increasingly prevalent in urban environments, their operational challenges underscore the importance of eliciting assistance from bystanders. However, the misaligned task objectives and various contextual factors present a gap in effective strategies to facilitate such casual collaborations. Responding to Cila’s call [18] for envisioning effective ways to foster human assistance, our research spotlights urban scenarios where robots get stuck as an example, exploring effective non-verbal strategies for them to seek help during operational difficulties.

A notable challenge with embodied design methods in human-robot interaction research is their speculative nature, which can sometimes distance them from practical real-world scenarios. Acknowledging the importance of grounding these speculative methods in tangible realities, our methodology fuses speculative embodied methods with real-world scenarios. Our bodystorming activities are contextualised in real-world scenarios in which urban robots might encounter operational difficulties. These scenarios were drawn from a comprehensive online ethnography study we previously conducted. In this study, we analysed 177 user-generated videos that captured road users’ casual encounters with delivery robots on TikTok ¹. From this analysis, we identified three typical scenarios in which an urban robot may face operational difficulties: (1) The robot is stuck and requires assistance to be pushed out. (2) The robot is unable to cross the road and needs someone to press the traffic light button for it. (3) The robot is blocked and requires people to clear a path for it.

We utilised simple markers and physical props to replicate these scenarios. For example, we used masking tape to delineate the divisions between driveways and sidewalks, as well as to indicate zebra crossings (see Fig. 1, middle).

Low-tech prostheses and props have been shown to facilitate perspective shifting and stimulate imagination in human-robot interaction bodystorming [29]. Guided by these insights, our robot costume design sought to emulate the appearance and constraints of a box-shaped delivery robot. We narrowed the communication and interaction modalities of the robot player to exploring the help-requesting interactions of robots in abstract forms, in a minimal anthropomorphic manner. In addition, the design sought to exclude the potential effects of interpersonal communication when two participants can see each other.

The robot costume was made out of an 80 cm × 60 cm × 60 cm cardboard box (see Fig. 1, left). The bottom was removed and replaced with a dolly, allowing participants to sit inside the box and move freely. In addition, the four walls and top surface of the cardboard structure were supplanted with one-way, see-through reflective film. This modification endowed the robot player with the ability to observe the external environment, while concurrently shielding the interior from outside view, thereby inhibiting any possibility of eye contact with external observers. The robot player was also provided with an adjustable stick pointer that they could hold and reach out from the top of the box, to imitate the flagpole featured on commercial delivery robots.

We conducted 4 focus group sessions with 17 expert participants (8 males, 9 females; aged between 18-44): the first three sessions each included 4 participants, while the final session accommodated 5. These participants came from diverse academic or professional domains related to urban robots [90], including four PhD students in human-computer interaction and human-robot interaction, three PhD students in urbanism, two postdoctoral researchers in robotics, three interaction designers, and five postgraduate students specialising in interaction design. The selection of participants enhances the discourse by integrating their specialised expertise, while simultaneously bringing their lived experience as pedestrians into the role-play activity. Participants were recruited from our university’s mailing lists, flyers, and social networking platforms, following the study protocol approved by our university’s human research ethics committee.

The focus groups were audio and video recorded, and observation notes were taken both during the sessions and afterward when analysing the session videos. We transcribed the interviews and made detailed observation notes based on video recordings captured during the bodystorming activity. We conducted a thematic analysis  [12] on both the interview data and the observation notes. This cross-analysis approach enabled us to gain deeper insights into specific observations and enriched the contextual data that supported the comments made during the interviews.

The first author examined the data from the first focus group session, thus generating preliminary codes and themes. This was followed by a one-hour coding meeting amongst the three authors to deliberate upon this initial coding scheme. The coding scheme was refined based on the collective feedback and applied by the first author to code the data derived from the second and third sessions. During this process, flexibility was maintained for the generation of new codes, allowing for their integration into either a new theme or an existing coding scheme. Subsequently, another coding meeting was convened amongst the authors for further iteration of the coding scheme. Upon reaching a collective agreement, the first author coded the data from the fourth session and refined the coding from the previously analysed three sessions. This iterative and collaborative process ensured a holistic analysis of the data, leading to more concrete conclusions drawn from the focus groups.

Based on the results of the word sorting activity and insights derived from the participants’ discussions, we identified three characteristics that help-seeking robots could implement, which we present in this section.

Given humans’ innate psychological tendency to interpret social cues from moving objects [42], physical movement has become a pivotal medium in human-agent interaction to facilitate social communication  [3, 33, 62, 99, 101]. An example of extreme abstraction with minimal movement serving as social cues is the ‘Greeting Machine’ [3] – a small ball rolling on a bigger dome in varied trajectories. This design effectively elicits both positive and negative social encounter responses. Similar to ‘Greeting Machine’, participants in our design investigation were constrained by a robotic costume made of a box and a pole that had only minimum expressive capabilities. This costume had no extensions beyond the basic form factor of a conventional delivery robot that is primarily intended for the task of transporting goods. Nonetheless, subtle cues, such as the robot’s orientation in its shape (i.e. orienting the front of the box costume toward an object) or the directionality of simple components (i.e., pointing in the intended direction using the pole), proved effective in addressing bystanders and conveying the robot’s need for assistance. This was evident, as in all the sessions, pedestrian players accurately understood the robot player’s request for assistance. In addition, we even witnessed conversations formulated through the back-and-forth interplay between pedestrian inquiries and the subtle motions of robot players (as shown in Fig. 4, left)

In addition to empathising with the robot’s feelings of frustration and vulnerability when seeking help, it was evident that people also recognise social signals gestured through subtle movements, such as gratitude. For instance, a simple tilt of the box-shaped robot body, can be perceived as a ‘bow’, even prompting the pedestrian player P15 to bow back (as shown in Fig. 4, right).

As pointed out in our literature review, linguistic utterances have been central in human-agent collaboration [82] and represent a key method for help-seeking requests [5, 46, 87] given their effectiveness in conveying information. Nonetheless, challenges such as cultural and language barriers, cognitive load, and issues related to noise and distortion in public settings limit their utility in the context of casual human-agent collaboration in public spaces. In addition to that, our study revealed further concerns regarding the appropriateness of agents verbally asking for help in urban public contexts (n=7). P1, for instance, suggested it could be a ‘bit abrupt or out of place’ if a robot suddenly started to talk human language on the street, underscoring the need for robots to have their own unique, natural communication modes. Additionally, safety concerns were expressed by three participants who suggested that language used by robots might potentially distract other road users. Notably, P14 expressed potential discomfort in feeling exploited by commercial entities when robots use human language to solicit assistance, suggesting a ‘feeling of being used as free labour for those commercial companies’.

Robots and intelligent agents, growing rapidly in sophistication and sociability, have spurred enhanced research into agent social identity [43], delving into aspects like gender [31], age [30], and race [6]. This trajectory was echoed at a recent workshop [96], which emphasised the importance of designing robots that can effectively convey social identities to optimise human-robot interaction outcomes. While this workshop primarily centred on social identities associated with specific attributes (i.e. gender), our findings broaden this scope to encompass wider social categories (e.g. occupations), which should be considered in the design of casual collaborations between agents and bystanders.

In our investigation of the help-seeking delivery robot, participants perceived these robots primarily as service providers or professional workers. This perception led them to expect high proficiency from these robots, favoring the robot presenting qualities reminiscent of confidence over neediness. Consequently, even when assistance was necessary, participants displayed a general reluctance to interact with robots that seemed overly needy or showcased negative emotions. This inclination stands in contrast with studies on eliciting human prosocial behaviour towards social companion robots [20, 26] (e.g., robotic pets). In these settings, negative emotional expressions in robots often serve as a catalyst, motivating individuals to step in and offer help. This difference could be rooted in the distinct perceived categories: ‘worker’ versus ‘companion’. In public urban settings, intelligent agents participate integrally in various facets of urban life. As a result, they embody a wide range of social categories, from service providers (e.g., delivery robots [32], street cleaning robots [47]) and authoritative entities (e.g., smart traffic regulators [58], patrol robots [88]) to street entertainers (e.g., playful urban robots [44, 59]).

Though in 7 out of 9 sessions, pedestrian participants offered help, 4 of them mentioned they might not behave the same way in real-life settings. Furthermore, the misaligned objectives and the imbalanced benefits between agents and bystanders in such casual collaborations highlight the need for incentives.

Previous research on bystander assistance for commercially-deployed robots has divided ‘help’ into two broader categories: either ‘helping-as-work’, emphasising precarity and invisible labour, or ‘helping-as-care’, which accentuates the emotional and relational dynamics of help [27]. This work sheds light on the inherent ambiguity of these helping behaviours and prompts a rethinking of robot design to better shape these engagements. Our focus group findings resonate with this notion, while also shedding light on how different perspectives on helping robots call for various forms of incentives.

Offering material rewards, such as vouchers or discounts from companies deploying robots, turns casual collaborations into mutually beneficial exchanges. This model of paid crowdsourcing, evident in cases like identifying shared bicycle locations [38] or urban data collection on platforms like OpenStreetMap [24], could be adapted for interactions with intelligent agents in public spaces. By framing casual collaborations as beneficial exchanges through material rewards, it can effectively align disparate objectives between parties into actions that are mutually advantageous.

In our study, empathy – as an act of care – emerged as a predominant motivator for offering assistance, manifested as a form of internal incentive. Empathy, essential in shaping communication and social bonds, has been underscored as a central component in human-agent interactions [8]. A robust body of research validates robots’ capacity to elicit empathetic responses from humans [55, 74, 77]. Correspondingly, studies on social robots have shown that these forms of empathy can drive prosocial behaviour, compelling humans to intervene against robot mistreatment [21] or engage in affectionate actions, like petting [41]. Our findings resonate with these prior studies, suggesting that individuals exhibit empathy also towards public agents in need, driven by the observation of context and agent expressions.

In addition to viewing helping robots as a form of work or act of care, ‘helping-as-play’ has emerged as another perspective in our findings, for which the resulting entertaining value can function as an incentive. Playful strategies have been used in various human-agent interaction settings, such as motivating children’s learning [2] or encouraging factory workers in collaboration with robots [17]. Furthermore, it has been shown to effectively engage the public and generate enjoyable experiences among bystanders  [44, 59]. One of the primary challenges for urban robots to ask for help from bystanders is convincing them to invest their time and tolerate potential disruptions. The inherent playfulness in humans could potentially act as an incentive for casual collaborations by transforming disruptions into pleasure and enjoyment. That being said, ‘helping-as-play’ also needs to be carefully employed, as it can present ethical concerns similar to those in ‘helping-as-care’ [4].

The physical constraints introduced by the robot costume, such as limited field of view, changes in perspective, and restricted mobility, while not capable of entirely replicating a robot’s perspective, facilitated participants in departing from a conventional human viewpoint and immersing themselves in the sensations of robotic alienation and otherness  [29]. As articulated by P5, ‘There’s a sense of feeling out of place or not quite fitting in. It seems like there are no peers of my kind in the surroundings. Everyone else is tall, and I am short, so I feel a bit out of place or different.’ This sense of otherness could contribute to participants’ emotional engagement when they assumed the robot role, which was evident in the frequently conveying sentiments of ‘frustration’ or ‘depression’. Additionally, beyond mere empathy with the robot’s emotional state, the robot players also displayed a recognition of its societal function (i.e. its duty as a delivery service entity). For instance, P4 suggested feelings of ‘motivated’ and ‘happy’ when he ‘could continue working’. This profound resonance with the agent’s perspective underscores that our bodystorming approach effectively incorporated the agent’s perspective into the design exploration. However, it is worth noting that, despite the effectiveness of the physical probes in consciously shifting participant’s sensations towards the perspective of an agent [29], it is impossible to completely transcend the human standpoint through these methods as our human nature inherently separates us from things [86].

Consequently, the ‘participation’ of agents in the design process surfaces tensions between humans and agents that stem from the misalignment in goals and benefits in a casual collaboration encounter. P9’s behaviours while playing the robot role offer a striking example. He assertively used the robot costume to brush up against the pedestrian player’s chest — a distinctly pushy gesture meant to force attention and assistance. He later admitted,  ‘I was about to give up being nice’, and even pondered,  ‘That’s why I was considering adopting a more aggressive strategy. I thought, I might just try pushing him onto the road.’. This elicitation of tension and physical friction is less likely to surface in methods that take a purely human-centred perspective or those that rely solely on cognitive abilities. Furthermore, this tension was leveraged to stimulate design ideation, exemplified by P9’s subsequent ideas that emerged from this physical friction. He suggested infusing robots with seemingly annoying nudging behaviours and ‘creating entertainment value’ to possibly uplift people’s moods and thus promote pro-social behaviour.

Despite the strengths of our approach in generating design insights, we acknowledge its limitations. Discussions and reactions concerning helping a casually encountered agent were elicited by role-playing activities, without the incorporation of real technology. Even though the immersive nature of the replicated scenario and participant engagement is evident, the absence of real intelligent agents and the artificial nature of scenarios replicated in laboratory settings might limit the direct applicability of our findings. Acknowledging these inherent limitations of bodystorming method, future research should validate and refine our design considerations through more spontaneous interactions between humans and high-fidelity artifacts (e.g., through Wizard of Oz testing [25] or virtual reality study [92]).