Augmented speakers, who use technological devices as a primary verbal communication method, are commonly constrained in their ability to use nonverbal behaviors. This can have significant consequences in conversation — it may limit their opportunities to participate in conversation and to express themselves. In this project we worked towards designing a socially assistive embodied agent that might provide augmented communicators with a nonverbal communication channel.

In order to explore this design space and opportunities within it, we conducted a co-design workshop with augmented communicators (ACs), their caregivers, and puppeteers as experts in motion and nonverbal behavior.

The first part of the workshop we aimed to discover augmented communicators’ needs and challenges in conversation that would benefit from nonverbal support. We used brainstorming and story sharing activities to uncover opportunities and tensions related to communication, and to hear about successful, difficult and challenging communication experiences.

Some of the tools we used in this part were ideation and brainstorming activities, for example using metaphors to describe some abstract ideas and challenges in concrete terms. The results from these activities served as a first step toward identifying potential goals for a sidekick.

Example of “metaphors cards” that were used as options to describe “What does communication mean to me?”

Method and cards by Dan Lockton.

The second part of the workshop was dedicated to making and exploring form, aesthetics and use-cases. We asked participants to select one challenge that came up in previous exercises and that resonated with them, and to use craft materials to prototype an assistive sidekick in two AC-led teams. The goal was to think-through-making by rapidly prototyping ideas in a group, and thus explore how a sidekick might address the needs and values identified during the telling activity.

We selected materials that would allow participants to create basic structures and decorate them using fabric, feathers, and decorative elements. After prototyping, teams described the communication challenge they have selected, demonstrated their sidekick prototype, and discussed possible behaviors within the rest of the group.

The two AC-lead teams came up with two prototypes: one was an anthropomorphic prototype that attempted to have some characteristics similar to the AC, and the other was a tangible sign that used playfulness and humor to communicate a message to conversation partners.

The third part of the workshop aimed to design sidekick motions and behaviors for the goals that were determined in prior parts. In this last stage, puppeteers enacted situations and behaviors using improvisation as a form of “bodystorming”. Improv-based sessions enabled us to explore how previously identified AC needs would play out through motion and behavior, and to generate concrete behaviors for supportive nonverbal sidekicks.

Motion experts explored conversational goals using prototypes, motion and performance: a sidekick that fills in the “silence gap” while the AC is typing, a sidekick that draws attention to the AC, a sidekick that softens a message, and more.

After exploring possible appropriate gestures for each goal, we held discussions to understand what worked, what did not, and what was learnt in the process of improvising motion to support nonverbal supportive goals for sidekicks.

The workshop and design process presented in this case-study resulted in several findings:

• A refined set of augmented communicators conversational goals that a nonverbal robotic sidekick can support.
• Design parameters to guide the design of future nonverbal communication sidekicks: Attention, Precision, and Timing.
• A set of physical prototypes to represent potential sidekick forms and aesthetics.
• A set of motions and behaviors that are suitable for nonverbal sidekicks for a range of specific goals and behaviors.

These four findings tackle four layers of this design space, one building on top of the other. Together, they form a coherent picture of how one might design robotic sidekicks to support ACs’ nonverbal behavior. Read more about our findings in our HRI’2021 paper here.