Lessons in Choreorobotics
Released on 12/07/2023
Here's how I might make a dance.
I'll start with a move.
Maybe I'll mirror that move to the other side.
How about I repeat that move twice.
Maybe I'll change the scale.
Maybe I'll change the speed.
How about orientation?
Maybe the energetic emphasis.
These phrases are choreographic tools
that I use to modify an existing dance motion.
They were mirroring, repetition, scale, speed,
orientation, and energetic emphasis.
These types of tools, it's just a short list
and a much longer list of ways
that you can modify dance motions.
Choreographers have a very long practice in history
of modifying dance motions in these ways,
and you could see how I could immediately take one motion
and turn it into this long extended, creative,
compelling, and original dance.
But one very critical thing to note
is that these tools are not specific to a human body.
They could be done on any moving body, including a robot.
Shared tools between roboticists and choreographers
are just one thing that these two domains have in common.
And choreo-robotics is the emerging field
that captures all of these intersections
between dance and robotics.
I'm a robot choreographer.
At Stanford, I use artificial intelligence
to teach robots how to do all kinds of new tasks.
Like open doors, much harder than you think it is,
respond to gestures, play music, and dance.
During my PhD, I spent three years
as the inaugural artist in residence
at Everyday Robots at Google X.
I've spent the last several years traveling the world,
dancing with robots,
and choreographing over a dozen different kinds of robots
for art installations and performances.
So people often ask me like, Why I dance with robots?
And as my former colleague and friend, Benjie Holson put it,
Well, the main difference between a robot and a computer
is that a robot moves.
How a robot moves is a vital part of the way
that a robot is perceived?
And choreographers are experts
at making meaning out of movement.
In fact, art and culture is one of the most important ways
that we incorporate new technologies
into our society and our public discourse.
One fundamental part of our species
is our relationships with our tools that we build.
And so if we aren't incorporating them
in art and culture,
then we will have missed this opportunity
to kind of codify them into the public dialogue.
Dance for me is that art and culture.
So we used to dance around fires, we dance for the rain,
we dance at proms and weddings, we dance on TikTok,
and now we dance with robots.
Robots are increasingly becoming part of our everyday lives,
and as this transition continues,
they're going to be recognized for more than just
the utilitarian value that they provide.
They're going to be recognized for
how they show up in the world, how they move.
Movement elicits emotions.
If we aren't intelligent about the way that a robot moves
relative to people in our everyday lives,
we will have missed this incredibly important opportunity
for robots to not only be safe,
but for them to create harmonious, dynamic interactions
with other humans as a part of our daily lives.
We've seen a very similar process before.
When computers moved from military and research applications
into personal computing,
all of these other domains became super relevant,
like psychology, semiotics, animation, sound design.
We observed that process in personal computing,
and we're experiencing something very similar in robotics.
As we move from research labs and factories
into office buildings and sidewalks,
we need domains like dance to help us create these
legible, compatible and safe relationships
between humans and robots.
So towards this end, all of my art,
research, and business endeavors ask this question,
what kinds of novel experiences can we create
that leverage a robot's features as both
a utilitarian tool and an evocative, moving, social agent?
When I was an artist in residence at Everyday Robots,
there were dozens of these kinds of robots running around.
They were wiping tables and sorting trash
and doing all kinds of useful things.
Well, I'm an artist in residence,
so I have these ideas about how can we imbue these robots
with more personality
and give a sense of wonder and delight
in sides of these offices.
So my first idea was that we would have the robots
play these bespoke, sculptural musical instruments.
So the robots are going down the hall, they wipe the table,
they play a little cello, and then they keep going.
And that was my first idea.
So I was brainstorming on this with my friend Tom Emberson,
who's a software engineer at Everyday Robots.
And Tom had this incredible provocation.
He said, What if the robot was the instrument?
So after a few months working with Tom
and a couple of software engineers, Adrian Lee,
Daniel Lamb, Emery Fisher, Kyle Jeffrey,
and our marvelous composer, Peter Van Stratton,
we turned Tom's provocation
into a new software called Music Mode.
In Music Mode, we take every single one of the joint,
the robot's, joint velocities,
and we map them to different sounds
so that the robot makes music while it moves.
This is so different
from how many choreographers are used to working
where you pick a piece of music
and you choreograph something to it.
In this case, we literally took those encoder values
and we translated them into sound, we sonified them.
So you have this robot that's wiggling around
and it's creating music.
This was very artistically challenging
because we needed to have all of these different joints
be able to play in combination with each other
and sound great in any permutation at any time.
And we also wanted the joint itself
to sort of match up with that kind of music.
So if you had a big heavy torso joint,
you wanted a low sound
or a tiny, tiny gripper joint, you needed a high bell sound.
So we decided to go with analog instruments
because we thought it add this warmth and depth
and richness to the robots
that we weren't getting otherwise.
And here's what it looks like just on the robot's elbow,
so you can hear the woodwinds and listen for it.
[gentle music]
Here's what it looked like to have many joints
moving in combination with each other.
For all the musicians out there,
you'll see that the head is going
with these kind of pulsating brass sounds.
We've got the bell of the gripper, the strings on the base.
And here's what it sounds like.
[soft music]
[soft music continues]
This feature became so popular
that we put it on all 180 robots,
and we allowed the software to be turned on at any time
because it didn't interfere with the robot's motions.
So the robots could be doing anything,
and you would hear the music like wipe a table
and you hear the symphony.
[soft music]
[audience clapping]
I got a number of cold emails from different engineers
who were sort of breaking into tears
in the middle of the day
because they were feeling themselves having this moment
of human connection with a robot.
And again, it was this sort of symbiotic nature
between art and human robot interaction and technology
where it just elevated this task
from a table wiping task
into this beautiful part of everyday life.
And made me think a lot about,
wow, we could live that way if we wanted to.
So now that we had music
that you could dynamically generate with movement,
I wanted to create this project that would get the robots
to improvisationally move around
so we could have many robots
who were all playing this at once.
And this is now called Project Starling.
So Project Starling started from this question,
what would it be like for humans and robots to move together
as if they were in a canonical,
sort of beautiful flock together,
as if it was as natural
as all of us moving through this building?
And we started with this algorithm called Boids Algorithm,
it's a very well-known flocking algorithm
and then we added a ton of stuff on top of it.
So one of the features that we added
was this circling feature.
We could get the robot to sort of swirl in these whirlpools
around the middle of the atrium.
We also started working on a following feature.
So you can imagine kind of being a camp counselor
and having your kids follow you around.
That's a little bit what this felt like.
So this was our following feature that we incorporated.
We added these parallel lines so you could have this sort of
very beautiful wave-like action
that you were experiencing with the robots
as they were going by.
And then I trained a machine learning model
to select between the gains
for all of these different features,
much like I myself, a choreographer would.
What does that mean?
It means that every single time these robots would flock,
the experience was completely different.
It was improvisational, it was surprising,
it was organic, it felt different, it was compelling.
So every single time that this experience happened,
it was never the same.
And this is exactly what we were trying to capture
about this sort of symbiotic nature of art and technology.
We then added some gesture recognition on top of it
so I could put my hands together
and the robots would all look it up at the sky.
I put my left hand in the air, they would all turn,
put my right hand in the air,
they'd all open and close their grippers.
So again, we kind of, little by little,
we were layering on this artistry
and it allowed you to have a new novel experience
every single time.
So this is what everything looks like all together.
The music is music mode.
The flocking behavior, of course, is from Project Starling.
[gentle music]
[audience clapping]
Kids love this, by the way,
and they're a very tough audience.
So I thought that was a great endorsement.
And we actually ran a research experiment,
both with music mode and with Project Starling.
And we found that robots that were running music mode
while they were doing a variety of different kinds of tasks
were considered more anthropomorphic,
animate, likable, safe, and intelligent.
This was what we learned from music mode.
And so then we ran a study with this flocking behavior
with Project Starling,
and we asked people what they experienced
as they did this experiment.
And these are some of the responses that we got.
I feel like I'm in heaven.
Am I ascending?
They were so cute out there singing and dancing.
I have no idea what they were actually supposed to be doing,
but it made me very happy.
And This feels poetic.
I'm having a visceral emotional reaction.
We are all spending quite a bit of time these days
talking about artificial intelligence and robotics.
I invite you to consider all the ways
in which you can use artificial intelligence and robotics
to unlock deep, human feeling
to create businesses, research and art
that are profound, joyful, and meaningful.
Maybe even something that feels like you're ascending.
My work is about making robots more friendly,
interactive, easy to teach, and easy to navigate around
and to take robots a step further,
to elevate them from a tool
to a whimsical part of everyday life.
And again, I get asked all the time, why should we do this?
I've got a lot of great answers about robot safety,
robot legibility, human robot collaborative effectiveness.
Yet my tendency is to ask why not?
What kind of world do we wanna live in?
One where our experiences with robots and other technologies
make us feel fear, boredom, confusion, less human?
Or one where our proximity to and collaborations with robots
give us a sense of curiosity, joy, and deep humanness?
Let's keep exploring,
building businesses, research and art
that creates the world we want to live in.
[audience clapping]
Meet the NASA Scientist Who Tracks Dangerous Asteroids in Earth’s Orbit
App Pack | These Apps Offer Relief During Tax Season
Meet the Robot that Presses Vinyl in 30 Seconds
New Rules for Stylish and Proper Behavior with Guest Voice of Reason: Jerry Seinfeld
Super Bowl 100: How the NFL Can Survive the Concussion Epidemic
Depicting Computer Graphics
Cove Puts You In Touch With Your Emotions Through Music
The DIY Space Suit
Teens Move into One House to Pursue Their Dreams in Silicon Valley as the 2014 Thiel Fellows
The Two-Minute Lightning Pitch