Design Feeling: Emotion as a Language for Design

Empathy Lab sits at the intersection of the Design and Machine Intelligence teams at Google, a fascinating space to think and dream into right now. The way the lab was founded wasn’t in a Shark Tank way; I didn’t think 'Hey, I’ll pitch this and they’ll fund it and it’ll be rad'. Instead, it slowly rose from a series of projects over time; like a magic thread I kept following, figuring out where it would lead. We're an interdisciplinary gang that blends the social sciences with the creative arts; doing experimental research, designing product experiences and making films. I describe it as Design Feeling (a new riff on Design Thinking). Ultimately, I speak human at a table with a bunch of incredibly brilliant folks who are fluent in machine.  

Danielle Krettek at Semi Permanent 2018. Photo: Topy Peet

I believe emotion is a design language of its own — invisible, immaterial and woven within absolutely everything. Tapping into this deep current of human experience is something designers often do instinctually, and I dove particularly deeply into it because it’s so critical to AI right now. It's time for technology’s IQ to match its EQ.  Because it’s missing the thing that most defines us as humans — it’s not our efficient cognition or neat behavior…no.  It’s the watery, feeling space that matters just as much, if not more.  The quiet whispering layer of experience, where intuition and gut-feels rule, where we truly “know” things or people.  It’s where comfort, trust, safety, harmony and ease live for us.  When tech gets it right, it feels harmonious and humane.  

So getting to that…In a Carl Jung-meets-Carmen Sandiego way, I tend to draw inspiration equally from things that move very fast (like advances in tech) and things that never really change (like the human spirit’s search for growth and meaning). My former boss Steve Jobs and Jane Goodall are equal heroes to me in this way.  

Have you heard of John and Julie Gottman? They became famous in the 90s when divorce rates were hockey-sticking because of their science-based knack of telling in 20 minutes whether your relationship was going to make it or not (worst dinner party guests ever). They nailed a ratio that captures the health of a relationship: a 5:1 positive-negative interaction balance is a good relationship, below that is….not.  A really healthy relationship is 20:1, and so on.

With this in mind, I designed a study with three different types of assistive presences. One information driven, one emotionally attuned, and one bland, neutral control. The findings showed that while the information-driven assistant could do much more for people, the emotionally-attuned one managed to achieve a healthy ratio and was far preferred. It just felt better.  So they’d rather have it around than one that could “do more” functionally. It might not make sense at first, but think about who you’d rather spend an 18-hour flight with.

Now let’s move from on-boarding to connecting. In 2015, The New York Times published ‘The 36 Questions That Lead to Love': based on Arthur and Elaine Aron’s Love Lab work.  Their experiment was famous for a “blueprint for intimacy” method where two people ask three sets of 12 questions followed by four minutes of uninterrupted eye contact. They brought total strangers in who came out and ended up in relationships and even marriages. There’s a secret recipe for human connection based on increasing mutual vulnerability, it’s very heady for us.  Something to carefully consider and design for (with a great deal of integrity). The big question is — while this is true of humans, does it apply to the way we could relate to machines too?

Frank Capra said film is one of three universal languages (the other two are math and music). I adore narrative and films as a language for connection; they’re such beautifully designed machines for empathy and a shortcut to a shared emotional experience.

Steven Spielberg to me is hands down one of the best machine intelligence designers of our time. As a child he went stargazing with his father and was fascinated by space exploration. To him it was both an act of science and imagination. While working on Close Encounters of the Third Kind he said 'I knew for an advanced civilization to get to Earth, of course they would have mastered math and physics. But we're humans — emotional creatures — so they’d need to use an emotional language to connect.'

The scene everyone remembers is the 'first contact' scene. The aliens use color and music as their form of expression because these are the languages we’re born with, they’re quite literally neurobiologically wired inside us. Genius. You see the same thing in 2015's Arrival. When Amy Adams' character is on the chopper as they're approaching the alien vessel, the scientist says 'OK, we need to ask them how they got here, let's get into the physics and materials'. And she responds 'How about instead of throwing math problems at them, we start a conversation first?' 

I think thats true for the space of AI as well. It's really easy to get caught up in the math problems, or the 'robot overlords' hyperbole. Instead I like to start humbly; let’s look at where things are at right now and how we can design for all the ordinary and simple aspects of connection; trust, help, mistakes, loyalty, aligning values…that first.

Let's also remember the classic scene from E.T, where Elliott shares his world by showing E.T. all of his favorite things. He’s making this map as he goes, 'this is a fish, this is a peanut, this is a car, this is Lando Calrissian and this is Grito' but what he's really saying is 'this is me'. Because when you look at what a relationship is, it's a series of points over time that (with color and texture) become the topography of the your stories and experiences together. What you see at the end of the scene is the very beginning of the central empathy story that builds towards the end, where they share a heart, feeling what the other is feeling.  It all starts with Elliott saying, 'I'm feeling hungry, are you hungry?'

This is one of the most popular films of all time…and you might think it’s about friendly creatures from space, but it’s not. Spielberg already made that film with Close Encounters, and creatively, he never does the same topic twice. E.T is actually a film about divorce. Spielberg used it to explore this touching question: Is there anything on earth that could possibly heal the broken heart of a boy who’d lost his father?

What could heal that heart? Only something from a completely outrageous place, something from far beyond this child’s ordinary world. Only something that extraordinary could come in and heal that kind of hurt.  

Another reason this film is so beloved is that it gave the world hope, because it floats this idea that if a boy and alien could love each other and connect in this special way… there are no two humans on earth any further apart than that. Which is why it just moves my heart. It’s so beautiful. 

If you look at applying empathy to the AI space, one question I get asked a lot is 'Will it be possible for the machine to feel?’  Based on what I know so far…I believe not. Others may disagree or get into semantics, but I agree with Brene Brown’s definition of true empathy as the ability to say 'that thing you’re feeling, I know it, I’ve felt that too. You’re not alone, me too.'  What I think is far more interesting a promise for humans and machines is the possibility of an empathic leap.

In 2016, 18-time Go world champion Lee Sedol played five games against AlphaGo, a machine player trained by DeepMind (a Google company). Come game two, AlphaGo played move 37, one that has since become famous precisely because it was a move no human would play.

Commentators believed it a rookie move, a simple-minded mistake. Yet by placing that move with all the others played in sequence before, they turned into a network. The series of moves became a team, with move 37 as MVP.  To this, Go Master Sedol said 'I thought AlphaGo was based on a probability calculation, but clearly it is creative because that was a beautiful move'.

I call that an empathic leap because in that moment, you have two Go players: one man, one machine, both at the feet of an ancient honored game. The machine player saw the game in a different way, playing a move a human wouldn't. And so a Go master saw the game anew.  What a gift — for wisdom to see with fresh eyes.  Which moved me again to think, ‘Connecting to a feeling you know through the eyes of another, then sharing a new perspective, learning, growing… that is a core human experience.’ Shared with a machine, that is an empathic leap.  And there are plenty more to be made. 

This idea of going beyond ourselves is so true to the bright promise of AI. 

In 1977, the twin Voyager missions were due to travel to Jupiter and Saturn before cruising off into the universe. Carl Sagan had this great idea, because we all know Carl wasn't just an astrophysicist, but a storyteller and myth maker too: 'Why don't we put something that captures the full spectrum of the human experience on board'. So they curated what eventually became The Golden Record.

What most folks don’t know is that the summer of 1977 was also when Sagan fell in love with his (later to be) wife, Anne. She creative directed the Voyager Interstellar Message Project and described it this way, “It was a chance to tell something of what life on Earth was like to beings of, perhaps, a thousand-million years from now". If that didn’t raise goose bumps, then you’d have to be made of wood.

The Golden Record carried all kinds of beautiful human material: people sneezing, greetings in 59 languages, the sound of a kiss, a mother’s first words to her child. But when the time came to pick a song to include on the record, Anne wondered how she could possibly select one piece of music to represent all of humanity. When she finally did, she was ecstatic, so she called Carl to share this amazing 2,500 year old Chinese song called 'Flowing River'. At the beginning of the call they were collaborators, and by the end of it they were engaged. No one knows what happened in-between but them.

What I love the most about this is another thing that Ann managed to sneak in. The two had decided to include human biorhythms on the record: the electrical impulses of the brain and the nervous system so whoever found it could translate the experience of the body. What she didn't tell anyone was that as she went to the hospital to make that recording, she spent the entire time thinking about Carl and the wonder of being in love.  So Ann smuggled love onto this singular and historic pressing of human history, a signature of that most distinct and wondrous human feeling.

So Carl and Ann taught me a lesson that I now hold as my secret mission: when designing for AI, be sure to sneak the love in.

Design is always a function of empathy, and right now it can also be an act of courage.  AI is an about human values, beliefs, rights and most powerfully, human potential. An essential qualification for working in AI right now is being a courageous, feeling person who cares deeply about people — both the ones you know and the ones you don’t. 

I invite you into the conversation, because this is not just something for computer scientists and engineers to chit-chat about.  This new paradigm requires our EQ to match our IQ in an entirely new way — it takes all kinds of designers, technologists and makers.  AI will learn from all of us, will touch all of us, so it’s about all of us.  This conversation requires a big table. It’s inclusive and you belong here too.  

We need designers, storytellers, psychologists, philosophers, filmmakers, musicians, artists and YOU. There’s a reason that the Nobel Table isn’t just physicists and economists, it’s peacemakers and poets too.

Because at the end of all of this, the essence and beauty of our human experience is entirely about color, not math.  That’s why our motto at the Empathy Lab is TEAM HUMAN.  Now let’s go dream a bright-hearted world into being.