!!Con West 2020: Dr. Kate Compton - Of Muppets, Metaballs, and Ballroom Dance!

And I’m your emergency speaker for today. They asked me to give this talk on Thursday. I have not practiced this, but it does have a lot of animated JIFES in it. So prepare yourself. JIFES is the RC Cola of GIF pronunciation. It angers everyone. So a little about me, I’m galaxykate, recently graduated from UC Santa Cruz, I’m now Dr. Galaxy Kate. I’m probably best known for creating the language called Tracery which powers the website called cheapbotsdonequick so I’m the proud grandmother of 10,000 Twitter bots that we know of, including Infinite Scream, the greatest ever Twitter bot.

01:12 - So my mission statement is: I bring AI to people that AI doesn’t deserve. AI is a little bit of a trash fire. And I don’t want to just bring poets to it, so they too can live in the trash fire. But to take the tools from AI and bring them out to other people. And so I’m currently being generously funded by the Center for Research in Open Source Software. I also brought a bunch of swag. As part of my mission to bring cool tools to poets.

01:46 - If anybody has made bots, however you define them, I have merit badges, if you think you’ve earned a bot merit badge, and I have lots of zines. I also make an absolute ton of weird prototypes. I almost never finish anything because there’s always new shiny stuff to be made. Yeah. Just lots and lots of prototypes. But my research over the last eight years… So I wrote this dissertation on something called casual creators, which I’ll tell you more about in a moment.

02:16 - This is software used for creativity, but not in the way that Photoshop or Garage Band is about creativity. Those are things that are getting you to a product. Your boss wants this thing. Your brain wants this opera. They’re all about the product. It turns out there is this whole other weird class of software. And I played Kid Pix back in the day. This guy wrote a lovely essay about what it means to have built it and why he built it the way it did. He says: The process of making a picture should be as important as the picture produced.

02:52 - If you’re working for Pixar, this is not a thing to tell your boss. They don’t care about the process or how you feel when you’re making it. They care about the product. But this is a whole class of software that I think is really interesting, because it is about the process. So like I needed a term for this. I called them casual creators. These are everything from Burger King’s Simpson-ify yourself, weird doodly make a cake pop apps any of the things where you doodle on your phone, it multiplies your effort in interesting ways, things where you can doodle music or sound, even things like K’Nex where you’re not so much about making the thing, but the physical pleasure about snapping things together, Rainbow Loom, if any of you are of that generation. But this took me forever when I was writing this.

03:45 - I had to go through the process of: What is creativity? We often say people are creative, like Mozart or Steve Jobs. In my research, this seems to not be true. There’s not a class of creative and uncreative people. As far as we can tell, everyone is creative. However, has everyone seen the movie Inside Out? The Pixar movie? And at one point, she develops this new superpower of disgust? There is in fact a moment in your childhood from about the age of 6 to 12, where you discover disgust. And it goes from physical disgust of… I don’t like broccoli…

04:22 - To social disgust of I don’t like this person or this activity. And you see other people being disgusted by you. And so if you ask a group of 6-year-olds… Can you paint? Can you draw? Can you dance? They will all say yes! Because at that moment, they know “can” as the physical act. I can move my body, hold a pencil, make my mouth tell a story. If you ask 12-year-olds or heaven forfend 35-year-olds, they sit on their hands, because “can you” means “can you do so without being disgusting to yourself and the world?” And this is actually an issue.

04:56 - When you read about creativity and you’re not Steve Jobs, you read about people’s fear of creativity. And so it’s not so much about are you creative, but like, what is currently keeping you from being creative? What is your fear of creativity? I love this. Can I draw? How to tell if you’re good enough. Not even just good at drawing enough. Are you good enough? Do you have the moral fabric to draw? This is Calvinist art education. So a little bit about problems and creativity.

05:27 - These are just a couple of the problems that have been discovered. You can read any of the previous books on the previous slide or anything on Pinterest to see: What are the issues with creativity? Things like being undirected. Freezing up. Being trained to come up with the “right” answer. Are you good enough for this crowd? Is this the right answer? Is this what you wanted? Being afraid of negative judgments. Running out of ideas. There’s a Winston Churchill… After World War II and being probably one of the most powerful men to ever live on this earth, retired and started watercolor painting and was very bad at it, and he wrote a book about being bad at watercolor painting.

06:08 - And he has this anecdote where he is staring in terror at a blank page and one of his lady friends comes up in a car and throws paint on it, like… Now you can start! And miracle of miracles, it turns out he could. This is not something that affects people who are not good enough or creative enough or powerful enough. Everybody has this aspect in their life. Winston Churchill being a very complicated example. But no matter what you can say about Winston Churchill, he was bad at watercolor painting. So I worked on a game called Spore way back in the day, in 2007. And Spore had this real gem of something called a creature creator. Where you could sculpt out a little creature and it would come to life under your hands. And we got really wonderful letters, hand written letters from people, saying… This game made me feel creative again. Which was kind of awesome. We’re getting a lot of hate on the internet, but we would get hand written letters from people who said…

07:06 - I thought I wasn’t creative anymore, and now I know I was wrong! And this is designed with improv kind of knowledge. Because the guy who designed this was taking high level improv classes at Berkeley while he was designing this. So this is just jammed full of every improv principle he could think of. There’s books like… Impro, which laid out what Improv would later be. So these are things like… You give people prompts. You give people acknowledgment when they do something.

07:35 - Whatever arm or stupid accessory you put on your creature, it will look at it and nine times out of ten, it will go… Yeah! That was really good! And sometimes it’ll be like no. It was terrible. Just to give you feedback of… It acknowledged you. It saw you. Yes, and. This is the most caricatured bit of improv. This yes, and is like… It sees what you did. It internalizes and then it, like, adds something new. So now you have something to bounce off too. So the questions of…

08:05 - So I’m not gonna go a lot into casual creators in today’s talk. I would encourage you to read my zine about it, which is outside. Or just ask me about it on Twitter. The question is: How do we get people to be creative? How do we get people to move? And how do we get people to move creatively? This is gonna be about casual creatives for dance or ways to help people move creatively. And then how do we do all this with a computer? So let’s do some improv. Raise your hand if you’ve felt a sick feeling in your stomach when I said let’s do some improv. All right. So first off… Do something. (cheering) All right. Very good. Say something. Your hand is a puppet. Say hello to your neighbor. Name a Pokemon. Oh, the year was 1778… Yep.

09:07 - So what did this teach us? Aside from the fact that we now know who the Canadians are. (laughter) So we like to be right when we’re answering. If there’s not a right answer, we’ll often freeze up. Like… Reflect on when I said “do something” and you thought… Is this thing that I’m about to do too big? Is it too small? Am I doing the right thing? What’s my neighbor doing? Is my neighbor looking at me? Oh, gosh.

09:34 - Prompts are really great for this, because they give us a few possible right answers. Or they’ll give us a prompt. If I say your hand is a puppet, there are probably lots of ways your hand could be a puppet, but probably the right one is to do this. And I give you a prompt. Say hi to your neighbor. It makes it easy to say hello. There are lots of things you could have done with that prompt, but there was an easy right answer. That gives people just really two choices. Go with the easy route or try to add onto the easy route.

10:02 - And limiting people to two choices seems like a really strong pattern in this sort of thing. So lessons. Prompts can be verbal. I just gave you verbal prompts. They can also be props. I can hand you a shovel on the improv stage and that shovel will tell you what the right answer is and what your options are for creative “righter” answers. They can be music, things like clapping or dancing to whatever beat is… Many of us have personal dancing styles that are just event reaction to beats. They can be clothing. So this is… Things that you should not do to your cat.

10:48 - But this is apparently – this was used in How to Train Your Dragon. They put tape on the back of a cat and used it as art reference. When you add things to your body, it makes you move differently. We’ll talk about that later. There can also be cultural expectations. The 1778 one is – people have expectations of what you say after that. YMCA – after you’re trained into it, you can always dance that dance in the future.

11:18 - Different subcultures can have the dance that everybody can do because this is the dance everybody knows how to do. It makes people who otherwise feel terrified of dancing know that… Yes, I can do the hora, the YMCA, the Electric Slide. And you will do it well enough to not be disgusting to those around you. And again, this is like… Everyone can dance. We’re letting them know that they’re allowed to dance. So one thing that I would like to say here is: Clothing is an improv partner. This is one of the biggest things that I found. Everybody knows if you wear a cape that there are a certain number of cape moves that you can do. If you ever need to do motion capture and somebody is not moving enough, give them a cape. And suddenly they’ll move a lot. There’s a board called the Greatest Movie Capes and there’s like 60 different capes in movies that are great. So a cape is a yes, and.

12:12 - Like, I move and the cape moves more! In an unexpected way! It’s a prompt. I know there are things I can do with this cape. It’s a prop. Like… Maybe it makes me feel like I’m a superhero. It’s a partner. It’s reacting to my emotion. It’s making emotion that I can then react to. And in animation, this is called secondary motion. This idea that when you move, something else moves with you. And so Bugs Bunny’s pigtails, Ariel’s tale, Bugs Bunny’s ears. These are secondary motion. At spore, we had a whole system called wiggles and jiggles, figuring out… Should your legs move? When you run, should your knees move more, or should they be fixed? We got a lot of interesting stuff out this. The animators could make one animation. This is their internal animation tool. They could say… Do the jump animation, and with any possible body morphology, what should happen then? They could tune this very carefully, so stuff could move in a way that they thought was about right. So yeah. That’s how humans do it. Let’s figure out how computers do it. I had to have one all of you AI slide.

13:33 - So now let’s talk about dancing with a computer. This is not specifically what I mean. Although this is something that came out… Gosh… Probably in the early 2000s of a ballroom dance partner out of Japan. But we can talk about types of human-computer dance interaction. I wrote a paper… A very short paper… Trying to at least tease these out. As far as I can tell, there are three different ways that you can respond to somebody’s body movements. There’s costumes and wearables, which is like attached physically to your body, which are like… Immediate body-centric motions.

14:06 - There’s fields, where you’re disrupting a field of something, so I’m walking through grass and the grass, like, parts and waves as I move through it. Or you’re in one of those lagoons with the bioluminescent bacteria, as you move, little sparkle flows flow out around you. And you do that that digitally. And there’s agents, where something is capable of acting on its own. And the question is: How does the computer listen? What is it listening to do? Your body movement? Your heart rate? How does it know your body movement? Is it looking for – maybe there’s a security camera and it’s just looking for pixel differences? Or is it actually trying to figure out where your skeleton is? Is it listening to your voice, trying to match your pitch? Things like that. How does it speak? Is it making visuals around you? Is it making the audio change? Is it physically pressing against you? And then the last one is kind of interesting.

14:59 - Because this is a lever that you can kind of tune. Is it a mirror or an agent or something in between? So here’s a cape. Our good old friend, the improv cape. But this is in Dr. Strange, and his cape is actually a sentient being that fights him sometimes. And it will change throughout the scene and throughout the choreography of… It’s just a cape. I can do things to it. To this object is physically fighting and is a fight partner with me.

15:24 - It’s interesting to watch that choreography. The one on the bottom is the Marx Brothers, a classic comedy scene where somebody is trying to hide from another person by pretending he’s in the mirror but imperfectly matching his movements. You can have something where I’m looking at the computer and the computer is responding to me directly, directly, directly, and then it says… Stop. I’m gonna do some improv here. So you can tune this back and forth. Oh yeah. So… Okay. Let’s talk a little bit about costumes. I do a little bit of dance. I’m really bad at dance. I am just a casual creator at dance. Just dancing well enough to be not horrifying to myself and others, or at least, I have convinced myself that. But dance has this wonderful tradition of adding a lot of secondary motion.

16:13 - Because as anyone in ballroom dance knows, you have you, your partner, and then you have all the flowing dresses around you that augment and basically yes, and your motion. So you do the twirl and the dress says… Wow! What a twirl! Or you jump and your fringe says… Wow! What a jump! So your outfit is your hype man. And like anybody who goes to raves knows that this is true. You have these flow toys. If you just saw somebody standing there, dancing, doing this, you would be like… Wow, that person is not good at dance.

16:43 - But suddenly if there’s fire involved, it’s great! So again, this is like yes, and for physical gestures. Some important things about costumes – it’s locationally, directionally, and temporally linked to the movement. So I do this, and my sleeve should do that immediately. I shouldn’t do this and then the computer chugs for a while and 15 minutes later it’s like… I figured out what to do! It should cause a very direct causal link.

17:10 - And the nice thing is you get kind of emergence, when you move and these things persist over time. So that cool glowing shoe time lapse thing going on down there. I went to a taiko drumming thing recently and thought that was really fascinating. It’s a dance form where you’re hitting drums. And so they’re like… They’ll do this, but as the accent to that, the drum is almost like a wearable or a dance toy, where as you do this, you get a big boom at the end.

17:40 - The one on top is Fred Astaire where he’s throwing down those poppers and tap dancing on top of them. As his feet hit the ground, there’s a snap of his shoe hitting the ground and there’s a little explosion. So it’s like taking the normal tap accessory of hitting your shoe against the ground to create rhythm but even more so, and you get smoke plumes. I promised y’all more Muppets. We’re just gonna keep doing Muppets here. I hope everyone is okay with that. Muppets are really great, because a lot of this talk was inspired by a visit I did to the Atlanta Center for Puppetry Arts, which everyone should go to. And the Henson Wing was open and you can see all the Henson Muppets.

18:20 - I wasn’t looking at a VHS tape for the first time. I was looking at the Muppet itself. Almost all of them are Muppets. They’re not bare felt. Almost all of them have this stuff on them, these dance accessories. And they’re props that tell you how to move this Muppet. Imagine you had Janice over there on your hand and somebody called in sick at the Jim Henson lab. Saying… You’re a visitor. Come in and play Janice for the day. Immediately you put her on and you start doing this. Because her hair says do that.

18:54 - I forget what that guy’s name is, because you start doing this. Because his hair wants to do that. These are hand capes. They tell your hand what to do. This is great, because you can make people play with puppets easily and have new experiences. So there’s an artist, Nick Cave, not that Nick Cave, who makes these things called sound suits that are really incredible, and other people are doing digital animations on top of bodies to have these surreal secondary animations all over the body. Cats does not do this. There’s no body movement in Cats. Why?!?!?!?! And just like one whole scene… One whole slide of Animal, because Animal is the best.

19:36 - Look at all the different things that his hair is doing! Especially in this one, where it’s like… Movement, movement, movement, and then still. As he’s coming to this revelation. So his hair and how it moves and when it’s still and when it’s moving is part of his character. Muppets have almost no facial animation. Many of them don’t have eyes. Count how many Muppets you think have eyes and then look, and it’s almost none of them! The Swedish Chef has almost nothing on his face. He’s like a nose with eyebrows! And yet you would swear up and down that you’ve seen the Swedish Chef blink! And again, it’s all this secondary animation. So yeah.

20:17 - I actually got to play a little bit with this, in a project that I did for Google Creative Labs. They had something that was kind of like the Kinect, but for a regular webcam, so it was able to detect your body pose. It didn’t end up working out and this didn’t end up shipping, but this was me trying to interpret Nick Cave’s sound suits and Muppets into a digital costume people could wear. They wanted something so people could download an app, walk in front of their webcam, and do something. We know from before how bad “do something” is.

20:50 - So I wanted to give them a costume so they would walk in front of the camera and imagine they had very long arms and a giant mohawk. You would move in different ways. There’s a bunch of mask theory stuff you can go into, of like… When you look at yourself and you’re not yourself, you suddenly have wild freedom. As exposited by The Mask, 1995, Jim Carrey? It’s nonfiction, people! But yeah. So that’s like another way to get people over their fear of creativity, is to look at themselves as not themselves.

21:29 - I had a little virtual body, and this is controlled with Perl in Noise, but at that point, I didn’t have access to body data, so I faked my own. You can remap that, accent the curves in different ways, and I wound up with different body costumes that mimic your limbs in different ways. One of these days I’ll ship this when the technology becomes more available. But the most important thing in this sort of project is immediacy, and the second most important is connection. Knowing that you have influenced the world. But that’s often driven by immediacy.

22:12 - I did a lot of research on various academic projects. There’s a lot of generative dance out there. Especially in the computational creativity community. I don’t think they’re getting it right. There are a couple of projects that are getting it more right, so I wanted to show you a couple of those. This is Lumen AI, a huge multiperson project out of Georgia Tech. There’s a dome and you’re in there with a virtual shadow that’s dancing. One of the issues, though, is that what they chose to do was not immediate. They decided to go very agent-based, where this thing is autonomous. So you dance for a while and you pause and it will dance like you, except you’re not sure if it’s improving or the Kinect stopped working or it’s just doing really creative improv. So this has one of the issues. That if you as the designer cannot tell whether or not the Kinect is plugged in, anybody who has been in this space making a really creative thing that yields some input and you’re like…

23:14 - Wow, it’s doing it really well, it’s doing it really well… Oh, the socket wasn’t plugged in… Yeah. You can easily lie to yourself that more improv is happening and not just noise. This is another project from Google that did end up using that TensorFlow stuff. You can see kind of how janky – it’s following the body. So what they did is they very cleverly didn’t do a continuous thing. It feels almost continuous.

23:37 - But it’s just searching the Google library for the pose that most closely fits your thing. So it’s disconnected and kind of jank, but that’s very cleverly papering over an issue with the software. I’m running out of time, so I’ll just go through this real quick. Oculus First Steps does some great stuff improving with the robot and you get to hold hands with the robot, the only time I’ve been able to hold hands with something in VR, which is amazing. But this is the demo and it’s awful. The whole question is: What is dance about? There’s dance notation.

24:13 - You have to notate arm positions, physical positions, where you are in relationship to your partner, which is that ornate scrolly thing, and there’s a whole book on dance notation that is fascinating to a computer scientist. But I do ballroom dance and swing, and I think of them as forces. The one on the bottom is somebody at Google who did this, and they were assigned to do this, and they hand… Swung danced?… Before. And I was like… Here, let’s swing dance. See how it’s all about your forces? You’re pulling back and forth. They ended up capturing that really nicely. So I do a lot of stuff with particles. This is a lot of stuff with particles. I do hand tracking.

24:58 - Make yourself a set of infinite virtual hands that do this, and that way, you don’t have to switch between that all the time. I have a lot of virtual hands that oscillate forever. And then just particles that respond to those hands in some way. They’re either following them or just kind of… Attracted to them. Or generated from them. And so I had this game that I had made a while back, called Falling For You, that’s dead from bitrot, but you have people falling through space and falling in love and making polyamorous families.

25:32 - Which is accidental, because people have two hands, so obviously… But that’s just springs and particles. There’s no actual body animation in here. It’s just springs and particles and force between them. Inverse kinematics is something that happens to bad people. Anyway… So yeah. I was like… I can make a body out of that. And if I’ve got springs and particles, then I can add dance Tribbles… I know I’m out of time. So I’ll rush through the rest of this, but I want to get into Dance Tribbles. I don’t want to say that and leave.

26:06 - So you have a Dance Tribble, a little sentient being that likes bass beats, and every time it hears a bass beat, it’ll pull up your hand. Another one likes treble beats, so you have a rudimentary way to define force-based choreography. So I started making a generator for this. I had AI… Genetic algorithm-based bodies, so you could evolve little different dance morphologies, a dance descriptor, so I got to put that – jazz hands in an actual academic paper. And this is like applying Dance Tribbles to arbitrary meshes. You can see they’re mostly working out here.

26:52 - I’ll let this go for one and a half songs. (chill Latin music) Come on, Switch. Yeah. So anyway… You can see that you can get kind of different stuff going on. If I added that to that, I got something that in fact did not work at all. (laughter) I will revisit this someday. So yeah. We have more input devices than ever.

27:23 - We have Kinects and PoseNet, LeapMotion, Oculus controls can now listen to your bare hands, so you can do fancy hand tracking, and so far all that anybody is using them for is a button! Like… Do this to press a button! Do that to press a button! Do this to press a button! They’re not responding continuously. So my question here… And this is just a bunch of random stuff that I’ve made… To respond to like hand tracking things… How would you make… So what is the future? I tried to search for VR puppets. And there’s… That one up there, and then it’s mostly scary puppets that follow you in the dark.

28:00 - No one is actually having you be the puppets. It’s dreadful. And the one in the middle is the Waldo System, which is basically… You do this with your hand and this is how the Muppets did a lot of their long distance Muppetry. But we’re not… We haven’t revisited that 1980s technology but now with good hand tracking. So yeah. Any questions? This is my talk. Meet me outside. We can talk more about Muppets or I can teach you how to polka. And yeah.

28:31 - Max, who is in here somewhere, and I are doing a workshop in casual creators, at ICCC in Portugal this year, if anybody has anything to send to us. But yeah. That was my talk! .