Chelsea Parlett-Pelleriti - Deep Learning is Just LEGO: & Other Hands on Machine Learning Activities

Hello, my name is Chelsea Parlett-Pelleriti, and today I want to tell you that deep learning is just LEGO, and I want to talk to you about other hands-on machine learning activities. Now, if you know anything about me, you may know that recently I made a pretty big career change. I, for years, was teaching as an academic, mostly statistics, machine learning, and recently I transitioned to have a job in industry, and you may think that because I left the classroom that I also left behind teaching, but it turns out that a lot of my job, even in industry, is basically teaching, explaining complicated things to people who may not have heard of those things before.

So, whether you're in the classroom or you're in the boardroom, today I want to talk about hands-on activities that make math tangible, increasing the accessibility, the intuition, collaboration, and therefore increasing the impact of your communication.

Lego convolutions and computer vision

If you've ever taken a deep learning class, this is probably a really familiar sight. In the context of computer vision, this formula represents something called a convolution, and a convolution is basically just this idea that we take a filter and we slide it across our image, allowing it to learn different features about that image so we can use those features to do something exciting, like classify an image or maybe even generate a new image.

And this is exactly what you would see in a typical deep learning classroom, but this is what you would see on computer vision day in my classroom. That's right, we do Lego activities. Instead of just looking at the math alone, we also build physical representations of that with Legos.

Not only is this fun, but it actually makes computer vision more accessible. I want to tell you about a tweet that went viral a few months ago, but before I do, I want to give you a second to just picture a red delicious apple.

Okay, so now that you're picturing the red delicious apple, I want you to take a second to look at this scale and see which end or which point on the scale best matches what you were seeing in your head. On the right hand of the scale, we have complete aphantasia. This is essentially the idea that when you picture something, you don't actually see anything in your mind. On the other left hand side of the scale, you can see the complete opposite. This is people who, when they picture a red delicious apple, can like literally see the apple right in front of them. So take a second and figure out where do you fall.

When I saw the tweet, it was really interesting because I realized that I fell more on the aphantasia side of the scale than I thought. When I picture a red delicious apple, I can't see the apple in front of me. It's more just like a fuzzy vibe of a red delicious apple. And I realized that this might be a reason why some of the math that I've had to learn throughout my entire career has been a little harder for me to grasp, but then it might be for other people. Essentially, when I'm thinking about math, it's hard for me to picture the objects I'm actually working with.

But when we do activities like these Lego convolutions, it completely removes the burden of imagining these objects and just puts them right in front of you. Not only do you get to see them, but you actually get to physically hold and manipulate them, which allows you to focus on the actual math and not having to imagine all the objects that you're working with.

But when we do activities like these Lego convolutions, it completely removes the burden of imagining these objects and just puts them right in front of you.

And it's just kind of fun. Imagine walking into your deep learning classroom and your professor has a ton of Legos out. Maybe it makes it a little less intimidating to do something like computer vision. Maybe you think, I could do that.

Clay gradients and gradient descent

Here's another example of a slide you might see in a deep learning class. In fact, you definitely might see it in a deep learning class because this is a slide from my deep learning class. In this slide, we're talking about something called gradient descent, specifically something called an atom optimizer. And in this case, what we're doing is we're talking about ways to find the minimum of a function. That's all we're doing. In machine learning, there's often a need to find the minimum value of something so that our models perform well.

And while that math is well and good, sometimes it helps to have a physical object in front of you that can sort of help you understand that math. So, in my class, we might do something like make a clay gradient. Or if you don't have time to let clay dry, maybe Play-Doh or a 3D printed gradient. Either way, you have this object in front of you that allows you to not have to imagine all of the gradient math that's happening, but rather be able to see it in front of you.

And there's something to be said about the physical intuition that you get when you're looking at an object. In gradient descent, the idea is that we want to minimize a value, essentially by starting at an initial value and just taking steps downhill until we reach a minimum. And when you physically see an object in front of you, it makes it a lot easier to conceptualize that happening. Maybe you can imagine a raindrop or a little rubber ball rolling down the hill to find a minimum. Or maybe you imagine it rolling down a hill, picking up some momentum and completely overshooting that minimum. In these cases, having this physical object allows us to gain that intuition.

And not to be weird, but you know how sometimes when you look at an object or touch an object, you can just tell what it would be like to lick that object, even though you have never licked that object in your life? Well, that's sort of what these activities are like too. By building a physical representation in the world, we can take advantage of the physical intuition we have about how the world works, and we can use it in order to understand that.

Backpropagation coloring sheet

For this last activity, I'm not going to tell you what the concept is. I want to give you a chance to guess. So I'll give you a couple of seconds to look at this coloring sheet activity and see if you can figure out what the concept behind it is.

If you guessed backpropagation, you would be correct. Backpropagation is a key idea in deep learning and machine learning, which essentially represents the idea that if you have a chain of functions, each one acting on the output of another function, then changing any of the earlier functions is going to have downstream impacts on the outputs of the other functions.

And this is one of those concepts where when someone explains it to you like that, you might go, oh, okay, I get it. You know how sometimes when a teacher is doing something in front of you, either coding or on a whiteboard, and you go, I totally get that. And then when you're forced to actually do it on your own, you go, hmm, maybe I didn't get that quite as much as I thought I did. Well, this coloring sheet is the perfect example of an activity that forces you to confront your knowledge so that you can actually tell if you understand a concept.

In this backpropagation coloring sheet, the idea is to look at neural networks. And what we want to do is we want to take the highlighted weights that are shown in red and think about if I changed this weight, what would the impact be on the output of my neural network? And in order to do that, we have to track all of the different paths through which this weight has an effect on what the output is. Now, if you were just listening to your teacher explain backpropagation, you might go, easy peasy. But then you get to actually coloring your coloring sheet and you realize, hmm, maybe there's something that I'm missing.

Because you're forced to color in every single path. It makes sure that you are doing this problem step by step without skipping any steps so that you can feel confident that you know the information in front of you. And if you don't, then it's a great and kind of silly activity for you to discuss with people around you. At least in my experience, doing activities like this really increases engagement and participation because when people are forced to confront every single step of a problem without skipping any, they realize there might be some gaps in their knowledge. And because it's a fun activity, they're more likely to reach out to their peers and chat about it, maybe explain something they understand or get an explanation for something they don't understand.

Designing hands-on activities

Now, you might not be teaching a deep learning class or you might not even be a teacher at all. But I want to talk to you a little bit about how I think about creating these activities so that whether you're in the boardroom or the classroom, you'll have an idea of what kind of activities might work for you.

The first thing that I think about is I want to design activities that remove the mental load. Like our Lego convolutions, we want to do something physical that allows people to have an easier time processing or thinking about the information we're presenting. If it's difficult for me to imagine the matrix algebra that's happening inside of a computer vision model, then maybe I can offload some of that to a physical object like a Lego convolution. This allows easier accessibility to that knowledge and also just makes it more intuitive, even for those that do have an easy time of visualizing it.

Second, I want to design activities that leverage physical or visual intuition that people have about how the world works. If I can build an activity that allows people to take a concept that might be a little bit abstract and put it physically in the world, that means they might get to use the information they have about how they expect the world to work in order to understand that concept. Just like our clay gradients. If I see this 3D object in front of me, I can sort of sense how would a ball roll down this hill, which allows me to understand the math better.

Lastly, I want to make activities that force people to complete a problem step by step. In having this physical activity where I'm completing something step by step, it makes me confront the ideas that I think I might have learned and make sure that I actually have learned them thoroughly. Good activities like this will also hopefully generate questions, either for you as the person who is leading this exercise or for peers who are also participating. This increases engagement, gets people talking, and allows them both to explain things they already understand, solidifying their knowledge, and get alternative explanations for things that just might not be sticking.

So whatever your context is, you're probably teaching someone something, and I want to encourage you to think about hands-on activities that you can use in order to teach those things better. These hands-on activities are not just fun, but they increase accessibility, intuition, and hopefully collaboration, making sure that whatever it is that you're communicating is actually received and digested by the people that you're trying to communicate to. Those are all the activities I have for you. Thank you so much. I will be around for questions.

These hands-on activities are not just fun, but they increase accessibility, intuition, and hopefully collaboration, making sure that whatever it is that you're communicating is actually received and digested by the people that you're trying to communicate to.

Q&A

Thank you so much, Chelsea, and I'm really happy you're joining us virtually.

Yeah, I have a question. In my experience, the scale that you shared is a really interesting one. I really liked seeing that. It tends to move left and right on the scale, depending on just how I'm feeling that day, even. Have you had that experience as well, or is it usually falling on a certain spot on the scale?

That's a good question. I think I've experienced that, too, that if things are unfamiliar to me, it's a much fuzzier vibe than if something is familiar to me. So I definitely oscillate on the scale a little bit, too, but I just didn't realize that people could literally picture things in front of them. So that was kind of mind-blowing to me. So I'm definitely on that side of the scale, moving around a little bit.

That's really cool. Have you had any experience with sharing these concepts with other departments that have similar needs and accessibility needs? Yeah, I definitely had experience sharing them across. I'm usually not the person who's actually giving it, using these activities in different contexts, but I think it's really valuable, especially in cases where people don't have as much intuition or experience or are feeling a little bit intimidated, because that was me.

I have a bit of a weird background for someone who ended up in statistics. I have a psychology degree, and so coming into some of these concepts was very intimidating. So I think passing these off to people that I know that have classes that are teaching people who have never done deep learning before, or maybe don't have a really big mathematical background, that's really valuable. Those seem to have been the most effective uses of these. Whereas if you have people that intuitively already understand matrix algebra, it's definitely fun and collaborative, but you don't get as much of that reduction in mental load as you would with someone who's struggling with the mental load of learning these concepts.

I feel like you're reading my mind there. That's something that really helps me. Visualizing or looking at something physically really helps me understand a concept like that. I have another question from the virtual audience. So can you give an idea of some elements that students struggled with before you use these physical interventions and that they still might struggle with after you've implemented them? So maybe things that don't really get affected by looking at physical interventions.

Physical implementations of these concepts. Interesting. Okay. Well, let me give you an opposite example first, but I do have a good idea, I think, of something. So one thing that people really struggled with that these activities did help with were things like, for instance, if you're familiar with computer vision, when we have filters that we're applying to an image, we're actually applying that filter to multiple channels. So imagine red, green, blue in the image we're applying it to multiple ones at once. And until you physically see this 3D filter, a lot of my students didn't realize that those filters have depth as well as width and height.

And so that was really helpful for them. What hasn't been super helpful or what these activities don't help with is things that we can't translate into a 3D world. So for instance, a lot of the actual matrix algebra that we're doing day to day is not constrained to be in 3D dimensions. And I think that students still struggle with taking these concepts that we're learning maybe on a simpler plane and projecting them into higher dimensions for more complicated problems.

And these physical activities can't help with that because I at least live in a 3D dimensional world. So I'm not really able to have physical intuitive examples that cross into a multi multidimensional world. Yeah, unfortunately, we all do too. We all live in 3D. And I'm sure we'd love to be able to visualize in the higher dimensions. I know I would.

So another question, what is the reception of the industry audience to your Lego or other toy physical models? I haven't gotten to break out the Legos yet in industry. I'm still really new. But that's not to say I won't. Let's see how it goes. But in terms of things that make things more visually intuitive, more fun, the perception has been largely really good. In fact, a lot of the time that I spend at work, quote unquote, teaching is going to be doing something to think about how to make something more engaging and intuitive to someone who probably has never heard about this before.

And so I do spend a lot of time going through those three things. How can I make things physically intuitive? How can I make it a little bit more engaging and collaborative? So I haven't pulled out the Legos yet. But when I do, I will let you know how it goes.

Awesome. Awesome. So I do have a question based on sharing some of these concepts. So Lori from our virtual audience asked, are there places where you share your activities or others share their activities where other educators could use them? Maybe I'm assuming she means maybe somewhere non-internal, where is there some sort of community where you can look at or, or analyze ideas like this and, and share?

I'm not aware of any besides Twitter is where I'm sharing these activities. I also have a GitHub for my courses. So if you look at my name and find my GitHub, you'll see a lot of these activities like the coloring sheet there. But I would love a community that shares these activity or like shares 3d printing files of cool things that you can use in your class. So if anyone else knows about something, let me know. I'll sign up.

Amazing. Thank you so much for joining us today. Really appreciate it.