WEBVTT 1 00:00:09.280 --> 00:00:15.120 Speaker 1: Today we're gonna talk about one of my favorite niche topics. We're gonna talk about 2 00:00:16.200 --> 00:00:21.100 Speaker 1: local minima. This is one of the concepts for which the channel itself is named. 3 00:00:22.280 --> 00:00:27.880 Speaker 1: We're gonna talk about how local and global minima sort of matter to overall 4 00:00:27.940 --> 00:00:33.700 Speaker 1: society, how you can think of them as why you get stuck in life, how to 5 00:00:33.780 --> 00:00:39.600 Speaker 1: move on and grow as a person, how to improve things, how to learn better, and how to 6 00:00:39.660 --> 00:00:43.220 Speaker 1: finally solve some of the problems you've been stuck on for a very long time. 7 00:00:43.520 --> 00:00:49.300 Speaker 1: [clears throat] So first, I'm gonna start with a prediction, and then we're 8 00:00:49.320 --> 00:00:55.280 Speaker 1: gonna back up into it. So here's the prediction. Reality will be 9 00:00:55.400 --> 00:01:01.240 Speaker 1: seeded to the people who still know how to do things. What does 10 00:01:01.280 --> 00:01:06.960 Speaker 1: that mean exactly? Right. Let's back up into what that prediction 11 00:01:07.000 --> 00:01:10.940 Speaker 1: means. This is a prediction for something that will happen in the future. So I'll 12 00:01:10.980 --> 00:01:16.840 Speaker 1: say it again. Reality will be seeded to the people who still know how 13 00:01:16.920 --> 00:01:18.340 Speaker 1: to do things. 14 00:01:22.880 --> 00:01:26.640 Speaker 1: For those of you who understand trading, you know a little bit about derivatives, 15 00:01:27.520 --> 00:01:33.000 Speaker 1: and you know about paper value versus real value. What's the difference? Okay. 16 00:01:33.720 --> 00:01:39.180 Speaker 1: So without diving too much into how currency works, let's say 17 00:01:39.640 --> 00:01:45.540 Speaker 1: you have a piece of gold, a gold bar. Okay? A one-ounce 18 00:01:45.600 --> 00:01:50.280 Speaker 1: gold bar. That is worth, you know, some amount of money or some amount of goods in 19 00:01:50.300 --> 00:01:54.220 Speaker 1: the real world to other people who are willing to trade you that gold for something 20 00:01:54.280 --> 00:02:00.080 Speaker 1: else. Okay? Now imagine I have an IOU. It's a piece of paper that says, "You can 21 00:02:00.140 --> 00:02:05.620 Speaker 1: turn in this piece of paper for one ounce of gold at any point in time." That is 22 00:02:05.700 --> 00:02:11.240 Speaker 1: paper value. It's not real value. Now, you can call your option 23 00:02:12.040 --> 00:02:15.280 Speaker 1: on the real value anytime you want. You just go turn it in at the bank, and you can 24 00:02:15.320 --> 00:02:21.280 Speaker 1: get your gold, right? It's almost as good. Now, those paper value vouchers are 25 00:02:21.320 --> 00:02:25.040 Speaker 1: pretty awesome because you can trade them on exchanges. You can create all sorts of 26 00:02:25.100 --> 00:02:29.380 Speaker 1: interesting derivatives. You can have leverage. You can do all sorts of crazy stuff 27 00:02:29.420 --> 00:02:29.780 Speaker 1: with them. 28 00:02:31.380 --> 00:02:33.280 Speaker 1: Here's where humanity gets into trouble. 29 00:02:34.940 --> 00:02:40.860 Speaker 1: You can continue to print paper value in excess of real value. I can make 30 00:02:41.120 --> 00:02:46.000 Speaker 1: 100,000 vouchers that say, "You can turn in this voucher for a piece of gold." 31 00:02:47.100 --> 00:02:53.000 Speaker 1: However, there may only be 1,000 pieces of gold. If people figure this out, 32 00:02:53.520 --> 00:02:57.460 Speaker 1: they will quickly have a run on the gold. This is actually what a run on a bank is, 33 00:02:58.220 --> 00:03:02.440 Speaker 1: is there is not enough money for you to go get it. It says you have this much in 34 00:03:02.480 --> 00:03:06.640 Speaker 1: your bank account, but they actually don't have that much. Now, there's all sorts of 35 00:03:06.720 --> 00:03:11.880 Speaker 1: layers of protections, but ultimately, that's how a lot of our economy works. The 36 00:03:11.940 --> 00:03:17.720 Speaker 1: paper value is much larger than the real value, and under good times 37 00:03:17.780 --> 00:03:22.560 Speaker 1: when nothing is under stress, that's not a big deal. Who cares, right? If paper 38 00:03:22.600 --> 00:03:26.100 Speaker 1: value exceeds real value, that's only a problem if everybody tries to turn in their 39 00:03:26.220 --> 00:03:31.440 Speaker 1: IOUs at the same time. It's a game of musical chairs, right? And that's the best 40 00:03:31.520 --> 00:03:35.620 Speaker 1: analogy I can think of. There's not enough chairs. It's fine as long as the music is 41 00:03:35.660 --> 00:03:39.080 Speaker 1: playing. But when the music stops, it's not so good. 42 00:03:40.800 --> 00:03:45.680 Speaker 1: So now that you kind of understand the difference between paper and real value, I 43 00:03:45.720 --> 00:03:50.480 Speaker 1: have a theory that... Well, this one is pretty well shared by everybody. Our economy 44 00:03:50.520 --> 00:03:56.360 Speaker 1: is very over-financialized, meaning we made up a bunch of stuff on paper 45 00:03:57.220 --> 00:04:02.260 Speaker 1: that isn't quite backed by the same amount of real value, the ability to 46 00:04:02.320 --> 00:04:08.140 Speaker 1: produce goods, the real goods themselves in storage, the ability to do useful 47 00:04:08.260 --> 00:04:12.380 Speaker 1: stuff in the real world. This is gonna be a running theme through all of Seeking 48 00:04:12.440 --> 00:04:18.339 Speaker 1: Minima. There is no replacement for being good at doing 49 00:04:18.459 --> 00:04:24.360 Speaker 1: stuff that matters to people. There's no replacement for that. No investment, no 50 00:04:24.440 --> 00:04:30.280 Speaker 1: idea, nothing is as good as being able to do something in the real world 51 00:04:30.800 --> 00:04:34.040 Speaker 1: that is useful to another person. All right. 52 00:04:35.920 --> 00:04:41.260 Speaker 1: So besides over-financialized economy, we also have an 53 00:04:41.360 --> 00:04:47.120 Speaker 1: over-specialized economy. I call this the tall tower problem. 54 00:04:48.100 --> 00:04:53.480 Speaker 1: So in finance, there's layers upon layers of derivatives, and you can see when this 55 00:04:53.520 --> 00:04:58.960 Speaker 1: becomes an option, like, mm, back in 2007 when there was the subprime mortgage 56 00:04:59.020 --> 00:05:03.160 Speaker 1: lending crisis. Basically, we just had stacks and stacks of derivatives. They were 57 00:05:03.200 --> 00:05:08.080 Speaker 1: all being shuffled and bundled around, and underneath it all were people's home 58 00:05:08.140 --> 00:05:13.140 Speaker 1: mortgages. But there were so many layers of derivatives, nobody even knew who owned 59 00:05:13.200 --> 00:05:18.920 Speaker 1: what anymore. Right? It was all paper value on paper value on paper value, many 60 00:05:18.980 --> 00:05:24.820 Speaker 1: orders away from a real thing, somebody's house. We have something kind 61 00:05:24.880 --> 00:05:27.160 Speaker 1: of like that happening with specialization. 62 00:05:29.080 --> 00:05:33.940 Speaker 1: So think of a very tall tower. All right? Let's say you have a specialized num- a 63 00:05:34.020 --> 00:05:39.320 Speaker 1: certain number of blocks. Each block represents a skill, a 64 00:05:39.360 --> 00:05:44.460 Speaker 1: capacity you can learn. Right? It's your ability to do something, 65 00:05:45.860 --> 00:05:51.540 Speaker 1: and you can choose to build a tower that is tall. Right? You can be... That is 66 00:05:51.640 --> 00:05:57.640 Speaker 1: narrow, deep expertise. You can be an incredible microbiologist on a 67 00:05:57.720 --> 00:06:03.392 Speaker 1: very specific strain of bacteria. And you can be very good at that thing. And that 68 00:06:03.412 --> 00:06:07.952 Speaker 1: can be incredibly useful to the world as long as that's what the world needs. Or you 69 00:06:08.012 --> 00:06:12.972 Speaker 1: can also know how to repair a car. You can also know how to fix your bike. You can 70 00:06:13.052 --> 00:06:18.112 Speaker 1: also know how to set up your own solar array. Like there's a lot of other things you 71 00:06:18.152 --> 00:06:22.372 Speaker 1: could be learning besides microbiology, a very specific kind, but you won't be as 72 00:06:22.432 --> 00:06:26.492 Speaker 1: good at any of them, right? We all sort of face this trade-off. Do I go deep? Do I 73 00:06:26.532 --> 00:06:31.112 Speaker 1: go broad? So we all have a certain number of skill blocks, finite cap on how much 74 00:06:31.852 --> 00:06:32.892 Speaker 1: stuff we can learn. 75 00:06:35.032 --> 00:06:40.672 Speaker 1: Now, what happens, here's, here's where these two things come into contact really, 76 00:06:41.212 --> 00:06:46.052 Speaker 1: is the over-financialized economy really, really loves specialization because 77 00:06:46.132 --> 00:06:51.452 Speaker 1: specialization is great. It ensures that you can be the best at the 78 00:06:51.552 --> 00:06:56.412 Speaker 1: forty-first layer on your tall tower, right? So if you just go super deep and you 79 00:06:56.472 --> 00:07:00.452 Speaker 1: just say, "Well, I'm gonna make the most money in a hyper-financialized economy," 80 00:07:01.412 --> 00:07:05.711 Speaker 1: the best way to do that is just to get really amazing at something super high value. 81 00:07:06.291 --> 00:07:12.112 Speaker 1: These are often in technology. It's like engineering, medicine, sometimes 82 00:07:12.212 --> 00:07:18.072 Speaker 1: law, or finance, finance itself, right? It's such an esoteric 83 00:07:18.152 --> 00:07:21.712 Speaker 1: thing when you get down and actually think about it. Think about what your job is 84 00:07:21.772 --> 00:07:27.592 Speaker 1: right now. What layer of a tall tower are you standing on, right? Are you the 85 00:07:29.252 --> 00:07:34.712 Speaker 1: digital marketer for Facebook, for pet, for dog walking companies? 86 00:07:35.272 --> 00:07:40.212 Speaker 1: Wow, that is very specific. Now, it makes sense to make a living in a 87 00:07:40.232 --> 00:07:44.312 Speaker 1: hyper-financialized economy being very specialized. That's where the money is. You 88 00:07:44.352 --> 00:07:47.952 Speaker 1: want to be able to distinguish yourself. If you just say, "Ah, I do everything," 89 00:07:48.692 --> 00:07:51.572 Speaker 1: nobody will hire you because they don't want somebody who does everything. They want 90 00:07:51.612 --> 00:07:54.972 Speaker 1: somebody who is the best at the exact thing they want. That's cool. 91 00:07:56.492 --> 00:08:00.292 Speaker 1: But the whole point of this is to say that there are downsides to the tall tower. 92 00:08:01.232 --> 00:08:06.012 Speaker 1: Here's an example. Facebook goes away. Now who are you? 93 00:08:07.392 --> 00:08:12.292 Speaker 1: You're the marketer for Facebook ads for dog walking companies. Sure. Okay. Maybe 94 00:08:12.312 --> 00:08:17.652 Speaker 1: you can pivot a little bit to another platform. Certainly. Yeah. But what happens if 95 00:08:18.112 --> 00:08:23.572 Speaker 1: the ad model for businesses changes on the internet? What if, I don't know, 96 00:08:23.812 --> 00:08:28.252 Speaker 1: something like a generalized, generalized language model makes it so that people 97 00:08:28.312 --> 00:08:34.192 Speaker 1: just aren't looking at very many ads anymore? Hmm. [laughs] Your 98 00:08:34.232 --> 00:08:39.572 Speaker 1: economy, ad economy implodes. There's just like ninety percent less revenue overall 99 00:08:39.652 --> 00:08:44.852 Speaker 1: in the future. You are sitting at the very top of a tall tower that has just been 100 00:08:44.912 --> 00:08:49.992 Speaker 1: shaken, and twenty of those blocks fall off. Now what do you know? 101 00:08:51.052 --> 00:08:55.812 Speaker 1: Nothing. You're useless. You don't know anything. You don't know how to fix your 102 00:08:55.872 --> 00:08:59.072 Speaker 1: car. You don't know how to fix your bike. You don't know how to set up a solar 103 00:08:59.112 --> 00:09:03.632 Speaker 1: array. You don't even know, like, how to pivot to something similar because you've 104 00:09:03.672 --> 00:09:06.092 Speaker 1: gotten so good at one thing and only one thing. 105 00:09:07.732 --> 00:09:11.892 Speaker 1: This is the trouble with tall towers, is that they're not very stable. I'm gonna 106 00:09:11.932 --> 00:09:17.252 Speaker 1: extrapolate this even further and say that our entire society 107 00:09:18.092 --> 00:09:24.032 Speaker 1: is like this. We have a very, very, very tall tower, and we just 108 00:09:24.112 --> 00:09:29.112 Speaker 1: keep putting one more block on top of the other, training people to be some 109 00:09:29.172 --> 00:09:33.732 Speaker 1: hyper-specialized thing, while really understanding very little of the fundamentals 110 00:09:33.792 --> 00:09:39.612 Speaker 1: underneath it or anything adjacent to them. Now, this is certainly not a 111 00:09:39.672 --> 00:09:44.032 Speaker 1: rule, but people who are sort of well-rounded and good at multiple things are 112 00:09:44.172 --> 00:09:48.872 Speaker 1: definitely the exception. And this is not to shame people who have become 113 00:09:48.952 --> 00:09:54.952 Speaker 1: specialized. It made sense in our economy, in our world. But if we're ever 114 00:09:55.052 --> 00:10:00.852 Speaker 1: shaken, as we sometimes are, um, we'll say with some regularity maybe, fi- every 115 00:10:00.912 --> 00:10:05.812 Speaker 1: five to ten years, let's say all of society is shaken by something. Are you gonna be 116 00:10:05.852 --> 00:10:11.812 Speaker 1: one of those blocks that gets shaken out? I think that we're in a very fragile place 117 00:10:11.912 --> 00:10:17.912 Speaker 1: overall in terms of human civilization because our tower is so tall 118 00:10:17.991 --> 00:10:19.172 Speaker 1: and it's not robust. 119 00:10:21.032 --> 00:10:26.472 Speaker 1: So now we're gonna talk about how this fits into minimas, specifically local 120 00:10:26.532 --> 00:10:32.532 Speaker 1: minimas. So I think a lot about how machine learning models actually learn, and 121 00:10:32.552 --> 00:10:36.932 Speaker 1: I think there's so many analogies that you can draw to real life. When you look at 122 00:10:37.012 --> 00:10:42.792 Speaker 1: how a non-human agent learns, there's just like philosophical wonder in 123 00:10:42.832 --> 00:10:46.552 Speaker 1: that. I don't know. I, I can draw so many con- so many interesting parallels to real 124 00:10:46.612 --> 00:10:51.832 Speaker 1: life. So when a machine learning model starts to learn, in this case, I'm talking 125 00:10:51.912 --> 00:10:55.532 Speaker 1: about deep neural networks. When they start to learn something, we'll say 126 00:10:55.592 --> 00:11:01.052 Speaker 1: recognizing objects in photos, they will start with a pretty broad, 127 00:11:01.312 --> 00:11:06.052 Speaker 1: messy approach, right? Let's just try stuff. Kind of like a baby, just trying random 128 00:11:06.112 --> 00:11:11.112 Speaker 1: things that makes no sense. And that's-- so it starts out broad, and then as soon as 129 00:11:11.152 --> 00:11:14.832 Speaker 1: it starts to find a winning strategy, it will start to go deep, right? Start to 130 00:11:14.892 --> 00:11:20.332 Speaker 1: build up that tower. It'll double down on this technique. So what you want as the 131 00:11:20.892 --> 00:11:25.512 Speaker 1: machine learning engineer is you want a model that learns lots of different ways to 132 00:11:25.552 --> 00:11:31.112 Speaker 1: detect things. It has a broad base to its tower, but it does get very good 133 00:11:32.072 --> 00:11:37.772 Speaker 1: as it-- at the actual task, right? You want to actually be able to tell a cat 134 00:11:38.272 --> 00:11:43.412 Speaker 1: from a bicycle or a cat from a cheetah. Chelling- telling a cat from a cheetah is 135 00:11:43.452 --> 00:11:48.872 Speaker 1: very difficult. So you want it to be great, a deep specialist to handle those 136 00:11:48.932 --> 00:11:53.972 Speaker 1: difficult edge cases, but you want it to be generalized as well. It can also tell a 137 00:11:54.032 --> 00:11:59.672 Speaker 1: cat from a bus, which sounds silly to us to say that. But if you have a very narrow 138 00:11:59.732 --> 00:12:04.232 Speaker 1: model and it's not well generalized, that's exactly what it can't do. [laughs] It 139 00:12:04.252 --> 00:12:07.112 Speaker 1: doesn't know the difference between a cat and a bus. It, it doesn't understand even 140 00:12:07.172 --> 00:12:08.592 Speaker 1: how to conceive of these problems. 141 00:12:10.942 --> 00:12:16.782 Speaker 1: So this problem is called overfitting. When you train-- when a machine 142 00:12:16.822 --> 00:12:21.222 Speaker 1: learning model doubles down on one strategy too much, it goes too deep. And in this 143 00:12:21.282 --> 00:12:26.881 Speaker 1: case, like our tall tower problem, this is a tower that is too tall and is flimsy. 144 00:12:28.362 --> 00:12:33.402 Speaker 1: That's called overfitting. It basically tries to memorize the data. It finds a 145 00:12:33.482 --> 00:12:38.282 Speaker 1: technique that works well, and it only does this one thing, no matter what. 146 00:12:40.042 --> 00:12:45.762 Speaker 1: This is what it's like to be s-stuck in a local minima. So if you take a graph, 147 00:12:47.642 --> 00:12:52.922 Speaker 1: we'll say every point on the graph represents how good the model is. It's literally 148 00:12:52.982 --> 00:12:58.462 Speaker 1: its score for how good it is at its specific problem, right? So higher points on the 149 00:12:58.522 --> 00:13:04.282 Speaker 1: graph, in this case, represent bad scores. Lower points on the graph 150 00:13:04.342 --> 00:13:09.862 Speaker 1: represent good scores. Now, if we have the x-axis as time, then you will 151 00:13:09.942 --> 00:13:15.742 Speaker 1: see that over time, you know, it's gonna go up and down, right? Typically, it starts 152 00:13:15.822 --> 00:13:21.142 Speaker 1: bad, and then it gets better, right? Up and down, up and down. Starts to learn a 153 00:13:21.202 --> 00:13:24.862 Speaker 1: little bit. It messes up a little bit, tries a new technique, and it learns over 154 00:13:24.922 --> 00:13:30.922 Speaker 1: time. You expect over time, it's gonna go down, meaning it's learning better. Lower 155 00:13:30.982 --> 00:13:36.342 Speaker 1: points means it's better over time. So you can think of it kind of like a hilly 156 00:13:36.402 --> 00:13:41.162 Speaker 1: terrain, right? It's got peaks and valleys, kind of like a sine wave. 157 00:13:43.322 --> 00:13:49.102 Speaker 1: The valleys are local minima, right? It is the lowest point on a 158 00:13:49.162 --> 00:13:54.502 Speaker 1: curve. Machine learning models can get stuck at this low point on the curve. 159 00:13:55.342 --> 00:14:00.322 Speaker 1: They can never leave. They can't leave because they found a strategy that works 160 00:14:00.562 --> 00:14:06.482 Speaker 1: well, but they can't find a strategy that is 161 00:14:06.742 --> 00:14:12.582 Speaker 1: better overall without first making their own score worse. In 162 00:14:12.662 --> 00:14:18.002 Speaker 1: order to actually get better overall, it actually has to get worse for a little bit. 163 00:14:19.002 --> 00:14:24.042 Speaker 1: So let's draw an analogy to real life. Let's say you are at block number forty on 164 00:14:24.082 --> 00:14:28.762 Speaker 1: your tall tower. You are the Facebook marketer [chuckles] for a dog walking 165 00:14:28.822 --> 00:14:34.542 Speaker 1: companies. Well, if you want to get better overall, you-- 166 00:14:35.122 --> 00:14:40.082 Speaker 1: as in more robust to tower shaking, and you wanna be better at overall life, less 167 00:14:40.142 --> 00:14:45.062 Speaker 1: fragile, you would probably need to learn an adjacent skill. Maybe you learn, like, 168 00:14:45.382 --> 00:14:50.662 Speaker 1: generalized copywriting, or maybe you also-- or just learn a separate tower, right? 169 00:14:50.962 --> 00:14:55.942 Speaker 1: You also learn photography. Okay? Something like this. It's sort of relevant to what 170 00:14:55.982 --> 00:15:01.522 Speaker 1: you're doing, but not exactly the same thing. So in this case, when you start 171 00:15:01.582 --> 00:15:05.862 Speaker 1: learning photography, you're gonna suck at it. You're gonna be bad. It's literally 172 00:15:05.882 --> 00:15:10.542 Speaker 1: going to make you worse overall. You're gonna make-- be made worse overall because 173 00:15:10.562 --> 00:15:15.342 Speaker 1: you can't focus on the one thing you're great at, and you have to start out bad. A 174 00:15:15.382 --> 00:15:21.242 Speaker 1: lot of people get stuck here. They get to a comfort zone in life. They get 175 00:15:21.322 --> 00:15:26.922 Speaker 1: pretty good at one thing, and then they stop. They get stuck in a local minima, just 176 00:15:26.982 --> 00:15:31.542 Speaker 1: like a machine learning model. They can't really branch out and try new things. They 177 00:15:31.622 --> 00:15:36.922 Speaker 1: can't really get into a new career, a new job. They can't really go back to school 178 00:15:36.962 --> 00:15:42.702 Speaker 1: because of the, the perceived risk. The penalty for trying to be something new, more 179 00:15:42.762 --> 00:15:48.202 Speaker 1: flexible, more robust is too high. They can't get over the hump 180 00:15:48.862 --> 00:15:53.322 Speaker 1: to get to a better place. Even if there were a lower valley, remember, low points 181 00:15:53.362 --> 00:15:58.162 Speaker 1: represent better overall fitness score at the problem you're trying to solve, they 182 00:15:58.222 --> 00:16:02.882 Speaker 1: can't get to this lower point because they keep getting stuck. They can't quite 183 00:16:02.942 --> 00:16:06.862 Speaker 1: build the momentum to get over the hill, so they stay where they are in a local 184 00:16:06.902 --> 00:16:07.242 Speaker 1: minima. 185 00:16:09.522 --> 00:16:14.682 Speaker 1: Again, I think this is where humanity is. We are [chuckles] currently stuck in a 186 00:16:14.742 --> 00:16:19.082 Speaker 1: local minima. We can't get to the global minima. We can't even get to lower local 187 00:16:19.142 --> 00:16:23.182 Speaker 1: minima because we're comfortable, because we figured things out. 188 00:16:25.082 --> 00:16:30.862 Speaker 1: So how do you fix it? Again, we can draw analogies to machine learning. In machine 189 00:16:30.901 --> 00:16:32.962 Speaker 1: learning, you have a lot of different strategies to sort of 190 00:16:34.522 --> 00:16:39.342 Speaker 1: get models to try new things, prevent them from over-optimizing. Okay, cool, you 191 00:16:39.402 --> 00:16:44.642 Speaker 1: found one strategy that works, but basically, you force it to be flexible. You force 192 00:16:44.702 --> 00:16:49.342 Speaker 1: it to be unable to use its one strategy that it finds, so it can't be a one-trick 193 00:16:49.402 --> 00:16:54.762 Speaker 1: pony. So what do you do? You do things like dropout. This means literally removing 194 00:16:54.822 --> 00:16:59.942 Speaker 1: parts of its neurons, right? So basically, parts of the network, you turn them off. 195 00:17:00.382 --> 00:17:04.382 Speaker 1: This is like making you selectively forget so that you have to find a more general 196 00:17:04.442 --> 00:17:09.162 Speaker 1: solution, right? Some of the clues are not always there. Some of the ways you did 197 00:17:09.202 --> 00:17:12.662 Speaker 1: things literally removes parts of the network. We can't really do that with humans. 198 00:17:13.302 --> 00:17:18.202 Speaker 1: Um, here's another one, data augmentation. Right. So th-- for the picture example, 199 00:17:18.702 --> 00:17:22.942 Speaker 1: instead of just sending the same pictures of cats and buses, sometimes you flip 200 00:17:23.002 --> 00:17:28.022 Speaker 1: them, sometimes you make them kind of blurry, sometimes you zoom in on them and crop 201 00:17:28.062 --> 00:17:33.002 Speaker 1: them so that it can't just memorize things, and it can't always look for a perfect 202 00:17:33.042 --> 00:17:37.402 Speaker 1: picture of a cat right in the center of the frame. It has to kind of learn, what do 203 00:17:37.422 --> 00:17:41.062 Speaker 1: you do when it's off-center a little bit? What do you do when it's kind of blurry? 204 00:17:41.402 --> 00:17:44.962 Speaker 1: So it has to learn other approaches to win. 205 00:17:46.742 --> 00:17:51.782 Speaker 1: So there are other things you can do. Noise injection. You can do loss function 206 00:17:51.842 --> 00:17:56.822 Speaker 1: penalties. Um, so this is a way of saying l-- uh, nonlinear loss function penalties. 207 00:17:57.202 --> 00:18:01.582 Speaker 1: This is a way of saying, if you're a little bit wrong, that's okay. If you're really 208 00:18:01.622 --> 00:18:06.622 Speaker 1: wrong, we're gonna penalize you, not just, like, the same amount more. If you're 209 00:18:06.702 --> 00:18:11.822 Speaker 1: fifty percent wrong, you don't get fifty percent penalty. You get five hundred 210 00:18:11.882 --> 00:18:17.422 Speaker 1: percent penalty, right? It goes exponential. So if you're really bad, you're way off 211 00:18:17.462 --> 00:18:22.992 Speaker 1: on something, we're gonna penalize you severely. We can do things like that in real 212 00:18:23.032 --> 00:18:29.032 Speaker 1: life too. This is like saying, instead of I'm already on the 41st block 213 00:18:29.072 --> 00:18:34.792 Speaker 1: with my day job. Do I really need to be at the 42nd block by 214 00:18:34.832 --> 00:18:40.472 Speaker 1: learning this other niche tool to be a better Facebook dog walking 215 00:18:40.532 --> 00:18:46.432 Speaker 1: marketer? Or maybe I could learn how 216 00:18:46.512 --> 00:18:51.852 Speaker 1: to seal my radiator in my car, right? Even though it's not optimal. 217 00:18:52.791 --> 00:18:58.092 Speaker 1: Maybe I could do something adjacent, right? Like learning photography or learning 218 00:18:58.152 --> 00:19:04.032 Speaker 1: something else. Become more robust. Not because you have to, but just in case the 219 00:19:04.112 --> 00:19:10.032 Speaker 1: tower gets shaken, right? And really, that's all you need to do 220 00:19:10.612 --> 00:19:16.132 Speaker 1: is deliberately shake yourself. And that's all humanity needs is 221 00:19:16.212 --> 00:19:21.132 Speaker 1: sometimes we need to be shaken. Because if we're not, we just keep building the 222 00:19:21.192 --> 00:19:26.492 Speaker 1: tower taller. We keep being stuck in local minima. 223 00:19:26.992 --> 00:19:32.932 Speaker 1: Hyper-optimizing for a tiny, tiny percentage point of a gain with our 10th layer of 224 00:19:33.252 --> 00:19:39.052 Speaker 1: financial derivative. This is how you get content that isn't content. 225 00:19:39.732 --> 00:19:42.412 Speaker 1: It's vapid. There's nothing there. 226 00:19:44.172 --> 00:19:49.972 Speaker 1: I think I've said that content is becoming double speak. What does it even mean? You 227 00:19:50.032 --> 00:19:55.972 Speaker 1: can't just pump this stuff out. What are we doing? It's very clear to me that we're 228 00:19:56.012 --> 00:20:01.012 Speaker 1: not just over-financialized, we're over-specialized. Too many layers of derivatives 229 00:20:01.092 --> 00:20:07.052 Speaker 1: at every level. And the answer is shake it. Because we're stuck in a local 230 00:20:07.112 --> 00:20:13.012 Speaker 1: minima. And if we ever want to get to the global minima of our society, we should 231 00:20:13.092 --> 00:20:18.892 Speaker 1: thoughtfully shake ourselves. We should thoughtfully start becoming more robust and 232 00:20:18.932 --> 00:20:24.492 Speaker 1: prepared and well-rounded where we can. So that when the tower does fall, 233 00:20:24.712 --> 00:20:25.972 Speaker 1: because it will fall, 234 00:20:27.832 --> 00:20:31.732 Speaker 1: hopefully it doesn't fall all the way down to the first block, if you catch my 235 00:20:31.772 --> 00:20:37.632 Speaker 1: drift. So that's how I think about local minima and why I think they're a useful 236 00:20:37.692 --> 00:20:43.132 Speaker 1: concept. We get stuck because we find something that works well, 237 00:20:44.012 --> 00:20:47.652 Speaker 1: right? This is sort of analogous to 238 00:20:49.412 --> 00:20:52.932 Speaker 1: being happy, being good instead of great, you know, from the book Good to Great. 239 00:20:54.192 --> 00:20:58.412 Speaker 1: Fantastic concept. Very similar. You'll find analogies all over the place. But the 240 00:20:58.432 --> 00:21:01.732 Speaker 1: way I like to think of it is local minima. Because I can just imagine myself being 241 00:21:01.772 --> 00:21:07.552 Speaker 1: on a bike, like stuck at the bottom of a hill in a valley between two hills. And 242 00:21:07.612 --> 00:21:12.752 Speaker 1: even though I know that there's a lower point farther away, I can't get there 243 00:21:12.792 --> 00:21:18.472 Speaker 1: because I'm not willing to climb this hill on my bike. So that is my advice. 244 00:21:19.672 --> 00:21:23.092 Speaker 1: Begin to notice when you're stuck in a local minima. When you're stuck, 245 00:21:24.752 --> 00:21:29.832 Speaker 1: you need to start climbing. You need to see what's over the next hill. Because 246 00:21:29.892 --> 00:21:34.832 Speaker 1: chances are, it's a lower point. It's a better local minima.