you2idea@video:~$ watch EV7WhVT270Q [4:25:13]
// transcript — 2908 segments
0:00 - The following is a conversation all about the state-of-the-art in artificial
0:04 intelligence, including some of the exciting technical breakthroughs and
0:07 developments in AI that happened over the past year, and some of the interesting things we think might happen this upcoming
0:15 year. At times, it does get super technical, but we do try to make sure that it remains accessible to folks
0:23 outside the field without ever dumbing it down. It is a great honor and pleasure to be able to do this kind of
0:31 episode with two of my favorite people in the AI community, Sebastian Raschka and Nathan
0:38 Lambert. They are both widely respected machine learning researchers and engineers who also happen to be great
0:45 communicators, educators, writers, and X posters. Sebastian is the author of two books
0:53 I highly recommend for beginners and experts alike. First is
0:57 Build a Large Language Model from Scratch and Build a Reasoning Model from Scratch. I
1:05 truly believe in the machine learning world, the best way to learn and understand something is to build it
1:13 yourself from scratch. Nathan is the post-training lead at the Allen Institute for AI,
1:21 author of the definitive book on Reinforcement Learning from Human Feedback.
1:26 Both of them have great X accounts, great Substacks. Sebastian has courses on YouTube, Nathan has a podcast.
1:34 And everyone should absolutely follow all of those. those. This is the Lex Fridman podcast. To support it, please
1:41 check out our sponsors in the description, where you can also find
1:45 links to contact me, ask questions, get feedback, and so on. And now, dear friends, here's Sebastian Raschka and Nathan Lambert.
1:57 So I think one useful lens to look at all this through is the so-called DeepSeek moment. This happened about
2:05 a year ago in January 2025, when the open-weight Chinese company DeepSeek released DeepSeek R1, that I
2:13 think it's fair to say surprised everyone with near-state-of-the-art
2:16 performance, with allegedly much less compute for much cheaper. And from then
2:24 to today, the AI competition has gotten insane, both on the research and product level. It's just been accelerating.
2:32 discuss all of this today, and maybe let's start with some spicy
2:36 questions if we can. Who's winning at the international level? Would you say it's the set
2:42 of companies in China or the set of companies in the United States?
2:46 And Sebastian, Nathan, it's good to see you guys. guys. So Sebastian, who do you think is winning?
2:53 - Winning is a very broad term. I would say you mentioned the DeepSeek moment, and I think DeepSeek is winning
3:01 the hearts of the people who work on open-weight models because they share
3:05 these as open models. Winning, I think, has multiple timescales to it. We have today, we have next year, we have in 10
3:12 years. One thing I know for sure is that I don't think nowadays, in 2026, that there will be any
3:20 company that has access to technology that no other company has access to. That is mainly because researchers
3:28 are frequently changing jobs and labs. They rotate. I don't think there will be a clear winner in terms of
3:36 technology access. However, I do think there will be, The differentiating factor will be budget and hardware constraints.
3:43 I don't think the ideas will be proprietary, but rather the resources needed to implement them. I don't see
3:54 currently a winner-take-all scenario. I can't see that. At the moment.
3:59 - Nathan, what do you think? - You see the labs put different energy into what they're trying to do, and
4:04 I think to demarcate the point in time when we're recording this, the hype
4:08 over Anthropic's Claude Opus 4.5 model has been absolutely insane, which is just... I mean, I've used it and built stuff
4:16 in the last few weeks, and it's... it's almost gotten to the point where it feels like a bit of
4:20 a meme in terms of the hype. And it's kind of funny because this is very organic, and then if we go back a few months
4:26 ago, we can see the release date and the notes, as Gemini 3 from Google got
4:30 released, and it seemed like the marketing and just, like, wow factor of that release was super
4:36 high. But then at the end of November, Claude Opus 4.5 was released and
4:40 the hype has been growing, but Gemini 3 was before this. And it kind of feels
4:44 like people don't really talk about it as much, even though when it came out, everybody was like, this
4:48 is Gemini's moment to retake Google's structural advantages in AI. And Gemini 3 is a fantastic model, and I still use it.
4:56 It's just kind of differentiation is lower. And I agree with Sebastian; what you're saying with all these, the idea space is
5:04 very fluid, but culturally Anthropic is known for betting very
5:08 hard on code, which is the Claude Code thing, is working out for them right now. So I
5:12 think that even if the ideas flow pretty freely, so much of this is
5:16 bottlenecked by human effort and the culture of organizations, where Anthropic
5:20 seems to at least be presenting as the least chaotic. It's a
5:24 bit of an advantage, if they can keep doing that for a while. But on the other
5:28 side of things, there's a lot of ominous technology from China where
5:32 there's way more labs than DeepSeek. So DeepSeek kicked off a movement within China, I say kind of similar to how
5:39 ChatGPT kicked off a movement in the US where everything had a chatbot. There's now
5:44 tons of tech companies in China that are releasing very strong frontier open-weight
5:48 models, to the point where I would say that DeepSeek is kind of losing its crown as the
5:52 preeminent open model maker in China, and the likes of Z.ai with their GLM models, Minimax's models,
6:00 Kimi Moonshot, especially in the last few months, has shown more
6:04 brightly. The new DeepSeek models are still very strong, but that's kind of
6:08 a... it could look back as a big narrative point where in 2025
6:11 DeepSeek came and it provided this platform for way more Chinese
6:15 companies that are releasing these fantastic models to kind of have this new
6:19 type of operation. So these models from these Chinese companies are open-weights, and
6:24 depending on this trajectory of business models that these American companies are
6:27 doing, they could be at risk. But currently, a lot of people are paying
6:31 for AI software in the US, and historically in China and other
6:35 parts of the world, people don't pay a lot for software. - So some of these models like DeepSeek have the love of the people because
6:41 they are open-weight. How long do you think the Chinese companies keep
6:45 releasing open-weight models? - I would say for a few years. I think that, like in the US, there's not a
6:51 clear business model for it. I have been writing about open models for a while,
6:55 and these Chinese companies have realized it. So I get inbound from some of them.
6:59 And they're smart and realize the same constraints: a lot of top US tech
7:02 companies and other IT companies won't pay for an API subscription to
7:07 Chinese companies for security concerns. This has been a long-standing
7:10 habit in tech, and the people at these companies then see open
7:14 weight models as an ability to influence and take part of a huge growing
7:19 AI expenditure market in the US. And they're very realistic about this,
7:23 and it's working for them. I think that the government will see that that is
7:27 building a lot of influence internationally in terms of uptake of the technology,
7:31 so there's going to be a lot of incentives to keep it going. But building
7:35 these models and doing the research is very expensive, so at some point, I expect
7:39 consolidation. But I don't expect that to be a story of 2026, where there will be
7:44 more open model builders throughout 2026 than there were in 2025. And a
7:48 lot of the notable ones will be in China. - You were going to say something?
7:51 - Yes. You mentioned DeepSeek losing its crown. I do think to some extent, yes, but
7:58 we also have to consider though, they are still, I would say, slightly ahead. And
8:02 the other ones—it's not that DeepSeek got worse, it's just that the other ones
8:06 are using the ideas from DeepSeek. For example, you mentioned Kimi—same
8:10 architecture, they're training it. And then again, we have this leapfrogging
8:13 where they might be at some point in time a bit better because they have the more recent
8:17 model. And I think this comes back to the fact that there won't be
8:21 a clear winner. It will just be like that: one person releases
8:25 something, the other one comes in, and the most recent model is probably always the
8:29 best model. - Yeah. We'll also see the Chinese companies have different incentives. Like,
8:33 DeepSeek is very secretive, whereas some of these startups are
8:37 like the MiniMaxs and Z.ais of the world. Those two literally have filed
8:42 IPO paperwork, and they're trying to get Western mindshare and do a lot of outreach there. So I don't know if these incentives will change the
8:49 model development, because DeepSeek famously is built by a hedge fund,
8:53 Highflyer Capital, and we don't know exactly what they use the
8:57 models for or if they care about this. - They're secretive in terms of communication; they're not secretive in terms of the technical reports that
9:03 describe how their models work. They're still open on that front. And we should also
9:06 say, on the Claude Opus 4.5 hype, there's the layer of something
9:14 being the darling of the X echo chamber, on the Twitter echo chamber, and the actual amount of people that are using the
9:22 model. I think it's probably fair to say that ChatGPT and Gemini are focused on the broad user base that just
9:29 want to solve problems in their daily lives, and that user base
9:33 is gigantic. So the hype about the coding may not be representative of the actual use.
9:39 - I would say also a lot of the usage patterns are, like you said, name recognition, brand and stuff, but also
9:47 muscle memory almost, where, you know, ChatGPT has been around
9:51 for a long time. People just got used to using it, and it's almost like a flywheel:
9:55 they recommend it to other users and that stuff. One interesting point is also
9:58 the customization of LLMs. For example, ChatGPT has a memory feature, right? And so you may have a subscription and you
10:07 use it for personal stuff, but I don't know if you want to use that same thing at work.
10:10 Because it's a boundary between private and work. If you're working at a company, they might not
10:14 allow that or you may not want that. And I think that's also an interesting point
10:18 where you might have multiple subscriptions. One is just clean code.
10:22 It has nothing of your personal images or hobby projects in there. It's just like the work thing. And then the other one is your personal thing.
10:30 So I think that's also something where there are two different use cases, and it doesn't mean
10:34 you only have to have one. I think the future is also multiple ones.
10:38 - What model do you think won 2025, and what model do you think is going to win '26?
10:43 - I think in the context of consumer chatbots, it's a question of: are you willing to
10:47 bet on Gemini over ChatGPT? Which I would say, in my gut, feels like a bit of a risky bet
10:54 because OpenAI has been the incumbent, and there are so many benefits to that in tech.
10:58 I think the momentum, if you look at 2025, was on Gemini's side, but they were starting from
11:05 such a low point. And RIP Bard and these earlier attempts at getting started.
11:13 Huge credit to them for powering through the organizational chaos to make that happen.
11:17 But also it's hard to bet against OpenAI because they always come off as
11:22 so chaotic, but they're very good at landing things. And I think,
11:26 personally, I have very mixed reviews of GPT-5, but it must have
11:30 saved them so much money with the high-line feature being a router where
11:34 most users are no longer charging their GPU costs as much. So I think it's very hard to dissociate
11:42 the things that I like out of models versus the things that are going to
11:45 actually be a general public differentiator. - What do you think about 2026? Who's going to win?
11:52 - I'll say something, even though it's risky. I think Gemini will continue to make progress on ChatGPT.
11:56 I think Google's scale, when both of these are operating at such extreme scales—and Google has the
12:03 ability to separate research and product a bit better, whereas you hear so much
12:07 about OpenAI being chaotic operationally and chasing the high-impact thing,
12:11 which is a very startup culture. And then on the software and enterprise side,
12:15 I think Anthropic will have continued success, as they've again and again been set up for that.
12:19 And obviously Google Cloud has a lot of offerings, but I think this kind of Gemini name brand is important for them to build.
12:27 Google Cloud will continue to do well, but that's a more complex thing to explain in the
12:34 ecosystem, because that's competing with the likes of Azure and AWS rather than
12:38 on the model provider side. - So in infrastructure, you think TPU is giving an advantage?
12:45 - Largely because the margin on NVIDIA chips is insane, and Google can develop everything from top to bottom to fit their stack and not have
12:53 to pay this margin. And they've had a head start in building data
12:57 centers. So all of these things that have both high lead times and very hard margins on
13:01 high costs, Google has a just kind of historical advantage there.
13:05 And if there's going to be a new paradigm, it's most likely to come from OpenAI where
13:09 their research division again and again has shown this ability to land a new research idea or a product. Like Deep
13:17 Research, Sora, o1 thinking models—all these definitional things have come from OpenAI, and that's got to be one of their
13:25 top traits as an organization. So it's kind of hard to bet against
13:29 that, but I think a lot of this year will be about scale and
13:33 optimizing what could be described as low-hanging fruit in models.
13:37 - And clearly there's a trade-off between intelligence and speed. This is what ChatGPT-5 was trying
13:44 to solve behind the scenes. It's like, do people actually want intelligence, the broad public, or do they want speed?
13:52 - I think it's a nice variety, or the option to have a toggle there.
13:56 I mean, for my personal usage, most of the time when I look something up, I use ChatGPT to ask a quick question, get the information I wanted
14:03 fast. For most daily tasks, I use the quick model. Nowadays, I think the auto mode is pretty good where you don't have to specifically
14:11 say thinking or non-thinking. Then again, I also sometimes want the pro mode. Very often what I do is, when I have something
14:18 written, I put it into ChatGPT and say, "Hey, do a very thorough check. Are all my references correct? Are all my thoughts
14:25 correct? Did I make any formatting mistakes and are the figure
14:29 numbers wrong?" Or something like that. And I don't need that right
14:33 away. I finish my stuff, maybe have dinner, let it run, come back
14:37 and go through this. I think this is where it's important to
14:41 have this option. I would go crazy if for each query I would have to wait 30 minutes or
14:45 10 minutes even. - That's me. I'm sitting over here losing my mind that you use the router and the non-thinking model.
14:52 I'm like, "How do you live with that?" That's like my reaction. I've been heavily on ChatGPT for a while.
15:01 I never touched ChatGPT-5 non-thinking. I find its tone and then
15:05 its propensity for errors—it has a higher likelihood of errors. Some of this
15:09 is from back when OpenAI released o3, which was the first model to
15:13 do this deep search and find many sources and integrate them for you.
15:17 I became habituated with that. So I will only use GPT-5.2 Thinking
15:21 or Pro when I'm finding any sort of information query for work,
15:25 whether that's a paper or some code reference that I found. And I will regularly have like five Pro
15:32 queries going simultaneously, each looking for one specific paper or
15:36 feedback on an equation or something. - I have a fun example where I needed the answer as fast as possible
15:42 for this podcast before I was going on the trip. like a local GPU running at home and I wanted to run a long RL
15:49 experiment. And usually I also unplug things because you never know if
15:53 you're not at home, you don't want things plugged in. And I accidentally unplugged the
15:57 GPU. My wife was already in the car and it's like, "Oh dang."
16:01 Then basically I wanted as fast as possible a Bash script that runs my different experiments and the evaluation. And
16:09 it's something I know, I learned how to use the Bash interface or Bash terminal, but in that moment I just needed
16:17 like 10 seconds, give me the command. - This is a hilarious situation but yeah, so what did you use?
16:21 - So I did the non-thinking fastest model. It gave me the Bash
16:25 command to chain different scripts to each other and then the thing is like you have the tee thing where you want to route this
16:33 to a log file. Top of my head I was just like in a hurry, I could have thought
16:37 about it myself. - By the way I don't know if there's a representative case, wife waiting in the car- ...
16:40 you have to run, you know, unplug the GPU. You have to generate a Bash script. This sounds like
16:44 a movie, like- Mission Impossible. - I use Gemini for that. So I use thinking for all the information stuff and then Gemini for
16:50 fast things or stuff that I could sometimes Google, which is like it's good at
16:54 explaining things and I trust that it has this kind of background of knowledge and
16:58 it's simple. And the Gemini app has gotten a lot better and-
17:01 It's good for those sorts of things. And then for code and any sort of philosophical
17:05 discussion, I use Claude Opus 4.5. Also always with extended thinking. Extended thinking and inference time scaling is just a way to make the
17:12 models marginally smarter. And I will always err on that side when the progress is very high because you don't know when that'll unlock a new
17:20 use case. And then sometimes use Grok for real-time information or finding something on AI Twitter that I knew I saw and I need
17:28 to dig up and I just fixated on. Although when Grok 4 came out, the Grok 4 SuperGrok Heavy, which
17:35 was like their pro variant was actually very good and I was pretty impressed with it, and then it
17:39 just kind of like muscle memory lost track of it with having the ChatGPT
17:43 app open. So I use many different things. - Yeah. I actually do use Grok 4 Heavy
17:49 for debugging. For like hardcore debugging that the other ones
17:53 can't solve, I find that it's the best at. And... it's interesting
17:57 'cause you say ChatGPT is the best interface. For me, for that same reason, but this could be just momentum-
18:04 Gemini is the better interface for me. I think because I fell in love with
18:09 their best needle in the haystack. If I ever put something that has a lot of context but I'm looking for very specific kinds of information to
18:17 make sure it tracks all of it, I find at least that Gemini for me has been the best. So it's
18:25 funny with some of these models, if they win your heart over-
18:28 for one particular feature on one particular day, for that particular query, that prompt, you're like, "This
18:35 model's better." And so you'll just stick with it for a bit until
18:40 it does something really dumb. There's like a threshold effect. Some smart
18:43 thing and then you fall in love with it and then it does some dumb thing and you're like, "You know what?
18:47 I'm gonna switch and try Claude or ChatGPT." And all that kind of stuff.
18:51 - This is exactly it: you use it until it breaks, until you have a problem, and
18:55 then you change the LLM. And I think it's the same as how we use anything, like our favorite text editor,
19:02 operating systems, or the browser. I mean, there are many options:
19:06 Safari, Firefox, Chrome. They're relatively similar, but then
19:10 there are edge cases, extensions you want, and then you switch.
19:14 But I don't think anyone types the same thing into different
19:19 browsers and compares them. You only do that when something breaks.
19:23 So that's a good point. You use it until it breaks, then you explore other options.
19:28 - On the long context thing, I was also a Gemini user, but the GPT-5.2
19:32 release blog had crazy long context scores. People were like, "Did they
19:36 just figure out some algorithmic change?" It went from 30% to
19:40 70% in this minor model update. It's very hard to keep track of all of these things, but now I look more
19:47 favorably at GPT-5.2's long context. So it's just like, "How do I actually get to testing this?" It's a never-ending battle.
19:57 - Well, it's interesting that none of us talked about the Chinese models
20:01 from a usage perspective. What does that say? Does it mean the
20:05 Chinese models are not as good, or are we just very biased and US-focused?
20:11 - I think currently there's a discrepancy between the model and the platform.
20:15 The open models are more known for the open weights, not the platform yet.
20:19 known for the open weights, not their platform yet. - Many companies will sell you open-model inference at a very low cost.
20:25 With OpenRouter, it's easy to look at multi-model things. You can run DeepSeek on Perplexity. Sitting here, we're like,
20:33 "We use OpenAI GPT-5 Pro consistently." We're all willing to pay for the marginal
20:39 intelligence gain. These models from the US are better in terms of the outputs. I think the question is,
20:47 will they stay better for this year and for years to come? As long as they're better, I'm gonna pay for them.
20:55 There's also analysis showing that the way the Chinese models are served—you could argue this is due to export controls—
21:02 is that they use fewer GPUs per replica, which makes them slower
21:06 and have different errors. If speed and intelligence are in your favor as a user,
21:10 in the US, a lot of users will go for this. And I think that will spur these
21:14 Chinese companies to want to compete in other ways, whether it's free or substantially lower costs, or it'll breed
21:22 creativity in terms of offerings, which is good for the ecosystem.
21:26 But the simple thing is: the US models are currently better, and we use them.
21:30 I tried these other open models, and I'm like, "Fun, but I don't go back."
21:34 models, and I'm like, "Fun, but not gonna... I don't go back to it."
21:38 - We didn't really mention programming. That's another use case
21:42 that a lot of people deeply care about. I use basically half-and-half Cursor and Claude Code, because they're... I
21:50 fundamentally different experiences and both are useful. What do you guys... You program quite a bit,
21:56 so what do you use? What's the current vibe? - So, I use the Codeium plugin for VS Code. You know, it's very
22:03 convenient. It's just like a plugin, and then it's a chat interface that has access to your
22:06 repository. I know that Claude Code is, I think, a bit different. It
22:10 is a bit more agentic. It touches more things. It does the whole project for you. I'm not quite
22:14 there yet where I'm comfortable with that because maybe I'm a
22:18 control freak, but I still would like to see a bit what's going on. And
22:22 Codeium is kind of, right now, for me, the sweet spot where it is
22:26 helping me, but it is not taking completely over. - I should mention, one of the reasons I do use Claude Code is to build the
22:33 skill of programming with English. I mean, the experience is fundamentally
22:37 different. You're... As opposed to micromanaging the details of the process of the generation of the code, and
22:44 looking at the diff, which you can in Cursor if that's the IDE you
22:48 use, and in changing, altering. Looking and reading the code and understanding the code deeply as you
22:55 progress, versus just thinking in this design space and just guiding it
23:03 at this macro level, which I think is another way of thinking about the programming process.
23:10 Also, we should say that Claude Code just seems to be somehow a better utilization of Claude Opus 4.5.
23:18 - It's a good side-by-side for people to do. You can have Claude Code open, you can have Cursor
23:22 open, you can have VS Code open, and you can select the same models on all of them—
23:26 ...and ask questions, and it's very interesting. Claude Code is way better
23:30 in that domain. It's remarkable. - All right, we should say that both of you are legit on multiple fronts:
23:36 researchers, programmers, educators, Tweeters. And on the book front, too. So Nathan, at some
23:47 point soon, hopefully has an RLHF book coming out. - It's available for preorder, and there's a full digital preprint. I'm
23:54 just making it pretty and better organized for the physical thing, which is a lot of why I do
23:58 it, because it's fun to create things that you think are excellent in the
24:02 physical form when so much of our life is digital. - I should say, going to Perplexity here, Sebastian Raschka is a machine learning
24:08 researcher and author known for several influential books. A couple of them that I
24:12 wanted to mention—which is a book I highly recommend—Build a Large
24:16 Language Model from Scratch, and the new one, Build a Reasoning Model from Scratch. So, I'm really excited about
24:23 that. Building stuff from scratch is one of the most powerful ways of learning.
24:27 - Honestly, building an LLM from scratch is a lot of fun. It's also a lot to learn.
24:31 And like you said, it's probably the best way to learn how something really works,
24:35 'cause you can look at figures, but figures can have mistakes. You can look
24:39 at concepts and explanations, but you might misunderstand them.
24:43 But if there is code, and the code works, you know it's correct. I mean, there's no misunderstanding. It's precise.
24:50 Otherwise, it wouldn't work. And I think that's the beauty behind coding.
24:54 It doesn't lie. It's math, basically. So, even though with math, I think you can have mistakes in a book you would never notice.
25:02 Because you are not running the math when you are reading the book, you can't verify this.
25:06 And with code, what's nice is you can verify it. - Yeah, I agree with you about the Build an LLM from Scratch book. It's nice to
25:13 tune out everything else, the internet and so on, and just focus on the book.
25:16 But, you know, I read several history books. It's just less lonely somehow. It's really more fun. Like for example, on the
25:27 programming front, I think it's genuinely more fun to program with an LLM.
25:31 And I think it's genuinely more fun to read with an LLM. But you're right. That distraction should be minimized.
25:40 So you use the LLM to basically enrich the experience, maybe add more context.
25:48 I just find the rate of aha moments for me is really high with LLMs.
25:54 - 100%. I also want to correct myself: I'm not suggesting not to use
25:58 LLMs. I suggest doing it in multiple passes. Like, one pass just offline, focus mode, and then after that... I mean, I also
26:06 take notes, but I, I try to resist the urge to immediately look things up. I do a second pass. It's just more
26:14 structured this way. Sometimes things are answered in the chapter, but sometimes also it just helps to let it sink in and think
26:21 about it. Other people have different preferences. I highly recommend using LLMs when
26:25 reading books. For me, it's not the first thing to do; it's the second pass.
26:30 - My recommendation is the opposite. I like to use the LLM at the beginning
26:35 to lay out the full context of what is this world that I'm now stepping into? But I try to avoid clicking out of
26:43 the LLM into the world of Twitter and blogs, because then you're down this rabbit hole. You're reading
26:50 somebody's opinion. There's a flame war about a particular topic and all of a
26:54 sudden you're in the realm of the internet and Reddit and so on. But if you're purely letting the
27:02 LLM give you the context of why this matters, what are the big picture
27:06 ideas... sometimes books are good at doing that, but not always.
27:12 - This is why I like the ChatGPT app, because it gives the AI a home on your
27:16 computer where you can focus on it, rather than just being another tab in
27:19 my mess of internet options. And I think Claude Code does a good job of making that a joy, where it seems
27:27 very engaging as a product design to be an interface that your AI will then go out into the world. It's something that is
27:35 intangible between it and Codex; it just feels warm and engaging, where Codex can often be as good from OpenAI, but it just,
27:43 feel a little bit rough around the edges. Whereas Claude Code
27:47 makes it fun to build things from scratch, where you just trust that it'll make something. Obviously this is good for
27:55 websites and kind of refreshing tooling and stuff like this, which I use it for, or data analysis. For my
28:02 On my blog, we scrape Hugging Face so we keep download numbers for every dataset and model.
28:06 over time, so we have them. And Claude was just like, "Yeah, I've made use of that
28:10 data, no problem." And I was like, "That would've taken me days."
28:13 And then I have enough situational awareness to be like, "Okay, these trends obviously make sense."
28:17 You can check things. But that's just a wonderful interface where you can have an
28:21 intermediary and not have to do the kind of awful low-level work
28:25 that you would have to do to maintain different web projects.
28:29 - All right. So we just talked about a bunch of the closed-weight models. Let's talk about the
28:36 open ones. Tell me about the landscape of open LLM models. Which are interesting? Which stand out to you and why? We
28:43 already mentioned DeepSeek R1. - Do you wanna see how many we can name off the top of our head?
28:47 - Yeah, without looking at notes. - DeepSeek, Kimi, MiniMax, Z.ai, Moonshot. We're just going Chinese.
28:57 - Let's throw in Mistral AI, Gemma... ...gpt-oss, the open weight model by OpenAI. Actually,
29:05 NVIDIA had a really cool one, Nemotron 3. There, there's a lot of stuff especially at the end of the year. Qwen might be the one—
29:12 - Oh, yeah. Qwen was the obvious name I was gonna say. You can get at least 10 Chinese and at least 10 Western. I think
29:18 that OpenAI released their first open model— ...since GPT-2. When I was writing about
29:26 OpenAI's open model release, they were like, "Don't forget about GPT-2," which I thought was
29:30 really funny 'cause it's just such a different time. But gpt-oss-120b is
29:33 actually a very strong model and does some things that other models
29:38 don't do very well. Selfishly, I'll promote a bunch of Western companies in the US
29:44 and Europe that have these fully open models. I work at the Allen Institute for AI,
29:48 where we've been building OLMo, which releases data and code. And now
29:52 we have actual competition for people that are trying to release everything
29:56 so that others can train these models. There's the Institute for Foundation
29:59 Models/LM360, which has had their K2 models of various types. Apertus is a Swiss research
30:07 consortium. Hugging Face has SmolLM, which is very popular. And NVIDIA's Nemotron 3 has started releasing data as
30:15 well. And then Stanford's Martini Community Project, which is kind
30:18 of making it so there's a pipeline for people to open a GitHub issue and implement a new
30:22 idea and then have it run in a stable language modeling stack.
30:26 This space, that list was way smaller in 2024— ...so I think it was just AI2. So it's a great thing for more people to get
30:34 involved and to understand language models, which doesn't really have a
30:38 Chinese analog. While I'm talking, I'll say that the Chinese open language models tend to be
30:46 much bigger, and that gives them higher peak performance as MoEs, where a lot of
30:50 these things that we like a lot, whether it was Gemma and Nemotron, have tended to be smaller models from the US, which is starting
30:58 to change from the US and Europe. Mistral Large 3 came out, which was a
31:01 giant MoE model, very similar to DeepSeek architecture in December. And then a startup, RCAI, and both Nemotron and NVIDIA have teased MoE
31:13 models way bigger than 100 billion parameters- like this 400 billion parameter range coming in this Q1
31:19 2026 timeline. So I think this kind of balance is set to change
31:23 this year in terms of what people are using the Chinese versus US open models
31:27 for, which I'm personally going to be very excited to watch.
31:32 - First of all, huge props for being able to name so many of
31:36 these. Did you actually name LLaMA? - No. - I feel like ... - RIP.
31:41 - This was not on purpose. - RIP LLaMA. All right. Can you mention some interesting models that stand out? You mentioned
31:48 Qwen 3 is obviously a standout. - So I would say the year's almost bookended by both DeepSeek
31:55 V3 and R1. And then on the other hand, in December, DeepSeek-V3.2.
31:59 Because what I like about those is they always have an interesting
32:02 architecture tweak that others don't have. But otherwise, if you want to go with the
32:07 familiar but really good performance, Qwen 3 and, like Nathan said, also gpt-oss-120b. And I think what's
32:15 interesting about it is it's kind of like the first public or open
32:19 weight model that was really trained with tool use in mind, which
32:23 I do think is kind of a paradigm shift where the ecosystem was not quite
32:27 ready for it. By tool use, I mean that the LLM is able to do a web
32:31 search or to call a Python interpreter. And I do think it's a standout because it's a huge
32:36 unlock. Because one of the most common complaints about LLMs are, for example, hallucinations, right?
32:43 And so, in my opinion, one of the best ways to solve hallucinations is to
32:47 not try to always remember information or make things up. For math, why not use a calculator app or Python?
32:54 If I ask the LLM, "Who won the soccer World Cup in 1998?" instead of just trying to memorize, it could go
33:02 do a search. I think mostly it's still a Google search. So ChatGPT and gpt-oss-120b, they would do a tool call to
33:09 Google, maybe find the FIFA website. Find, okay, it was France. It would get you that information reliably instead of just trying to
33:16 memorize it. So I think it's a huge unlock which right now is not fully utilized yet by the open-source, open-weight
33:24 ecosystem. A lot of people don't use tool call modes because I think, first, it's a trust thing. You don't want to run this on your computer
33:31 where it has access to tools, could wipe your hard drive or whatever. So you want to maybe
33:35 containerize that. But I do think that is like a really important step for the upcoming years to have this ability.
33:44 - So a few quick things. First of all, thank you for defining what you mean by tool use. I think that's a great thing to do in general
33:51 for the concepts we're talking about. Even things as sort of
33:54 well-established as MoEs. You have to say that means mixture of experts, and you kind of
34:01 have to build up an intuition for people what that means, how it's actually utilized, what
34:05 are the different flavors. So what does it mean that there's just such an
34:09 explosion of open models? What's your intuition? - If you're releasing an open model, you want people to use it, is the first and foremost thing.
34:17 And then after that comes things like transparency and trust. I think when
34:21 you look at China, the biggest reason is that they want people around the
34:25 world to use these models, and I think a lot of people will not. If you look outside
34:29 of the US, a lot of people will not pay for software, but they might have computing resources where you can put a model on
34:33 it and run it. I think there can also be data that you don't want to send to the
34:37 cloud. So the number one thing is getting people to use models, use
34:41 AI, or use your AI that might not be able to do it without having access
34:45 to the model. - I guess we should state explicitly, so we've been talking about these Chinese models
34:50 and open weight models. Oftentimes, the way they're run is locally. So it's not like you're sending your data to China or to whoever developed
35:01 Silicon Valley, or whoever developed the model. - A lot of American startups make money by hosting-
35:07 ...these models from China and selling them. It's called selling
35:10 tokens, which means somebody will call the model to do some piece of work.
35:15 I think the other reason is for US companies like OpenAI. They are so GPU
35:19 deprived. They're at the limits of the GPUs. Whenever they make a release,
35:23 they're always talking about like, "Our GPUs are hurting." And
35:27 I think during one of these gpt-oss-120b release sessions, Sam
35:31 Altman said, "Oh, we're releasing this because we can use your GPUs. We don't have to
35:34 use our GPUs, and OpenAI can still get distribution out of this," which is another very real thing, because it doesn't cost them anything.
35:43 - And for the user, I think also, there are users who just use the
35:47 model locally how they would use ChatGPT. But also for companies I think it's a huge
35:51 unlock to have these models because you can customize them, you can train them, you
35:55 can add post-training, add more data. Like, specialize them into, let's say, law, medical models, whatever you
36:02 have. And the appeal, you mentioned Llama, the appeal of the open-weight
36:06 models from China is that the open-weight models' licenses are even friendlier. I think they are just unrestricted open source licenses
36:13 where if we use something like Llama or Gemma, there are some strings
36:17 attached. I think it's like an upper limit in terms of how many users you have. And then if
36:21 you exceed, I don't know, so and so many million users, you have to report your
36:24 financial situation to, let's say, Meta or something like that.
36:28 And I think while it is a free model, there are strings attached, and people do like things where strings are not attached. So
36:36 I think that's also one of the reasons, besides performance, why the
36:40 open-weight models from China are so popular, because you can just use them.
36:43 There's no catch in that sense. - The ecosystem has gotten better on that front, but mostly downstream of these new
36:50 providers providing such open licenses. That was funny when you pulled up Perplexity and said,
36:53 "Kimi K2 Thinking hosted in the US." Which is just like an exact... I've never seen this, but
36:57 it's an exact example of what we're talking about where people are sensitive to this.
37:01 But Kimi K2 Thinking and Kimi K2 is a model that is very popular. People
37:05 say that has very good creative writing and also in doing some software
37:09 things. So it's just these little quirks that people pick up on with different models that they
37:13 like. - What are some interesting ideas that some of these models have
37:18 explored that you can speak to, that are particularly interesting to you?
37:22 - Maybe we can go chronologically. I mean, there was, of course, DeepSeek. DeepSeek
37:25 R1 that came out in January of 2025, if we just focus on 2025. However,
37:29 this was based on DeepSeek-V3, which came out the year before
37:33 in December 2024. There are multiple things on the architecture
37:37 side. What is fascinating is... I mean, that's what I do with my
37:41 from-scratch coding projects. You can still start with GPT-2, and you can
37:44 add things to that model to make it into this other model. So it's all
37:48 still kind of like the same lineage. It is a very close relationship between those. But top of my head, DeepSeek—what was
37:55 unique there is the Mixture of Experts. Not that they were inventing Mixture of
37:59 Experts—we can maybe talk a bit more about what Mixture of Experts
38:02 means—but just to list these things first before we dive into detail. Mixture of Experts, but then they also had Multi-head
38:10 Latent Attention, which is a tweak to the attention mechanism, where
38:14 this was, I would say, the main distinguishing factor between these open-weight models. Different
38:23 tweaks to make inference or KV cache size... We can also define
38:27 KV cache in a few moments, but to kind of make it more economical to have long context, to shrink the KV cache size. So what are
38:35 tweaks that we can do? And most of them focused on the attention mechanism. There
38:39 is Multi-head Latent Attention in DeepSeek. There is Group Query Attention, which is still very popular. It's not invented by any of those models.
38:46 It goes back a few years. But that would be the other option.
38:50 Sliding window attention—I think OLMo 3 uses it, if I remember
38:53 correctly. So there are these different tweaks that make the models
38:57 different. Otherwise, I put them all together in an article once where I just compared them. They are very,
39:04 surprisingly similar. It's just different numbers in terms of how many repetitions
39:08 of the transformer block you have in the center. And, like, just little knobs that people tune. But what's so nice about it is
39:16 it works no matter what. You can tweak things. You can move the
39:20 normalization layers around to get some performance gains. And OLMo is always very
39:24 good in ablation studies, showing what it actually does to the model if you move something around. Ablation studies: does it make it better or
39:32 worse? But there are so many, let's say, ways you can implement a transformer and make it still
39:36 work. The big ideas that are still prevalent is Mixture of Experts, multi-head latent attention, sliding window attention, group
39:43 query attention. And then at the end of the year, we saw a focus
39:47 on making the attention mechanism scale linearly with inference token
39:51 prediction. So there was Qwen2-VL, for example, which added a gated delta net. It's kind of inspired by
39:59 State space models, where you have a fixed state that you keep updating. But it
40:03 makes essentially this attention cheaper, or it replaces attention with a cheaper operation.
40:08 - And it may be useful to step back and talk about transformer
40:12 architecture in general. - Yeah, so maybe we should start with the GPT-2 architecture. The
40:17 transformer that was derived from the "Attention Is All You Need" paper.
40:22 The "Attention Is All You Need" paper had a transformer architecture that had two
40:26 parts, an encoder and a decoder. And GPT went just focusing in on the decoder part. It is essentially still a
40:34 neural network and it has this attention mechanism inside. And you predict one token at a time. You
40:41 pass it through an embedding layer. There's the transformer block. The transformer block
40:45 has attention modules and a fully connected layer. And there are some
40:49 normalization layers in between. But it's essentially neural network layers with this attention
40:52 mechanism. So coming from GPT-2 when we move on to gpt-oss-120b, there is, for example, the Mixture of Experts
41:00 layer. It's not invented by gpt-oss-120b. It's a few years old.
41:04 But it is essentially a tweak to make the model larger without consuming more
41:11 compute in each forward pass. So there is this fully connected layer,
41:15 and if listeners are familiar with multi-layer perceptrons, you can think of a mini multi-layer perceptron, a
41:22 fully connected neural network layer inside the transformer. And it's very
41:26 expensive, because it's fully connected. If you have a thousand inputs and a thousand outputs,
41:30 that's like one million connections. And it's a very expensive part in
41:34 this transformer. And the idea is to kind of expand that into multiple feedforward networks. So instead of having
41:41 one, let's say you have 256, but it would make it way more expensive,
41:45 because now you have 256, but you don't use all of them at the same time. So you
41:49 now have a router that says, "Okay, based on this input token, it would be
41:53 useful to use this fully connected network." And in that context, it's called an expert. So a Mixture of Experts means you have multiple
42:01 experts. And depending on what your input is, let's say it's more
42:05 math-heavy, it would use different experts, compared to, let's say,
42:09 translating input text from English to Spanish. It would maybe consult different
42:12 experts. It's not quite clear, I mean, not as clear-cut to say, "Okay, this
42:16 is only an expert for math and for Spanish." It's a bit more
42:20 fuzzy. But the idea is essentially that you pack more knowledge into the network, but not all the knowledge is used all the time.
42:27 That would be very wasteful. So, during the token generation, you are more selective. There's a router that selects which
42:34 tokens should go to which expert. It adds more complexity. It's
42:38 harder to train. There's a lot that can go wrong, like collapse and
42:42 everything. So I think that's why OLMo 3 still uses dense... I mean, you
42:46 have OLMo models with Mixture of Experts, but dense models, where dense
42:50 means... So also, it's jargon. There's a distinction between dense and sparse. So Mixture of Experts is considered sparse,
42:58 because we have a lot of experts, but only a few of them are active. So that's called
43:01 sparse. And then dense would be the opposite, where you only have one fully
43:05 connected module, and it's always utilized. - So maybe this is a good place to also talk about KV cache.
43:11 But actually, before that, even zooming out, like fundamentally, how many new ideas have been implemented from GPT-2 to today?
43:22 Like, how different really are these architectures? - Take the Mixture of Experts. The attention mechanism in
43:29 gpt-oss-120b, that would be the Group Query Attention mechanism. So it's a
43:32 slight tweak from Multi-Head Attention to Group Query Attention. So that we have
43:36 too... I think they replaced LayerNorm by RMSNorm, but it's just like a different normalization there and not a big change.
43:43 It's just like a tweak. The nonlinear activation function— people familiar with deep neural networks, I mean, it's the same as
43:51 changing sigmoid with ReLU. It's not changing the network fundamentally.
43:55 It's just a little tweak. And that's about it, I would say. It's not really fundamentally that different. It's still the same
44:02 architecture. So you can go from one into the other by just adding these changes basically.
44:10 - It fundamentally is still the same architecture. - Yep. For example, you mentioned my book earlier. That's a GPT-2 model in the
44:16 book because it's simple and it's very small, so 124 million parameters approximately. But in the bonus materials, I do have
44:24 OLMo from scratch, Gemini 3 from scratch, and other types of from-scratch
44:28 models. And I always start it with my GPT-2 model and just tweak the—well,
44:31 add different components and you get from one to the other. It's kind of like
44:35 a lineage in a sense. - Can you build up an intuition for people? Because when you zoom out,
44:41 you look at it, there's so much rapid advancement in the AI world.
44:46 And at the same time, fundamentally the architectures have not changed.
44:51 So where is all the turbulence, the turmoil of the advancement happening?
44:58 Where are the gains to be had? - So there are different stages where you develop the network or train the
45:05 network. You have the pre-training. Now back in the day, it was just
45:08 pre-training with GPT-2. Now you have pre-training, mid-training, and
45:12 post-training. So I think right now we are in the post-training focus stage. Pre-training still gives you
45:20 advantages if you scale it up with better, higher quality data. But then
45:24 we have capability unlocks that were not there with GPT-2, for
45:28 For example, ChatGPT is basically a GPT-3 model. And GPT-3 is the same as GPT-2 in terms of architecture.
45:36 What was new was adding supervised fine-tuning and reinforcement learning with human feedback. So it's more on the algorithmic side than the
45:44 architecture. - I would say that the systems also change a lot. If you listen to NVIDIA's announcements,
45:48 they talk about things like, "You now do FP8, you can now do FP4."
45:53 What's happening is these labs are figuring out how to utilize more compute to put
45:57 it into one model, which lets them train faster and put more
46:00 data in. And then you can find better configurations faster by doing
46:05 this. So you can look at, essentially, tokens per second per GPU
46:09 as a metric that you look at when you're doing large-scale training.
46:12 You can go from 10k to 13k by turning on FP8 training, which means you're using less memory per parameter in the
46:20 model. By saving less information, you do less communication and train
46:23 faster. So all of these system things underpin way faster experimentation on data and algorithms.
46:35 It's a loop that keeps going where it's hard to describe when you look at architectures and they're
46:39 exactly the same, but the code base used to train these models is vastly different-
46:44 -and you could probably... the GPUs are different but you probably
46:48 train gpt-oss-20b way faster in wall-clock time than GPT-2 was trained at the time.
46:54 - Yeah. Like you said, they had, for example, in Mixture of Experts this
46:58 FP4 optimization where you get more throughput. But I do think, for speed this is true, but it
47:05 doesn't give the model new capabilities. It's just: how much can we
47:09 make the computation coarser without suffering in terms of model performance degradation? But I do think- I mean,
47:17 there are alternatives popping up to the transformer. Text diffusion models,
47:21 a completely different paradigm. And there is also... I mean, although
47:25 text diffusion models might use transformer architectures, it's not an
47:28 autoregressive transformer. And also Mamba models. It's a state space model. But they do have trade-offs, and
47:36 nothing has yet replaced the autoregressive transformer as the state-of-the-art model. For state-of-the-art, you would still
47:44 go with that, but there are now alternatives for the cheaper
47:48 end—alternatives that are kind of making compromises. It's not just one architecture anymore. There are
47:56 little ones coming up. But if we talk about the state-of-the-art, it's pretty much still
48:00 the transformer architecture, autoregressive, derived from GPT-2 essentially.
48:06 - I guess the big question here is, we talked quite a bit about the architecture behind
48:10 the pre-training. Are the scaling laws holding strong across pre-training,
48:16 post-training, inference, context size, data, and synthetic data?
48:20 - I'd like to start with the technical definition of a scaling law-
48:23 -which informs all of this. The scaling law is the power law relationship
48:26 between... You can think of the x-axis, so kind of what you are scaling as a
48:30 combination of compute and data, which are kind of similar, and then the y-axis is like the held-out prediction accuracy over next tokens.
48:38 We talked about models being autoregressive. It's like if you keep a set of
48:44 text that the model has not seen, how accurate will it get when you train? And
48:47 the idea of scaling laws came when people figured out that that
48:51 was a very predictable relationship. And I think that that technical term is continuing, and then the question is, what
48:59 do users get out of it? Then there are more types of scaling where,
49:03 OpenAI's o1 was famous for introducing inference time scaling. And I think
49:07 less famously for also showing that you can scale reinforcement learning training
49:11 and get kind of this log x-axis and then a linear increase in performance on y-axis. So there's kind of these three axes now
49:19 where the traditional scaling laws are talked about for pre-training, which is how big
49:23 your model is and how big your dataset is, and then scaling reinforcement learning,
49:26 which is like how long can you do this trial and error learning that we'll talk about.
49:30 We'll define more of this, and then this inference time compute, which is just letting the
49:34 model generate more tokens on a specific problem. So I'm kind of bullish,
49:39 but they're all really still working, but the low-hanging fruit has mostly been taken,
49:43 especially in the last year on reinforcement learning with verifiable rewards,
49:46 which is this RLVR, and then inference time scaling, which is just why these models feel so different to use, where previously
49:54 you would get that first token immediately. And now they'll go off for seconds,
49:57 minutes, or even hours, generating these hidden thoughts before giving you the first word of your answer. And that's all about this
50:05 inference time scaling, which is such a wonderful kind of step function in terms of how the models change abilities.
50:11 They kind of enabled this tool use stuff and enabled this much better
50:14 software engineering that we were talking about. And this, when we say enabled, is almost entirely downstream of the fact that this
50:21 reinforcement learning with verifiable rewards training just kind of let the models
50:25 pick up these skills very easily. So let the models learn, so if you look at the reasoning process when the models are generating
50:33 a lot of tokens, what it'll often be doing is: it tries a tool, it looks at what it gets back.
50:37 It tries another API, it sees what it gets back and if it solves the problem.
50:41 So the models, when you're training them, very quickly learn to do this.
50:45 And then at the end of the day, that gives this kind of general foundation
50:49 where the model can use CLI commands very nicely in your repo
50:52 and handle Git for you and move things around and organize things
50:56 or search to find more information, which if we were sitting in these chairs a year ago
51:00 is something that we didn't really think of the models doing. So this is just
51:04 kind of something that has happened this year and has totally transformed how
51:07 has totally transformed how we think of using AI which evolution and just unlocks so much value. But it's like,
51:15 just so- pr- unlocks so much value. But it's- it's like, it's not clear what the next avenue will be in terms of unlocking stuff like this.
51:23 I think there's... we'll get to continual learning later, but there's a lot of buzz
51:27 around certain areas of AI, but no one knows when the next step function
51:31 will really come. - So you've actually said quite a lot of things there,
51:35 and said profound things quickly. It would be nice to unpack them a little bit. You say you're bullish basically on every version of
51:43 scaling. So can we just even start at the beginning? Pre-training,
51:49 are we kind of implying that the low- hanging fruit on pre-training
51:53 scaling has been picked? Has pre-training hit a plateau, or is even pre-training still something you're bullish on?
52:01 - Pre-training has gotten extremely expensive. I think to scale up
52:04 pre-training, it's also implying that you're gonna serve a very
52:08 large model to the users. So I think that it's been loosely established the likes of GPT-4 and similar models were around one
52:16 trillion parameters at the biggest size. There's a lot of rumors that they've
52:20 actually gotten smaller as training has gotten more efficient. You want to make
52:24 the model smaller because then your costs of serving go down proportionately.
52:28 These models, the cost of training them is really low relative to the
52:32 cost of serving them to hundreds of millions of users. I think DeepSeek had this famous number
52:36 of about five million dollars for pre-training at cloud market rates.
52:40 In OLMo 3, section 2.4 in the paper, we just detailed how long we had the GPU clusters sitting around for training which
52:47 includes engineering issues, multiple seeds, and it was like about two million dollars to rent the cluster to deal with all the headaches of
52:55 training a model. So these models are pretty— like, a lot of people could get one to 10 million dollars to train a model,
53:03 but the recurring costs of serving millions of users is really billions of dollars of compute. I think that you can look at a
53:10 thousand GPU rental you can pay 100 grand a day for. And these
53:15 companies could have millions of GPUs. Like you can look at how much these things
53:18 cost to sit around. So that's kind of a big thing, and then it's like, if scaling is actually giving you a better model,
53:25 is it gonna be financially worth it? And I think we'll slowly push it
53:29 out as AI solves more compelling tasks, so like the likes of Claude Opus 4.5,
53:33 making Claude Code just work for things. I— I launched this project called the ATOM project, which is American Truly
53:41 Open Models in July, and that was like a true vibe coded website, and like, I have a job to make
53:49 plots and stuff. And then I came back to refresh it in the last few weeks and it's like
53:52 Claude Opus 4.5 versus whatever model at the time was like, just crushed all the
53:56 issues that it had from building in June and July and like, it might be a
54:00 bigger model. There's a lot of things that go into this, but there's still progress coming.
54:04 - So what you're speaking to is the nuance of the y-axis of the scaling
54:07 laws—the way it's experienced versus on a benchmark, the actual intelligence might be different. But still, your intuition
54:15 about pre-training, if you scale the size of compute, will the models get better? Not whether it's financially viable but
54:24 just from the law aspect of it, do you think the models will get smarter?
54:28 - Yeah. And I think that there's... And this sometimes comes off as
54:32 almost like disillusionment from people, leadership at AI companies saying this, but they're like, "It's held for 13 orders of magnitude
54:39 of compute, why would it ever end?" So I think fundamentally it
54:43 is pretty unlikely to stop, it's just eventually we're not even gonna be able to
54:47 test the bigger scales because of all the problems that come with more compute.
54:50 I think that there's a lot of talk on how 2026 is a year when very large Blackwell compute clusters,
54:58 like gigawatt-scale facilities at hyperscalers, are coming online.
55:03 These were all contracts for power and data centers that were signed and sought out in 2022 and 2023.
55:11 So before or right after ChatGPT. It took this two-to-three-year
55:15 lead time to build these bigger clusters to train the models. While there's obviously
55:19 immense interest in building even more data centers than that. So that is the crux
55:23 that people are saying: these new clusters are coming. The labs are gonna have
55:27 more compute for training. They're going to utilize this, but it's not a given. I've seen so much progress that I expect it,
55:34 and I expect a little bit bigger models, and I expect... I would say it's more like we'll see a $2,000 subscription this year. We've seen
55:43 $200 subscriptions. That could 10X again, and these are the kind of things that
55:47 could come, and they're all downstream of this bigger model that offers just a little bit more cutting edge.
55:53 - So, you know, it's reported that xAI is gonna hit that one-gigawatt scale
55:57 early '26, and a full two gigawatts by year end. How do you think they'll utilize that
56:06 in the context of scaling laws? Is a lot of that inference? Is a lot of that training?
56:12 - It ends up being all of the above. So I think that all of your decisions when you're training a model come back to
56:20 pre-training. So if you're going to scale RL on a model, you still need to
56:23 decide on your architecture that enables this. We were talking about other
56:27 architectures and using different types of attention, or a mixture of experts models.
56:31 The sparse nature of MoE models makes it much more efficient to do generation, which becomes a big part
56:39 of post-training, and you need to have your architecture ready so
56:43 that you can actually scale up this compute. I still think most of the
56:47 compute is going in at pre-training. Because you can still make a model
56:51 better, you still want to go and revisit this. You still want the best base model
56:55 you can. And in a few years that'll saturate and the RL compute will just
56:59 go longer. - Are there people who disagree with you and say pre-training is dead?
57:06 It's all about scaling inference, scaling post-training, scaling context, continual learning, scaling data, synthetic data?
57:15 - People vibe that way and describe it in that way, but I think it's not the practice
57:19 that is happening. - It's just the general vibe of people saying this thing is dead-
57:21 - The excitement is elsewhere. So the low-hanging fruit- ...in RL is elsewhere. For example, we released our model in November...
57:28 Every company has deadlines. Our deadline was November 20th, and
57:32 for that, our run was five days, which compared to 2024 is a very long time to just be doing post- training at a model of 30 billion
57:39 parameters. It's not a big model. And then in December, we had another release,
57:43 where we let the RL run for another three and a half weeks, and the model got notably better, so we released it. And that's a
57:51 to just allocate to something that is going to be your peak-
57:55 ...for the year. So it's like- - The reasoning is- - There's these types of decisions when training a model where they just...
58:01 They can't leave it forever. You have to keep pulling in the improvements from researchers. So you redo
58:08 pre-training, you'll do this post-training for a month, but then you
58:12 need to give it to your users. You need to do safety testing. So it's
58:16 just... I think there's a lot in place that reinforces this cycle of
58:20 updating the models. Things improve. You get a new compute cluster
58:24 that lets you do something more stably or faster. It's like you hear a lot about Blackwell having rollout issues, where at
58:31 AI2, most of the models we're pre-training are on 1,000 to 2,000
58:35 GPUs. But when pre-training on 10,000 or 100,000 GPUs, you hit
58:39 very different failures. GPUs break in weird ways, and on a 100,000
58:43 GPU run, you're pretty much guaranteed to have one GPU that is down. Your training code must handle that redundancy, which is a
58:51 very different problem. Whereas what we're doing, like playing with post-training on a cluster,
58:55 or for people learning ML, what they're battling to train these biggest models is just-
59:02 ...mass distributed scale, and it's very different. But that's somewhat different than- That's a systems problem-
59:11 ...in order to enable scaling laws, especially at pre-training.
59:15 You need all these GPUs at once. When we shift to RL, it actually
59:19 lends itself to heterogeneous compute because you have many copies of the
59:23 model. To do a primer for language model reinforcement learning, what you're doing is having two sets of
59:31 GPUs. One you can call the actor, and one you call the learner.
59:34 The learner is where your actual reinforcement learning updates happen.
59:38 These are traditionally policy gradient algorithms. Proximal Policy Optimization, PPO, and Group Relative Policy
59:46 Optimization, GRPO, are the two popular classes. And on the other side you have actors which are generating
59:53 completions, and these completions are what you're going to grade.
59:57 Reinforcement learning is all about optimizing reward. In practice,
60:01 you can have a lot of different actors in different parts of the world
60:05 doing different types of problems, and then you send it back to
60:08 this highly networked compute cluster to do this actual learning
60:12 where you take the gradients. You need to have a tightly meshed network to do different types of parallelism and
60:20 spread out your model for efficient training. Every different type of
60:24 training and serving has these considerations to scale. We talked about pre-training and RL, and then inference time scaling-
60:32 how do you serve a model that's thinking for an hour to 100 million users?
60:35 I don't know about that, but I know that's a hard problem. In order
60:39 to give people this intelligence, there's all these systems problems, and we need
60:43 more compute and you need more stable compute to do it." - But you're bullish on all of these kinds of scaling is what I'm hearing. On the
60:50 inference, on the reasoning, even on the pre-training? - Yeah, so that's a big can of worms here, but there are basically two...
60:58 The knobs are the training and the inference scaling where you can
61:02 get gains. In a world where we had, let's say, infinite compute resources, you want to do all of them. So you have training, you have inference
61:10 scaling, and training is like a hierarchy: it's pre-training, mid-training,
61:13 post-training. Changing the model size, more training data, training a bigger
61:17 model gives you more knowledge in the model. Then the model, let's say, has a
61:22 better base model. Back in the day, or still, we call it a foundation
61:26 model, and it unlocks... But you don't, let's say, have the model be able to solve your most complex tasks
61:34 during pre-training or after pre-training. You still have these other unlock
61:38 phases where you have mid-training or, for example, post-training with
61:42 RL that unlocks capabilities that the model has in terms of knowledge in the pre-training.
61:45 And I think, sure, if you do more pre-training, you get a better base model that you can unlock
61:53 later. But like Nathan said, it just becomes too expensive. We don't have infinite compute,
61:57 so you have to decide, do I want to spend that compute more on making the model
62:01 larger? It's like a trade-off. In an ideal world, you want to do
62:05 all of them. And I think in that sense, scaling is still pretty much alive. You would
62:08 still get a better model, but like we saw with Claude Opus 4.5, it's just not worth it.
62:12 Because you can unlock more performance with other techniques at that current moment, especially if you
62:20 look at inference scaling. That's one of the biggest gains this year with o1,
62:24 where it took a smaller model further than pre-training a larger model like Claude Opus 4.5. So I wouldn't say
62:33 pre-training scaling is dead, it's just that there are other more attractive ways to scale
62:36 right now. But at some point, you will still want to make some
62:40 progress on the pre-training. The thing also to consider is where you want to spend your money. If you spend it more
62:48 on the pre-training, it's like a fixed cost. You train the model, and then it has this
62:52 capability forever. You can always use it. With inference scaling, you don't spend money during training, you spend money
63:00 later per query, and then it's also like math. How long is my model gonna be
63:04 on the market if I replace it in half a year? Maybe it's not worth
63:07 spending $5 million, $10 million, $100 million on training it longer. Maybe I will just do more
63:15 inference scaling and get performance there. It maybe costs me $2 million in terms
63:19 of user queries. It becomes a question of how many users you have and doing the
63:23 math, and I think that's also where it's interesting where ChatGPT is in a
63:26 position. I think they have a lot of users where they need to go a bit cheaper, where they have that
63:30 GPT-5 model that is a bit smaller. Other companies that have... Let's say, if your customers have other
63:37 trade-offs. For example, there was also the Math Olympiad or some
63:41 of these math problems where ChatGPT or they had a proprietary model, and I'm pretty sure it's just
63:48 like a model that has been fine-tuned a little bit more, but most of it
63:52 was during inference scaling to achieve peak performance in certain tasks.
63:56 need that all the time. But yeah, long story short, I do think all of these pre-training, mid-training, post-training,
64:03 inference scaling, they are all still things you want to do. It's just
64:07 finding—at the moment, in this year, it's finding the right ratio that gives you the best
64:11 bang for the buck, basically. - I think this might be a good place to define pre-training, mid-training, and
64:17 post-training. - So, pre-training is the classic training one next token prediction at a time.
64:21 You have a big corpus of data. And Nathan probably also has very interesting
64:26 insights there because of OLMo 3. A big portion of the paper focuses on the
64:29 right data mix. So, pre-training is essentially just, you know, training
64:33 cross entropy loss, training on next token prediction on a vast corpus of internet data, books, papers and so forth. It
64:41 has changed a little bit over the years in the sense people used to throw in
64:45 everything they can. Now, it's not just raw data. It's also synthetic data where people, let's say,
64:53 rephrase certain things. So synthetic data doesn't necessarily mean
64:56 purely AI-made data. It's also taking something from an article, a Wikipedia article, and then rephrasing it as
65:04 a Q&A question or summarizing it, rewording it, and making better data that way.
65:12 Because I think of it also like with humans. If someone, let's say, reads a book compared to a messy—no offense, but like—Reddit post
65:20 or something like that, I do think you learn— - There's going to be a post about this, Sebastian.
65:28 - Some Reddit data is very coveted and excellent for training.
65:31 You just have to filter it. - And I think that's the idea. I think it's like if someone
65:37 took that and rephrased it in a, let's say, more concise and structured way,
65:42 I think it's higher quality data that gets the LLM there faster.
65:46 You get the same LLM out of it at the end, but it trains faster because
65:50 if the grammar and the punctuation are correct, it already learns the correct way, versus getting
65:58 information from a messy source and then learning later how to correct that.
66:02 So, I think that is how pre-training evolved and why scaling still works.
66:09 It's not just about the amount of data, it's also the tricks to make that data better for you, in a sense. And
66:17 then mid-training is... I mean, it used to be called pre-training.
66:21 I think it's called mid-training because it was awkward to have pre-training and post-training but nothing in the
66:25 middle, right? It sounds a bit weird. You have pre-training and post-training, but what's the actual training?
66:29 So, the mid-training is usually similar to pre-training, but it's a bit more specialized. It's the same
66:37 algorithm, but what you do is you focus, for example, on long-context documents. The reason you
66:44 don't do that during pre-training is because you don't have that many long
66:48 context documents. We have a specific phase. And one problem of LLMs is
66:52 still that it's a neural network. It has the problem of catastrophic
66:55 forgetting. So, you teach it something, it forgets other things. And you
66:59 wanna... I mean, it's not 100% forgetting, but it's like "no free lunch."
67:03 It's the same with humans. If you ask me some math I learned 10 years
67:07 ago, I would have to look at it again. - Nathan was actually saying that he's consuming so much content that there's a catastrophic
67:13 forgetting issue. - Yeah, I'm trying to learn so much about AI, and it's like I was learning about
67:18 pre-training parallelism. I'm like, "I lost something and I don't know what it was."
67:22 - I don't want to anthropomorphize LLMs, but it's the same kind of thing
67:25 in how humans learn. I mean, quantity is not always better because you have to be selective. And mid-training is being
67:33 selective in terms of quality content at the end. So the last thing the
67:37 LLM has seen is the quality stuff. And then post-training is all
67:40 the fine-tuning, supervised fine-tuning, DPO, Reinforcement Learning with Verifiable Rewards (RLVR),
67:49 with human feedback, and so forth. So the refinement stages. And it's also
67:53 interesting, it's a cost thing. You spend a lot of money on pre-training right now.
67:57 RL a bit less. With RL, you don't really teach it knowledge. It's more like unlocking the knowledge; it's more like a skill
68:05 learning, like how to solve problems with the knowledge that it has from pre-training.
68:09 There are actually three papers this year, or last year, 2025,
68:12 on RL for pre-training. But I don't think anyone does that in production.
68:17 - Toy examples for now. - Toy examples, right? But to generalize, RL post-training is more like
68:22 the skill unlock, where pre-training is like soaking up the knowledge.
68:27 - A few things that could be helpful. A lot of people think of synthetic data as being bad for training the models. You mentioned
68:35 how DeepSeek got almost... OCR, which is Optical Character Recognition. A lot of labs did it.
68:41 Ai2 had one, Meta had multiple. And the reason each of these labs has these is because there are vast
68:48 amounts of PDFs and other digital documents on the web that aren't in formats
68:52 that are encoded with text easily. So you use these Almost-OCR,
68:56 DeepSeek OCR, or what we called our Almost-OCR, to extract trillions of
69:00 tokens of candidate data for pre-training. Pre-training dataset size is measured in trillions of tokens.
69:08 Smaller models from researchers can be something like five to 10 trillion.
69:11 researchers can be something like five to 10 trillion. Um, Qwen is documented going up to 50 trillion, and there are rumors that these closed
69:19 labs can go to 100 trillion tokens. Getting this potential data to put
69:23 in—they have a very big funnel, and the data you actually train on is
69:27 a small percentage of this. This character recognition data would be described as synthetic data for pre-training in a lab. And then there's
69:35 also the fact that ChatGPT now gives wonderful answers, and you
69:39 can train on those best answers, and that's synthetic data. It's very different than
69:42 early ChatGPT with lots of hallucination data. when people became grounded in synthetic data.
69:48 - One interesting question is, if I recall correctly, OLMo 3 was trained with less
69:52 data than specifically some other open-weight models, maybe even
69:56 OLMo 2. But you still got better performance, and that might be one example of how
70:00 the data helped. - It's mostly down to data quality. I think if we had more compute, we would train for longer. I think we'd ultimately see
70:06 that as something we would want to do. And especially with big
70:10 models, you need more compute, because we talked about having more parameters and
70:14 we talked about knowledge. Essentially, there's a ratio where big models can absorb
70:18 more from data, and then you get more benefit out of this. Any logarithmic graph in your mind is like a small model will level off
70:26 sooner if you're measuring tons of tokens, and bigger models need
70:29 more. But mostly, we aren't training that big of models right now at
70:33 AI2, and getting the highest quality data we can is the natural starting point.
70:38 - Is there something to be said about the topic of data quality? Is there some
70:42 low-hanging fruit there still where the quality could be improved?
70:46 - It's like turning the crank. Historically, in the open, there's been a
70:50 canonical best pre-training dataset that has moved around between who has the most recent one or the best recent effort. Like
70:57 AI2's Dolma was very early with the first OLMo, and Hugging Face had
71:00 FineWeb. And there's a DCLM project, which has been kind of like a,
71:04 which stands for Data Comp Language Model. There's been Data Comp
71:08 for other machine learning projects, and they had a very strong
71:12 dataset. And a lot of it is the internet is becoming fairly closed off, so we have Common Crawl, which is hundreds of trillions of tokens,
71:20 and you filter it. It looks like scientific work where you're training classifiers and making decisions based on how you
71:27 prune down this dataset into the highest quality stuff and the stuff
71:31 that suits your tasks. Previously, language models were tested a lot more
71:35 on knowledge and conversational things, but now they're expected
71:39 to do math and code. To train a reasoning model, you need to remix your whole
71:43 dataset. And there's a lot of wonderful scientific methods here where you can,
71:47 you can take your gigantic dataset, sample really tiny things from
71:51 different sources, such as GitHub, Stack Exchange, Reddit, Wikipedia. You can sample small things from them, and train small models on each of these
71:59 mixes and measure their performance on your evaluations. You can just do basic linear
72:03 regression, and it's like, "Here's your optimal dataset." But if your evaluations change, your
72:07 dataset changes a lot. So a lot of OLMo 3 was new sources for
72:11 reasoning to be better at math and code, and then you do this mixing
72:15 procedure and it gives you the answer. I think that's happened at labs this
72:19 year; there's new hot things, whether it's coding environments or web
72:22 navigation, and you need to bring in new data, change your whole pre-training so that your
72:26 post-training can work better. And that's like the constant evolution and the redetermining of what they care about for their models.
72:35 - Are there fun anecdotes of what sources of data are particularly high quality that we wouldn't expect? You mentioned Reddit
72:42 sometimes can be a source. - Reddit was very useful. I think PDFs is definitely one.
72:51 - Oh, especially arXiv. - Yeah, so AI2 has run Semantic Scholar for a long time, which is
72:58 what you can say is a competitor to Google Scholar with a lot more features.
73:01 And to do this, AI2 has found and scraped a lot of PDFs for openly accessible papers that might not be behind the
73:09 closed walled garden of a certain publisher. So, truly open scientific PDFs. And if you sit on all of these and you process
73:16 it, you can get value out of it. And I think that like, a lot of that style of work has been done by
73:22 the frontier labs did much earlier. You just need to have a pretty skilled researcher that understands how things change
73:29 models; they bring it in, clean it, and it's a lot of labor.
73:33 When frontier labs scale researchers, a lot more goes into data. If you join a frontier lab and you want to
73:41 have impact, the best way to do it is just find new data that's
73:44 better. And then, the fancy, glamorous algorithmic things like
73:49 figuring out how to make o1 is like the sexiest thought of a scientist. It's
73:52 like, "Oh, I figured out how to scale RL." There's a group that did that, but most of the
73:56 contribution is like— - On the dataset - ..."I'm gonna make the data better," or, "I'm gonna make the infrastructure better so everyone on
74:02 my team can run experiments 5% faster." - At the same time, I think it's also one of the closest guarded secrets, what your training data is,
74:08 for legal reasons. And so there's also, I think, a lot of work that goes into hiding
74:12 what your training data was essentially. Like training the model
74:16 to not give away the sources because you have legal reasons.
74:19 - The other thing, to be complete, is that some people are trying to train on only
74:22 licensed data, whereas Common Crawl is a scrape of the whole internet.
74:26 So if I host multiple websites, I'm happy to have them train language models, but I'm not explicitly
74:33 licensing what governs it. And therefore, Common Crawl is largely unlicensed, which means that your consent really hasn't been
74:41 provided for how to use the data. There's another idea where you can train language models only
74:45 on data that has been licensed explicitly, so that the kind of governing
74:49 contract is provided, and I'm not sure if Apertus is the copyright thing or the
74:53 license thing. I know that the reason that they did it was for an EU compliance thing, where they wanted to make
74:58 sure that their model fit one of those checks. - On that note, there's also the distinction in licensing. Some people
75:11 just purchase the license. Let's say they buy an Amazon Kindle book, or a Manning book, and then use that in training. That is a gray
75:19 zone 'cause you paid for the content and you might want to train on it. But then there
75:23 are also restrictions where even that shouldn't be allowed. And so that is
75:26 where it gets a bit fuzzy. And yeah, I think that is right now still a hot topic. Big companies like OpenAI
75:35 approached private companies for their proprietary data and private companies, they become more and more, let's say,
75:42 protective of their data because they know, "Okay, this is going to be my moat
75:46 in a few years." And I do think that's like the interesting question, where
75:52 if LLMs become more commoditized, and I think a lot of people learn about LLMs, there
75:56 will be a lot more people able to train LLMs. Of course, there are infrastructure challenges.
76:00 But if you think of big industries like pharmaceutical industries, law, finance industries, I do think they, at some point, will
76:07 hire people from other frontier labs to build their in-house models on their proprietary data, which will be then, again, another
76:15 unlock with pre-training that is currently not there. Because even if you
76:19 wanted to, you can't get that data. You can't get access to clinical trials
76:23 most of the time and these types of things. So, I do think scaling, in that sense,
76:27 might be still pretty much alive if you also look in domain-specific applications, because we are still right now, in this year, just looking
76:34 at general purpose LLMs on, on ChatGPT, Anthropic and so forth. They are just
76:38 general purpose, they're not even, I think, scratching the surface of what an LLM
76:42 can do if it is really specifically trained and designed for a specific task.
76:47 - I think on the data thing—this is one of the things that happened in 2025, and we totally
76:50 forget it—is Anthropic lost in court and owed $1.5 billion to
76:54 authors. Anthropic, I think, bought thousands of books and scanned them and was cleared legally for that because they bought
77:02 the books, and that is kind of going through the system. And then the other side, they also
77:06 torrented some books, and I think this torrenting was the path where the court
77:10 said that they were then culpable to pay these billions of dollars to authors, which is just
77:14 such a mind-boggling lawsuit that kind of just came and went. That is
77:18 so much money- ... from the VC ecosystem. - These are court cases that will define the future of human civilization, because it's clear
77:26 that data drives a lot of this, and there's this very complicated human
77:29 tension of... I mean, you can empathize. You're both authors.
77:34 And there's some degree to which, I mean, you put your heart and soul and
77:38 your sweat and tears into the writing that you do. It feels a little bit like theft for somebody to train your data
77:47 without giving you credit. - And there are, like Nathan said, also two layers to it. Someone might buy the book and
77:53 then train on it, which could be argued fair or not fair, but then there
77:57 are the straight-up companies who use pirated books where it's not even compensating the author. That is, I think, where people
78:04 got a bit angry about it specifically, I would say. - Yeah, but there has to be some kind of compensation scheme. This is like moving towards-
78:11 ... towards something like Spotify streaming did- ... originally for music. You know, what does that-
78:15 ... compensation look like? You have to define those kinds of models. You have to think through all of that.
78:19 One other thing I think people are generally curious about, I'd love to get your
78:23 thoughts, as LLMs are used more and more. If you look at even arXiv, but
78:28 GitHub, more and more of the data is generated by LLMs. What do you do in that kind of world? How big of a problem is that?
78:38 - Largest problem's the infrastructure and systems, but from an AI point
78:42 of view, it's kind of inevitable. - So it's basically LLM-generated data that's curated by humans essentially, right?
78:49 - Yes, and I think that a lot of open source contributors are legitimately burning
78:53 out. If you have a popular open source repo, somebody's like, "Oh, I want to do
78:56 open source AI. It's good for my career," and they just vibe-
79:00 -code something and they throw it in. You might get more of this-
79:05 - I have a- - - than I do. - Yeah, so I have actually a case study here.
79:09 I have a repository called MLxtend that I developed as a student around 10 years ago, and it is a reasonably
79:16 popular library still for certain algorithms, I think especially like frequent
79:20 data mining stuff. And there were recently two or three people who submitted a lot of PRs in a very short amount of time.
79:28 I do think LLMs have been involved in submitting these PRs. Me, as the
79:32 maintainer, there are two things. First, I'm a bit overwhelmed. I don't have time to
79:36 read through it because, especially as an older library, that is not a priority for me.
79:40 At the same time, I kind of also appreciate it because I think something
79:44 people forget is it's not just using the LLM. There's still a human layer that
79:48 verifies something, and that is in a sense also how data is labeled, right? One of the most expensive things is
79:55 getting labeled data for RL from human feedback phases. And this is kind of like that, where it goes through
80:03 phases, and then you actually get higher quality data out of it. And so I don't
80:07 mind it in a sense. It can feel overwhelming, but I do think there is also value
80:11 in it. - It feels like there's a fundamental difference between raw LLM-generated
80:15 data and LLM-generated data with a human in the loop that does some kind of verification, even if that verification is a small percent
80:24 of the lines of code. - I think this goes with anything where
80:29 people think also sometimes, "Oh, yeah. I can just use an LLM to learn
80:33 about XYZ," which is true. You can, but there might be a person who
80:37 is an expert who might have used an LLM to write specific code.
80:41 There is kind of like this human work that went into it to make it nice,
80:45 throwing out the not-so-nice parts to kind of pre-digest it for you, and that saves you time. I think that's the
80:53 value-add, where you have someone filtering things or even using the LLMs correctly. This is still labor that you
81:01 get for free. For example, if you read a Substack article, I could maybe ask an LLM to give me opinions on that, but
81:08 I wouldn't even know what to ask. And I think there is still value in
81:12 reading that article compared to me going to the LLM because you are the
81:16 expert. You select what knowledge is actually spot on, should be
81:20 included, and you give me this very... this executive summary. And this is a huge value-add because now I
81:28 don't have to waste three to five hours to go through this myself, maybe get some incorrect information and so on. And so I think
81:35 that's also where the future still is for writers even though there are LLMs that... Can kind of save you time.
81:44 - It's kinda fascinating to watch. I'm sure you guys do this,
81:47 but for me, I look at the difference between a summary and the original content. Even if it's a page-long summary of a
81:56 page-long content, it's interesting to see how the LLM-based summary takes the edge off. What
82:04 is the signal it removes from the thing? - The voice is what I talk about a lot.
82:09 - Voice? Well, voice... I would love to hear what you mean by voice, but
82:13 sometimes there's literally insights. By removing an insight, you're changing the meaning of
82:20 the thing. So I'm continuously disappointed how bad LLMs are
82:25 at really getting to the core insights, which is what a great summary does. Yet even if I have these extremely elaborate prompts where I'm
82:37 really trying to dig for the insights, it's still not quite there, which... I mean, that's a whole deep philosophical
82:45 question about what is human knowledge and wisdom and what does it mean to be
82:49 insightful and so on. But when you talk about the voice, what do you mean?
82:52 - When I write, I think a lot of what I'm trying to do is take what you think as a researcher, which is very raw. A
82:59 researcher is trying to encapsulate an idea at the frontier of their
83:03 understanding and they're trying to put what is a feeling into
83:07 words. I try to do this in my writing, which makes it come across as raw but also high-information in a way that
83:15 some people will get it and some won't. And that's the nature of research.
83:19 And language models don't do this well. They're all trained with Reinforcement Learning from Human Feedback, which takes feedback from many
83:26 people and averages how the model behaves from this. And I think it's going to be hard for a model to
83:34 be very incisive when there's that sort of filter. This is a
83:40 wonderful fundamental problem for researchers in RLHF. This provides so much utility in making the models better, but also
83:48 the problem formulation is kind of... there's this knot in it that you can't get past. These language
83:56 models don't have this prior in their deep expression they're trying to get
83:59 at. I don't think it's impossible. There are stories of models that
84:03 really shock people. Like, I think of... I would love to have
84:07 tried Bing Sydney. Did that have more voice? Because it would so often go off the rails, which is historically
84:14 obviously a scary way—like telling a reporter to leave his wife—is a crazy model
84:19 to potentially put in general adoption. But that's kind of like
84:23 a trade-off; is this RLHF process in some ways adding limitations?
84:28 - That's a terrifying place to be as one of these frontier labs and companies,
84:33 because millions of people are using them. - There was a lot of backlash last year with GPT-4o getting
84:39 removed, and I've personally never used the model, but I've talked to people at
84:43 OpenAI where they get emails from users that might be detecting subtle differences in the deployments
84:51 in the middle of the night. And they email them like, "My friend is different."
84:55 And they find these employees' emails and send them things because they are so
84:59 attached to this set of model weights and configuration that is deployed to the users.
85:05 We see this with TikTok. You open it... I don't use TikTok, but supposedly in five minutes the algorithm gets you. It's locked in.
85:12 And those are language models doing recommendations. I think
85:16 there are ways you can do this. Within five minutes of chatting with it,
85:20 the model just gets you. And that is something that people aren't really ready for. Like, don't give that to kids.
85:28 At least until we know what's happening. - But there's also going to be this mechanism... What's going to happen
85:34 with these LLMs as they're used more and more... Unfortunately, the nature of the human condition is such that people commit suicide.
85:39 And so what journalists will do is they will report extensively
85:43 on the people who commit suicide. And they would very likely
85:47 link it to the LLMs because they have that data about the conversations.
85:50 If you're really struggling in your life, if you're depressed,
85:54 if you're thinking about suicide, you're going to probably talk to LLMs about it.
85:58 And so what journalists will do is say, "Well, the suicide was committed
86:02 because of the LLM." And that's going to lead to the companies,
86:06 because of legal issues and so on, more and more taking the edge off of the LLM. So it's going to be
86:14 as generic as possible. It's so difficult to operate in this space
86:18 because you don't want an LLM to cause harm to humans at that level, but also
86:24 this is the nature of the human experience, is to have a rich conversation, a fulfilling conversation,
86:31 one that challenges you from which you grow. You need that edge.
86:35 And that's something extremely difficult for AI researchers on the RLHF front to
86:43 actually have to solve, because you're dealing with the human condition.
86:47 - A lot of researchers at these companies are so well-motivated,
86:50 and definitely Anthropic and OpenAI culturally want to do good for the world.
86:56 And it's such a—I'm like, "Ooh, I don't wanna work on this,"
87:00 because on the one hand, a lot of people see AI as a health ally,
87:04 as somebody they can talk to about their health confidentially,
87:08 but then it bleeds all the way into this, like talking about mental health, where it's
87:16 heartbreaking that this will be the thing where somebody goes over the edge,
87:19 but other people might be saved. And I'm like, "I don't..." As a researcher, it's like, I don't want to train image generation models and release
87:27 them openly because I don't want to enable somebody to have a tool on their
87:31 laptop that can harm other people. I don't have the infrastructure in my company to do that safely. But
87:38 there's a lot of areas like this where it just needs people that will
87:42 approach it with complexity and conviction. It's just such a hard problem.
87:48 - But also, we as a society, as users of these technologies, need to make sure that we're
87:51 having the complicated conversation about it versus just fearmongering—
87:55 that Big Tech is causing harm to humans or stealing your data.
87:59 It's more complicated than that. And you're right. There's a very large number of people inside these companies, many of whom I know,
88:07 who deeply care about helping people. They are considering the full human experience of people from across the world, not just Silicon
88:14 Valley. People across the United States and the world, what their needs are.
88:18 It's really difficult to design this one system that is able to help all these different kinds of people across different age groups,
88:26 cultures, and mental states. - I wish that the timing of AI was different relative to the relationship
88:35 of Big Tech to the average person. Big Tech's reputation was so low,
88:38 and with how AI is so expensive, it's inevitably going to be a Big Tech thing.
88:42 It takes so many resources, and people say the US is, "betting the economy on AI" with this build-out. To have these be intertwined
88:50 at the same time makes for such a hard communication environment.
88:54 It would be good for me to go talk to more people in the world who hate Big Tech
88:58 and see AI as a continuation of this. - And one of the things you recommend, one of the antidotes that
89:06 you talk about, is to find agency in this system. As opposed to
89:13 sitting back in a powerless way and consuming the AI slop as it rapidly takes over the internet. Find agency by using AI to build stuff,
89:24 build apps, build... One, that actually helps you build intuition,
89:28 but two, it's empowering because you can understand how it works,
89:32 what the weaknesses are. It gives your voice power to say, "This is a bad use of the technology, and this is a good use."
89:40 And you're more plugged into the system then, so you can understand it better and you can steer it better as a consumer.
89:48 - I think that's a good point you brought up about agency. Instead of ignoring it and saying,
89:52 "Okay, I'm not going to use it," I think it's probably long-term
89:56 healthier to say, "Okay, it's out there. I can't put it back."
90:00 when they came out. How do I make best use of it, and how does it help me to up-level myself? The one thing I worry about here, though,
90:08 is, if you just fully use it for something you love to do, the thing you love to do
90:12 is no longer there. And that could potentially, I feel, lead to burnout.
90:16 For example, if I use an LLM to do all my coding for me, now there's no coding. I'm just managing something that is coding for me.
90:23 Two years later, let's say, if I just do that eight hours a day,
90:27 having something code for me, do I feel fulfilled still? Is this hurting me in terms of being excited about my job, excited about what I'm doing?
90:40 Am I still proud to build something? - On that topic of enjoyment, it's quite interesting. We should just throw this in there,
90:47 that there's this recent survey of about 791 professional developers, meaning 10-plus years of experience.
90:55 - That's a long time. As a junior developer? - Yeah, in this day and age. So, there's also many fronts
91:05 that are surprising. They break it down by junior and senior developers. But, I mean, it just shows
91:12 that both junior and senior developers use AI-generated code in code they ship.
91:22 So this is not just for fun or intermediate learning things.
91:26 This is code they ship. 25%—like, most of them use around 50% or more. And what's interesting is,
91:33 for the category of over 50% of your code that you ship is AI-generated, senior developers are much more likely
91:41 to do so. But you don't want AI to take away the thing you love.
91:46 I think this speaks to my experience, these results I'm about to say.
91:49 Together, about 80% of people find it either somewhat more enjoyable or significantly more enjoyable to use AI as part of the work.
91:59 - I think it depends on the task. From my personal usage, for example,
92:03 I have a website where I sometimes tweak things. I personally don't enjoy this. So in that sense,
92:11 if the AI can help me to implement something on my website, I'm all for it.
92:15 It's great. But then, at the same time, when I solve a complex problem—
92:19 well, if there's a bug, and I hunt this bug, and I find it, it's the best feeling in the world. You feel great.
92:27 But now, if you don't even think about the bug, you just go directly to the LLM, well, you never have this kind of feeling, right?
92:35 But then there could be the middle ground where, well, you try yourself, you can't find it, you use the LLM,
92:43 and then you don't get frustrated because it helps you and you move on to something that you enjoy.
92:46 And so, looking at these statistics, I think what is not factored in
92:50 is that it's averaging over all the different scenarios. We don't know if it's for the core task
92:58 or if it's for something mundane that people would not have enjoyed otherwise.
93:02 So, in a sense, AI is really great for doing mundane things that take a lot of work.
93:06 For example, my wife the other day—she has a podcast for book discussions, a book club, and she was transferring the show notes from
93:17 Spotify to YouTube, and then the links somehow broke. And she had in some episodes, because it is so many books, like 100 links,
93:25 and it would have been really painful to go in there and fix each link manually.
93:29 So I suggested, "Hey, let's try ChatGPT." We copied the text into ChatGPT,
93:33 and it fixed them. Instead of two hours going from link to link fixing that, it made that type of
93:41 work much more seamless. I think everyone has a use case where AI is useful for something that would be
93:49 really boring, really mundane. - For me personally, since we're talking about coding, and you mentioned debugging...
93:58 the source of enjoyment for me, more on the Cursor side than Claude Code,
94:02 is that I have a friend, I have a pair programmer. It's less
94:09 lonely. You made debugging sound like this great joy. No, I would say debugging is like a drink of
94:18 water after you've been going through a desert for days. You skip the whole desert part
94:25 where you're suffering. Sometimes it's nice to have a friend who can't really find the bug, but can give you some intuition about
94:33 the code, and together you're going through the desert and finding that drink of water. At least
94:41 for me, maybe it speaks to the loneliness of the programming experience.
94:45 That is a source of joy. - It's maybe also related to delayed gratification. I'm a person who,
94:52 even as a kid, I liked the idea of Christmas presents—having them, getting them—better than actually
95:00 receiving the presents. I would look forward to the day I get the
95:04 presents, but then it's over and I'm disappointed. And maybe it's the same
95:08 with food. I think food tastes better when you're really hungry. You're right, with debugging, it is not always great. It's often
95:18 frustrating, but then if you can solve it, then it's great. But there's a sweet Goldilocks zone; if it's too hard, then
95:26 it's just wasting your time. But I think another challenge is how will people learn? We
95:34 looked at the chart and saw that more senior developers are shipping more AI-generated code than the junior ones.
95:42 It's very interesting, because intuitively you would think it's the junior developers
95:45 because they don't know how to do the thing yet, and so they use AI to do that thing. It could mean the AI is
95:53 not good enough yet to solve that task, but it could also mean
95:57 experts are more effective at using it. They know how to use it better, review the code, and then they trust the code more.
96:06 One issue for society in the future will be: how do you become an
96:10 expert if you never try to do the thing yourself? One way I always learned
96:17 is by trying things myself. If you look at math textbooks and the
96:21 solutions, you learn something, but you learn actually better if you try first. Then you appreciate the solution differently
96:29 because you know how to put it into your mental framework. If
96:33 LLMs are here all the time, would you actually go to the length
96:37 of struggling? Would you be willing to struggle? Struggle is not nice, right? But
96:43 if you use the LLM to do everything, at some point you will never really take
96:46 the next step, and then you will maybe not get that unlock that you would get as an expert using an
96:52 LLM. So, I think there's like a Goldilocks sweet spot where maybe the trick here is you make dedicated
96:59 offline time where you study two hours a day, and the rest of the day use
97:03 LLMs. But I think it's important also for people to still invest in
97:07 themselves, in my opinion, to not just LLM everything. - Yeah, as a civilization, we each individually have to find that Goldilocks zone.
97:16 And in the programming context as developers. Now, we've had this fascinating
97:20 conversation that started with pre-training and mid-training.
97:24 Let's get to post-training. A lot of fun stuff in post-training. So, what are some of the interesting ideas in post-training?
97:31 - The biggest one from 2025 is learning this reinforcement learning with verifiable rewards. You can scale up the training
97:39 there, which means doing a lot of this kind of iterative generate-grade
97:43 loop, and that lets the models learn both interesting behaviors on the tool use and software side. This could be searching, running
97:51 commands on their own and seeing the outputs, and then also that training
97:55 enables this inference time scaling very nicely. And it just
97:58 turned out that this paradigm was very nicely linked, where this
98:02 kind of RL training enables inference time scaling. But inference time scaling could have
98:06 been found in different ways. So, it was kind of this perfect storm where the models change a lot,
98:11 and the way that they're trained is a major factor in doing so. And this has changed
98:17 how people approach post-training dramatically. - Can you describe RLVR, popularized by DeepSeek R1? Can you describe how it works?
98:25 - Yeah. Fun fact: I was on the team that came up with the term
98:29 RLVR, which is from our Tulu 3 work before DeepSeek. We don't take a lot of credit for being the people to popularize the scaling
98:37 RL, but as fun as what academics get, as an aside, is the ability to name and influence
98:43 the discourse, because the closed labs can only say so much. That
98:47 one of the things you can do as an academic is you might not have the compute to train the
98:51 model, but you can frame things in a way that ends up being described as a community coming together around this RLVR term, which
98:59 is very fun. And then DeepSeek are the people that did the training breakthrough,
99:03 which is, they scaled the reinforcement learning. You have the model generate answers and then grade the completion if it
99:11 was right, and then that accuracy is your reward for reinforcement learning. So reinforcement learning is
99:18 classically an agent that acts in an environment, and the environment gives
99:22 it a state and a reward back, and you try to maximize this reward. In the case of language models, the reward is normally
99:29 accuracy on a set of verifiable tasks, whether it's math problems or coding tasks. And it starts to get blurry with things like
99:38 factual domains. That is also, in some ways, verifiable, or constraints on your instruction, like respond only with
99:45 words that start with A." All of these things are verifiable in
99:49 some way, and the core idea of this is you find a lot more of these problems that are verifiable and you let the model try it
99:57 many times while taking these RL steps, these RL gradient updates. The infrastructure evolved from
100:05 reinforcement learning from human feedback, where in that era the score
100:09 they were trying to optimize was a learned reward model of human
100:12 preferences. So you kind of changed the problem domains and that
100:16 let the optimization go on to much bigger scales, which kind of
100:20 kickstarted a major change in what the models can do and how people use them.
100:24 - What kind of domains is RLVR amenable to? - Math and code are the famous ones, and then there's a lot of
100:32 work on what is called the rubrics, which is related to a word people might
100:36 have heard: LLM-as-a-judge. For each problem, I'll have a set of problems in my dataset. I will then have an LLM
100:44 and ask it, "What would a good answer to this problem look like?" And then you could try the problem over and over again and
100:52 assign a score based on this rubric. That's not necessarily verifiable like
100:56 math and code domains, but this rubrics idea and other scientific problems that might be a little bit more vague is where
101:03 the attention is, where they're trying to push this set of methods into
101:07 these kind of more open-ended domains so the models can learn a lot more.
101:11 - I think that's called reinforcement learning with AI feedback, right?
101:14 - That's the older term for it coined in Anthropic's Constitutional
101:17 AI paper. It's like a lot of these things come in cycles. - Also, just one step back for RLVR. I think the
101:25 interesting thing here is that you ask the LLM a, let's say,
101:29 math question, and then you know the correct answer, and you let the LLM,
101:33 as you said, figure it out. How it does it—you don't constrain it
101:37 much. There are some constraints like "use the same language, don't switch
101:41 between Spanish and English." But let's say you're pretty much hands-off.
101:45 You only give the question and the answer, and then the LLM has
101:49 the task to arrive at the right answer, but the beautiful thing here is what happens in practice: the LLM will
101:56 do a step-by-step description, like as a student or as a mathematician would derive the solution. It will use those steps,
102:04 and that helps the model to improve its own accuracy. And then, like you said, the inference scaling.
102:11 Inference scaling loosely means spending more compute during inference, and here the inference scaling is
102:20 that the model would use more tokens. In the DeepSeek R1 paper, they
102:24 showed the longer they train the model, the longer the responses are.
102:28 They grow over time. They use more tokens, so it becomes more expensive.
102:32 It becomes expensive for simple tasks, but these explanations help accuracy.
102:36 There are also papers showing what the model explains does not necessarily have to be correct or maybe it's unrelated to the
102:44 answer, but for some reason, it still helps the model that it
102:47 is explaining. And I think it's also—again, I don't want to anthropomorphize these LLMs, but it's kind of like how we humans operate.
102:55 If there's a complex math problem in a math class, class, you usually have a note paper and you do it step by step. You cross out
103:02 things. And the model also self-corrects, and that was, I think, the
103:06 aha moment in the DeepSeek R1 paper. They called it the aha moment because the model itself
103:10 recognized it made a mistake and then said, "Ah, I did something wrong, let me try again."
103:14 And I think that's just so cool that this falls out of just giving it the correct answer and having it figure out how to do
103:22 it, that it kind of does in a sense what a human would do. Although LLMs don't think like humans, it's kind of like an interesting coincidence
103:30 and it... And the other nice side effect is it's great for us humans often to see these steps. It builds trust, but also we
103:38 us humans to see these steps. It builds trust, but also we learn and can double check things.
103:40 - There's a lot in here. I think some of the debate... There's been a lot of debate this year on if the language models
103:44 like these... I think the aha moments are kind of fake because in pre-training
103:49 you essentially have seen the whole internet. so you have definitely seen people explaining their work, even verbally, like
103:55 a transcript of a math lecture. "You try this, oh, I messed this up." And what RLVR
103:59 is very good at doing is amplifying these behaviors because they're very useful in enabling the model to think longer and to
104:06 check its work. And I agree that it is very beautiful that this
104:10 training kind of... The model learns to amplify this in a way that is so useful for the
104:14 final answers being better. - I can give you also a hands-on example. I was training the Qwen 3 base model with
104:20 RLVR on MATH-500. The base model had an accuracy of about 15%. Just 50 steps,
104:28 like in a few minutes with RLVR, the model went from 15% to 50% accuracy. And the model... You can't tell me
104:35 it's learning anything fundamentally about math in - The Qwen example is weird because there've been two papers this year, one of which I was on, about
104:42 data contamination in Qwen and specifically that they train on a lot of this special mid-training phase
104:48 that we should take a minute on, because it's weird because they train on problems that are almost identical to MATH.
104:53 - Exactly. And so you can see that basically the RL, it's not teaching the model any new knowledge about math. You can't do that in 50 steps.
105:01 So the knowledge is already there, in the pre-training, you're just unlocking it.
105:03 - I still disagree with the premise because there's a lot of weird complexities that you
105:07 can't prove because one of the things that points to weirdness is that if you take the Qwen 3 so-called base
105:14 model and you... You could Google like "math dataset, Hugging Face", and you could take a problem and what you do if you put it
105:22 into Qwen 3 base... All these math problems have words, so it'd be like "Alice
105:26 has five apples and takes one... and gives three to whoever," and there are these
105:30 word problems. With these Qwen-based models, why people are suspicious of them
105:34 is if you change the numbers but keep the words- Qwen will produce, without tools, will produce a
105:40 very high accuracy decimal representation of the answer, which means there's some... At some time, it was shown
105:47 problems that were almost identical to the test set, and it was using
105:51 tools to get a very high precision answer, but a language model without tools
105:55 will never actually have this. So it's kind of been this big debate in the research community: how much of these reinforcement learning
106:03 papers that are training on Qwen and measuring specifically on this
106:06 math benchmark, where there's been multiple papers talking about contamination, is like,
106:10 how much can you believe them? And I think this is what caused the reputation of RLVR
106:14 being about formatting, because you can get these gains so quickly,
106:18 therefore it must already be in the model. But there's a lot of complexity here that we...
106:22 It's not really like controlled experimentation, so we don't really know.
106:26 - But if it weren't true, I would say distillation wouldn't work, right?
106:30 I mean, distillation can work to some extent, but the thing is that
106:34 is, I think, the biggest problem, and I research this contamination because we don't know what's in the
106:38 data. Unless you have a new dataset, it is really impossible. And the
106:42 same, you mentioned the math dataset, where you have a question and
106:46 then answer and an explanation is given, but then also even something simpler
106:50 like MMLU, which is a multiple-choice benchmark. If you just
106:54 change the format slightly, like, I don't know, if you use a dot instead of a parenthesis or something like that, the
107:02 model accuracy will vastly differ. - I think that that could be like a model issue rather than a general issue.
107:09 - It's not even malicious by the developers of the LLM, like, "Hey, we want to cheat at that benchmark."
107:13 It has seen something at some point. I think the only fair way to evaluate an LLM
107:17 is to have a new benchmark that is after the cutoff date when the LLM was deployed.
107:22 - Can we lay out what would be the recipe of all the things that go into post-training? And you mentioned RLVR was
107:30 a really exciting, effective thing. Maybe we should elaborate. RLHF still has a really important
107:36 component to play. What kind of other ideas are there on post-training?
107:40 - I think you can kind of take this in order. I think you could
107:43 view it as what made o1, which is this first reasoning model,
107:47 possible, or what will the latest model be? And they actually... You're going to have similar interventions at these,
107:54 where you start with mid-training, and the thing that is rumored to enable o1 and similar models is really careful data
108:02 curation, where you're providing a broad set of what is called reasoning traces, which is just the model generating
108:09 words in a forward process that is reflecting, like breaking down a problem into intermediate steps and trying to solve them. So at mid-training,
108:17 you need to have data that is similar to this to make it so that when you move into post-training,
108:23 primarily with these verifiable rewards, it can learn. And then what is happening today is you're figuring
108:30 out which problems to give the model and how out which problems to give the model and how long you can train it for and how much inference you can
108:38 enable the model to use when solving these verifiable problems. So as
108:42 models get better, certain problems models get better, certain problems are no longer... The model will solve them 100% of the time, and
108:49 therefore there's very little signal in this. If we look at the GRPO
108:52 equation, this one is famous for this because essentially the
108:56 reward given to the agent is based on how good a given action—an action is a completion—is relative to the other
109:04 answers to that same problem. So if all the problems get the same answer, there's no signal in these
109:08 types of algorithms. So what they're doing is they're finding harder problems, which is
109:12 why you hear about things like scientific domains, where it's so
109:16 hard to get anything right. If you have a lab or something, it just
109:19 generates so many tokens or much harder software problems. So the
109:23 frontier models are all pushing into these harder domains when they can train on more
109:27 problems and the model will learn more skills at once. The RLHF
109:31 link to this is that RLHF has been and still is kind of like the
109:35 finishing touch on the models, where it makes the models more useful
109:39 by improving the organization or style or tone. There are different things that
109:42 resonate with different audiences, like some people like a really quirky model and RLHF could
109:46 be good at enabling that personality, and some people hate this markdown bulleted list thing that the models do,
109:54 but it's actually really good for quickly parsing information. In RLHF, this
109:58 human feedback stage is really great for putting this into the model at the end of the day. It's what made ChatGPT
110:06 so magical for people. And that use has actually remained fairly
110:09 stable. This formatting can also help the models get better at math problems, for example. So it's
110:17 like the border between style and formatting, and like the method that you use to answer a problem is
110:24 actually all very closely linked in terms of when you're training these models, which is why RLHF can still make a model better at
110:32 math, but these verifiable domains are a much more direct process to
110:36 doing this because it makes more sense with the problem formulation, which is why it ends up all forming together. But to summarize, it's like
110:43 mid-training is give the model the skills it needs to then learn. RL with verifiable rewards is letting the model try
110:51 a lot of times, so put a lot of compute into trial-and-error learning across hard
110:54 problems. And then RLHF would be like finishing the model, making it easy to
110:59 use and kind of just rounding the model out. - Can you comment on the amount of compute required for RLVR?
111:06 - It's only gone up and up. I think Ilya Sutskever was famous for saying they use a similar amount of compute for pre-training and
111:12 post-training. Back to the scaling discussion, they involve very
111:16 different hardware for scaling. Pre-training is very compute-bound, which is like this
111:20 FLOPs discussion, which is just how many matrix multiplications can you get through at once.
111:24 And because with RL you're generating these answers, you're trying the
111:28 model in real-world environments, it ends up being much more memory-bound because
111:32 you're generating long sequences and the attention mechanisms have
111:35 this behavior where you get a quadratic increase in memory as you're getting to longer sequences. So the compute becomes very different.
111:44 In pre-training we would talk about a model—if we go back to like the Biden
111:47 administration executive order, it's like 10 to the 25th FLOPs to train a
111:51 model. If you're using FLOPs in post-training, it's a lot weirder because the
111:55 reality is just like: how many hours are you allocating? How many GPUs
111:59 for? And I think in terms of time, the RL compute is getting much closer because you just can't put it all into one system.
112:06 Pre-training is so computationally dense where all the GPUs are talking to each other and
112:10 it's extremely efficient, whereas RL has all these moving parts and can take a long time
112:15 to generate a sequence of 100,000 tokens. If you think about GPT-5.2 Pro taking an hour, it's like, what if your training run
112:22 has a sample for an hour and you have to make sure that's handled efficiently?
112:25 So I think in GPU hours or just wall-clock hours, the RL runs are probably approaching the same number of days as pre-training,
112:32 but they probably aren't using as many GPUs at the same time.
112:36 There are rules of thumb where in labs you don't want your pre-training
112:40 runs to last more than a month because they fail catastrophically. And if you
112:44 are planning a huge cluster to be held for two months and then it
112:48 fails on day 50, the opportunity costs are just so big. So people don't want to put all their eggs in one basket.
112:56 GPT-4 was the ultimate YOLO run, and nobody ever wanted to do it before, where it took three months to train and everybody was shocked that it
113:04 worked. I think people are a little bit more cautious and incremental now.
113:07 - So RLVR is more, let's say, unlimited in how much you can train or still get benefit, where RLHF, because it's a preference
113:15 tuning, you reach a certain point where it doesn't really make sense to spend more
113:19 RL budget on that. So just a step back with preference tuning:
113:23 there are multiple people that can give multiple, let's say,
113:26 explanations for the same thing and they can both be correct, but at some point you learn a certain
113:30 style and it doesn't make sense to iterate on it. My favorite example is: if relatives ask me what laptop they should buy,
113:38 I give them an explanation or ask, "What is your use case?" They, for example, prioritize battery life and
113:46 storage. Other people like us, for example, we would prioritize RAM and
113:50 compute. Both answers are correct, but different people require different answers. With preference tuning, you're trying to average somehow.
113:58 You are asking the data labelers to give you, not the right,
114:02 but the preferred answer and then you train on that. But at some point you learn
114:06 that average preferred answer. And there's no reason to keep
114:10 training longer on it because it's just a style, whereas with RLVR,
114:14 you let the model solve more and more complex, difficult problems. So I think it makes more sense to allocate
114:22 more budget long-term to RLVR. Also, right now we are in an RLVR 1.0
114:30 blend where it's still that simple thing where we have a question and
114:34 answer, but we don't do anything with the stuff in between. There were multiple research papers, also by Google for example,
114:42 on process reward models that also give scores for the explanation—how correct is the explanation. And I think that will be the
114:50 next thing, let's say RLVR 2.0 for this year, focusing in between question and answer, like how to leverage that
114:57 information, the explanation, to help it get better accuracy. So that's one angle. And there was a DeepSeek
115:05 Math-V2 paper where they also had interesting inference scaling there where, first, they had
115:12 developed models that grade themselves, a separate model. And I think that will be one aspect. And the other, like Nathan mentioned, it will
115:20 be for RLVR branching into other domains. - The place where people are excited are value functions,
115:26 which is pretty similar. So process reward models are kind of like...
115:30 Process reward models assign how good something is at each intermediate step in a reasoning
115:37 process, where value functions apply value to every token the language
115:40 model generates. Both of these have been largely unproven in the language
115:46 modeling and reasoning model era. People are more optimistic about
115:50 value functions for whatever reason now. I think process reward
115:54 models were tried a lot more in this pre-o1, pre-reasoning model era, and a lot of people had a lot of headaches with them. So I think a lot of it is human
116:02 nature... Value models have a very deep history in reinforcement learning. They're one of the
116:07 first things core to deep reinforcement learning existing, is
116:11 training value models. So right now people are excited about trying value models, but there's very little proof. And there are negative
116:19 examples in trying to scale up process reward models. These things don't always hold in the future. We came to this
116:25 discussion by talking about scaling. The simple way to summarize what you're
116:29 saying is you don't want to do too much RLHF, where the signal doesn't scale.
116:33 People have worked on RLHF for language models for years, especially with intense
116:37 interest after ChatGPT. And the first release of a reasoning model trained with RLVR, OpenAI's o1, had a scaling
116:45 plot where if you increase training compute logarithmically, you get a linear
116:49 increase in evaluations. This has been reproduced multiple times. DeepSeek
116:53 had a plot like this. But there's no scaling law for RLHF where if you
116:57 log-increase the compute, you get performance. In fact, the seminal scaling
117:01 paper for RLHF is scaling laws for reward model over-optimization.
117:05 So that's a big line to draw with RLVR and the methods we have now. In the future, they will follow this scaling
117:13 paradigm: where you can let the best runs run for an extra 10x and you get performance, but you can't do this with
117:20 RLHF. And that is just going to be field-defining in how people approach
117:24 them. While I'm a shill for people to academically do RLHF, to do the best RLHF you might not
117:32 need the extra 10 or 100x of compute, but to do the best RLVR you do. I think there's a seminal paper
117:41 from a Meta internship. It's called something like "The Art of
117:45 Scaling Reinforcement Learning with Language Models." What they describe as a
117:48 framework is Scale-RL. Their incremental experiment was like 10,000 V100 hours, which is like thousands or
117:56 tens of thousands of dollars per experiment. They do a lot of them, and
118:00 This cost is not accessible to the average academic, which is a hard equilibrium where
118:07 it's trying to figure out how to learn from each community. - I was wondering if we could take a bit of a tangent and talk
118:15 about education and learning. If you're someone listening to this who's a smart person interested in programming
118:22 and AI, I presume building something from scratch is a good beginning.
118:28 So can you take me through what you would recommend people do?
118:32 - I would personally start, like you said, implementing a simple model from
118:36 scratch that you can run on your computer. The goal is not, when you build a model from scratch, to have something for every day use.
118:43 It's not going to be your personal assistant replacing an existing open-weight model or ChatGPT. It's to see what exactly
118:50 goes into the LLM, what comes out, and how the pre-training works
118:54 on your own computer, preferably. Then you learn about pre-training, supervised fine-tuning, and the
119:02 attention mechanism. You get a solid understanding of how things work, but
119:06 at some point you reach a limit, because small models can only do so much.
119:11 The problem with learning about LLMs at scale is that it's exponentially more complex to make a larger model,
119:19 because the model isn't just larger—you have to shard your parameters across multiple GPUs. Even for the KV cache, there are multiple
119:26 ways to implement it. One is just to understand how it works, just to grow the
119:30 cache. You grow it step-by-step by, let's say, concatenating
119:35 lists, but then that wouldn't be optimal on GPUs. You would pre-allocate a tensor and then fill it in.
119:42 But that adds another 20 or 30 lines of code. And for each thing,
119:46 you add so much code. The goal with the book is basically to understand
119:50 how the LLM works. It's not going to be a production-level LLM, but once you have
119:54 that, you can understand the production-level LLM. - So you're trying to always build an LLM that's going to fit on one GPU?
120:00 - Yes. Most of them do. I have some bonus materials on some MoE models. One or two of them may require multiple GPUs,
120:08 but the goal is to have it on one GPU. And the beautiful thing is,
120:12 you can self-verify. It's almost like RLVR. When you code these from
120:16 scratch, you can take an existing model from the Hugging Face
120:20 Transformers library. The library is great, but if you want to learn about LLMs, it's not the best place to start because the
120:28 code is so complex to fit so many use cases. Because people use it in production, it has to be really sophisticated,
120:36 really intertwined, and hard to read. It's not linear. - It started as a fine-tuning library, and then it grew to be the
120:43 standard representation of every model architecture. Hugging Face is the default place to get a model, and Transformers is the software.
120:49 It enables it so people can easily load a model and do something basic with it.
120:56 - And all frontier labs that have open-weight models have a Transformers
121:00 version of it, like from DeepSeek to gpt-oss-120b. That's the
121:03 canonical weight format you can load. But even even Transformers, the library, is not used in production. People use
121:11 SGLang or vLLM, and it adds another layer of complexity. - We should say that the Transformers library has like 400 models.
121:19 - So it's the one library that tries to implement a lot of LLMs, and
121:23 so you have a huge codebase, basically. It's huge. It's like, I don't know, maybe millions—
121:30 - That's crazy. - hundreds of thousands of lines of code. Understanding the part
121:34 you want to understand is finding the needle in the haystack. But what's beautiful is
121:38 you have a working implementation, so you can work backwards. What I would
121:42 recommend doing, or what I also do, is if I want to understand, for example, how OLMo
121:46 is implemented, I would look at the weights in the model hub, the config file,
121:50 and then you can see, "Oh, they used so many layers. They use, let's say, Group Query Attention or Multi-Head Attention in that case."
121:58 Then you see all the components in a human-readable, 100-line config file.
122:02 And then you start, let's say, with your GPT-2 model and add these things.
122:05 The cool thing here is you can then load the pretrained weights
122:09 and see if they work in your model. You want to match the same output that you get with a Transformer model, and then you can use that,
122:17 basically as a verifiable reward to make your architecture correct.
122:21 Sometimes it takes me a day. With OLMo 3, the challenge was RoPE
122:25 for the position embeddings. They had a YaRN extension and there was some custom scaling there,
122:32 and I couldn't quite match these things. In this struggle, you kind of understand things. At the end, you know you have it correct
122:40 because you can unit test it. You can check against the reference
122:44 implementation. I think that's one of the best ways to learn, really.
122:48 To basically reverse-engineer something. - I think that is something everyone interested in getting into AI
122:55 today should do. That's why I liked your book. I came to language models from the RL and robotics field. I had never taken the
123:03 time to just learn all the fundamentals. This transformer architecture is so fundamental, just as deep
123:10 learning was in the past, and people need to do this. I think where a lot of people get overwhelmed
123:18 is, "How do I apply this to have impact or find a career path?"
123:22 Because language models make this fundamental stuff so accessible,
123:26 and people with motivation will learn it. Then it's like, "How do I get cycles on goal to contribute to research?"
123:34 I'm actually fairly optimistic because the field moves so fast
123:37 that a lot of times the best people don't fully solve a problem
123:41 because there's a bigger problem to solve that's very low-hanging fruit,
123:45 so they move on. I think that a lot of what I was trying to do in this
123:49 RLHF book is take post-training techniques and describe how people think about them influencing the model and what people are doing.
123:57 Then it's remarkable how many things I just think people stop studying or don't pursue.
124:05 I think people trying to go narrow after doing the fundamentals is good,
124:08 and then reading the relevant papers and being engaged in the ecosystem. It's like you actually...
124:16 actually... The proximity that random people online have to the leading researchers—no one knows who all the...
124:24 The anonymous accounts on X and ML are very popular, and no one knows who
124:27 all these people are. It could just be random people that study this stuff deeply, especially with
124:31 the AI tools. To just be like, "I don't understand this, keep digging into it," is a
124:35 very useful thing. But there's a lot of research areas that maybe have three papers that you need to read, and then
124:43 one of the authors will probably email you back. But you have to put in a lot
124:47 of effort into these emails to understand the field. I think it would
124:51 be for a newcomer easily weeks of work to feel like they can
124:54 truly grasp what is a very narrow area, but I think going narrow after you have the fundamentals will be very useful to people because
125:02 I've become very interested in character training, which is how you
125:07 make the model funny or sarcastic or serious, and what do you do to the data to do this? A student
125:15 at Oxford reached out to me and was like, "Hey, I'm interested in this," and I advised him.
125:18 And that paper now exists. There's like two or three people in the world that were very interested in this. He's a PhD student, which
125:26 gives him an advantage, but for me, that was a topic I was waiting for someone to be like, "Hey, I have time to spend
125:32 cycles on this." I'm sure there's a lot more very narrow things where you're just like,
125:36 "It doesn't make sense that there was no answer to this." I think it's just
125:40 there's so much information coming that people are like, "I can't grab onto any of
125:44 these," but if you just stick in an area, I think there's a lot of interesting things to learn.
125:48 - Yeah, I think you can't try to do it all because it would be very overwhelming and
125:52 you would burn out. For me, for example, I haven't kept up with
125:56 computer vision in a long time; I just focused on LLMs. But coming back to your book,
126:00 I think this is a really great book and a really good bang for the buck because
126:04 if you want to learn about RLHF, I wouldn't go out there and read RLHF
126:08 papers because you would be spending two years— - Some of them contradict.
126:11 I've just edited the book, and there's no chapter where I had to be like,
126:15 "X papers say one thing and Y papers say another, and we'll see what
126:19 comes out to be true." - Just to go through the table of contents, what are some ideas we might have missed in the
126:25 bigger picture of post-training? First of all, you did the problem setup, training
126:29 overview, what are preferences, preference data, and the optimization
126:33 tools, reward modeling, regularization, instruction tuning, rejection sampling, and reinforcement learning. Then,
126:43 Constitutional AI and AI feedback, reasoning, and inference-time
126:47 scaling to use in function calling, synthetic data and distillation,
126:51 evaluation, and then an open questions section, over-optimization, style and information, and then product
126:58 UX, character and post-training. What are some ideas worth mentioning that connect both the educational and the research components?
127:06 You mentioned character training, which is pretty interesting.
127:08 - Character training is interesting because there's so little on it. We talked about how people
127:12 engage with these models. We feel good using them because they're positive, but that can go too far; it can be too positive. And it's like,
127:19 essentially, it's: How do you change your data and decision-making to make it exactly what you want? And like,
127:27 OpenAI has this thing called a model spec, which is essentially their
127:30 internal guideline for what they want the model to do, and they publish this to
127:34 developers. So, essentially, you can know what is a failure of
127:38 OpenAI's training—where they have the intentions and they haven't met them yet—
127:42 versus what is something that they actually wanted to do and that you don't like.
127:46 And that transparency is very nice, but all the methods for curating these
127:49 documents and how easy it is to follow them is not very well known. I think the
127:53 way the book is designed is that the RL chapter is obviously what people want because
127:57 everybody hears about it with RLVR, and it's the same algorithms and the same
128:01 math, but you can use it in very different documents. So I think the core of RLHF is like how messy
128:09 preferences are. It's essentially a rehash of a paper I wrote years ago, but this is essentially the chapter that'll tell you why
128:16 RLHF is never ever fully solvable because, the way that even RL is set up, it assumes that preferences can be
128:28 quantified and that multiple preferences can be reduced to single values. And I think it relates in the economics literature to the
128:36 Von Neumann-Morgenstern utility theorem, and that is the chapter where all
128:40 of that philosophical, economic, and psychological context tells you what gets compressed into doing RLHF. So it's like you
128:48 have all of this and then later in the book it's like: You use this RL map to make the
128:52 number go up. And I think that's why it'll be very rewarding for people to do
128:56 research on, because quantifying preferences is something that humans have designed a problem in order to
129:03 make preferences studyable. But there's kind of fundamental debates, like,
129:08 an example is in a language model response you have different things you care about, like
129:12 accuracy or style. And when you're collecting the data, they all get compressed
129:15 into: "I like this more than another." And that is happening, and there's a lot of research in other areas of the
129:23 world that go into how you should actually do this. I think social choice theory is the
129:29 subfield of economics around how you should aggregate preferences.
129:33 And I went to a workshop that published a white paper on: "How can you think about using social choice theory for
129:40 RLHF?" So I mostly would want people that get excited about the math to
129:44 come and find things where they could stumble into this broader
129:48 context. I think there's a fun thing: I just keep a list of all the tech
129:52 reports of reasoning models I like. So in Chapter 14, where there's a short summary of RLVR, there's just a gigantic table where I
129:59 list every single reasoning model that I like. I think in education, a lot of it needs to be like, at this point, what I like,
130:08 because the language models are so good at the math. For example, the
130:12 famous paper, Direct Preference Optimization, which is a much simpler way of solving
130:16 the problem than RL. The derivations in the appendix skip steps
130:20 of math. And for this book, I redid the derivations and I'm like,
130:24 "What the heck is this log trick that they use to change the math?"
130:27 But doing it with language models, they're like, "This is the log trick." And I'm
130:31 like, "I don't know if I like this, that the math is so commoditized."
130:34 I think some of the struggle in reading this appendix- ...and following the math is good for learning.
130:44 - Yeah, we're returning to this often on the topic of education.
130:48 You both have brought up the word "struggle" quite a bit. So there is value. If you're not struggling as part of this process,
130:55 you're not fully following the proper process for learning. proper process for learning, I suppose.
131:02 - Some providers are working on models for education designed to not give-
131:06 actually, I haven't used them, but I'd guess they're designed to not give
131:10 all the information at once. And make people work for it. Training models to do this would be a wonderful contribution.
131:16 Where, like all of the stuff in the book, you had to reevaluate every decision.
131:19 decision for it- It's a great example. There's a chance we work on it at Ai2,
131:24 which I thought would be so fun. - It makes sense. I did something like that the other day for video games.
131:30 Sometimes for pastime I play video games, like I like- Video games with puzzles, like Zelda and Metroid. And there's this
131:38 new game where I really got stuck and was okay with it. I don't want to struggle for two days, so I used an LLM.
131:45 But then you say, "Hey, please don't add spoilers. Just, you know, I'm here and there.
131:49 What do I have to do next?" You can do the same thing for math where you say,
131:53 "Okay, I'm stuck at this point. Don't give me the full solution,
131:57 but what is something I could try?" Where you carefully probe it.
132:01 But the problem here is I think it requires discipline. Many people enjoy math, but there are also a lot of people
132:09 who need to do it for their homework, and then it's like a shortcut.
132:13 We could develop an educational LLM, but other LLMs are still there,
132:17 and there's still a temptation to use the other LLMs. - I think many people in college understand the stuff they're passionate about-
132:24 about- ...they're self-aware and they understand it shouldn't be easy.
132:27 I think we just have to develop a good taste- ...talk about research taste, school taste about
132:35 stuff that you should be struggling on- ...and stuff you shouldn't be. It's tricky, because you don't have good long-term vision
132:40 sometimes you don't have good long-term vision about what would be actually useful to you in your career.
132:48 But you have to develop that taste, yeah. - I was talking to my fiancee or friends about this,
132:56 there's this brief 10-year window where all of the homework and all the
132:59 exams could be digital. Before that, everybody had to do all the exams in blue books
133:03 because there was no other way. And now after AI, everyone's going to need to be in blue books and
133:07 oral exams because everyone could cheat so easily. It's like this brief generation that had a
133:11 different education system where everything could be digital, but you still couldn't cheat. And now it's just going back.
133:19 It's just very funny. - You mention character training. Just zooming out on a more general
133:24 topic, for that project how much compute was required? And in general, to contribute as a
133:30 researcher, are there places where not too much compute is required where you can actually contribute as an individual
133:38 researcher? - For the character training thing, I think this research is built on fine-tuning
133:43 about 7 billion parameter models with LoRA, which is essentially only fine-tuning a small subset of the weights of the model.
133:51 I don't know exactly how many GPU hours that would take. - But it's doable.
133:56 - Not doable for every academic. The situation for some academics is so dire
134:00 that the only work you can do is doing inference where you have closed models or open
134:04 models and you get completions from them and you can look at them and understand the
134:07 models. And that's very well-suited to evaluation, where you want to
134:11 be the best at creating representative problems that the models fail on or show certain abilities, which I think
134:18 that you can break through with this. I think that the top-end goal for a researcher working on evaluation, if you
134:26 want to have career momentum, is that Frontier Labs pick up your
134:30 evaluation. You don't need to have every project do this. But if you
134:33 go from a small university with no compute and find something that Claude struggles with,
134:38 and then the next Claude model has it in the blog post, there's your
134:42 career rocket ship. I think that's hard, but if you want to scope the maximum possible impact with minimum compute, it's something
134:49 like that, which is just get very narrow and it takes learning of where the
134:53 models are going. So you need to build a tool that tests where Claude 4.5 will fail. If I'm going to start
135:01 a research project, I need to think where the models in eight months are going to be struggling.
135:05 - But what about developing totally novel ideas? - This is a trade-off. I think that if you're doing a PhD, you could also be like,
135:13 "It's too risky to work in language models. I'm going way longer term,"
135:17 which is like what is— what is the thing that's going to define language model development in 10 years?
135:22 I end up being a person that's pretty practical. I mean, I went to
135:26 my PhD where it was like, "I got into Berkeley. Worst case, I get a master's, and then I go
135:30 work in tech." I'm very practical about it, so I'm like the life afforded to people to work at these AI companies, the amount of...
135:38 OpenAI's average compensation is over a million dollars in stock a year per
135:42 employee. For any normal person in the US, to get into this AI lab is transformative for your life. So I'm pretty practical about it.
135:50 there's still a lot of upward mobility working in language models if you're
135:54 focused. And look at these jobs. But from a research perspective,
135:57 the transformative impact in these academic awards... to be the next Yann LeCun is from not working on—
136:04 not caring about language model development very much. - It's a big financial sacrifice in that case.
136:09 - So I work with some awesome students, and they're like, "Should I go work at an
136:13 AI lab?" And I'm like, "You're getting a PhD at a top school. Are you gonna
136:17 leave to go to a lab?" I don't know. If you go work at a top lab,
136:21 I don't blame you. Don't go work at some random startup that might go to zero. But if you're going to
136:25 OpenAI, I'm like, "It could be worth leaving a PhD for." - Let's more rigorously think through this. So where would you
136:34 give a recommendation for people to do a research contribution? So the options
136:37 are academia: get a PhD. Spend five years publishing. Compute resources are constrained. There's—
136:46 there's research labs that are more focused on open-weight models,
136:54 and working there. Or closed frontier research labs. So OpenAI, Anthropic, xAI, and so on.
137:04 - The two gradients are: the more closed, the more money you tend to get,
137:08 but you also get less credit. In terms of building a portfolio of things that you've done, it's very clear
137:19 what you have done as an academic. Versus if you are going to
137:22 trade this fairly reasonable progression for being a cog in the machine, which could also be very fun.
137:30 So I think it's very different career paths. But the opportunity cost for being a researcher is very high because PhD students are
137:37 paid essentially nothing. So it ends up rewarding people that have a
137:41 fairly stable safety net, and they realize that they can operate in the
137:45 long term, wanting to do very interesting work and get a very interesting job.
137:49 So it is a privileged position to be like, "I'm gonna see out my PhD and figure it out after because I want to do this."
137:59 At the same time, the academic ecosystem is getting bombarded by funding
138:03 getting cut and stuff. So there's just so many different trade-offs where I
138:06 understand plenty of people that are like, "I don't enjoy it. I can't deal with this
138:10 funding search. My grant got cut for no reason by the government,"
138:14 or, "I don't know what's gonna happen." So I think there's a lot of
138:18 uncertainty and trade-offs that, in my opinion, favor just taking the well-paying job with meaningful impact. It's not like
138:25 you're getting paid to sit around at OpenAI. You're building the cutting edge of things
138:29 that are— changing millions of people's relationship to tech.
138:34 - But publication-wise, they're being more secretive, increasingly so.
138:38 So you're publishing less and less. And so you are having a positive impact at scale, but you're a cog in the machine.
138:47 - I think it honestly hasn't changed that much. I have been in academia. I'm not in academia anymore.
138:55 wouldn't want to miss my time in academia. But what I wanted to say before I get to that
138:59 is that I think it hasn't changed that much. I was working in
139:03 computational biology, using AI or machine learning methods with collaborators, and a lot of people went from academia directly to
139:15 Google. And I think it's the same. Back then, professors were
139:19 sad that their students went into industry because they couldn't carry on their legacy. I think it's the same.
139:26 It hasn't changed that much. The only thing that has changed is the scale. Cool stuff was always developed in
139:34 industry that was closed. You couldn't talk about it. And I think the difference now is your preference.
139:42 Do you like to publish your work, or are you more in a closed lab? That's one difference. The compensation, of course,
139:50 is another, but it's always been like that. It depends on where you feel comfortable. And nothing is
139:58 forever. Right now, there's a third option, which is launching a startup. A lot of people are doing that.
140:05 It's a very risky move, but it can be a high-risk, high-reward situation, whereas joining an industry lab
140:13 is pretty safe. You also have upward mobility. I think once you've been at an industry lab, it's easier to find
140:21 future jobs. But then again, how much do you enjoy the team and working on proprietary things versus how much you like
140:32 publishing work? I mean, publishing is stressful. Acceptance rates at conferences can be arbitrary
140:39 and very frustrating, but it's high reward if you have a paper
140:43 published. You feel good because your name is on there. It's a high accomplishment.
140:48 - I feel like my friends who are professors seem happier than those who work at
140:52 a frontier lab, to be honest. There's a grounding there. The frontier labs definitely do this 9-9-6,
141:00 which is shorthand for working all the time. - Can you describe 9-9-6? It's a culture invented, I believe,
141:07 in China and adopted in Silicon Valley. What is 9-9-6? It's 9:00 AM to 9:00 PM,
141:14 - Six days a week. - six days a week. What is that, 72 hours? Okay. So, is this basically the standard
141:21 in AI companies in Silicon Valley? This kind of grind mindset.
141:26 - Yeah, I mean, maybe not exactly like that, but I think there is a trend towards it.
141:30 And it's interesting. I think it almost flipped because when I was in
141:34 in academia, I felt like that. As a professor, you write grants,
141:38 you teach, and you do research. It's like three jobs in one,
141:41 and it's more than a full-time job if you want to be successful.
141:45 successful. And I feel like now, like Nathan just said, the professors, in comparison to a lab, I think they have
141:53 less pressure or workload than at a frontier lab because— - I think they work a lot. They're just so fulfilled. By working with students—
142:00 and having a constant runway of mentorship and a mission that is
142:04 very people-oriented, I think in a era when things are moving very fast
142:08 and are very chaotic, it's very rewarding to people. - Yeah, and I think at a startup, it's this pressure. It's like
142:15 you have to make it. And it's really important that people put in the time,
142:19 but it is really hard because you have to deliver constantly, and
142:23 I've been at a startup. I had a good time, but I don't know if I could do it
142:27 forever. It's an interesting pace and it's exactly like we talked about in the beginning. These models are leapfrogging
142:35 each other, and they are just constantly trying to take the next step compared to their competitors. It's just ruthless right now.
142:42 - I think this leapfrogging nature and having multiple players is actually an
142:45 underrated driver of language modeling progress where competition is
142:49 so deeply ingrained in people, and these companies have intentionally created very strong cultures. Like, Anthropic is
142:57 known to be so culturally, like, deeply committed and organized. I mean, we hear so little from them, and
143:05 everybody at Anthropic seems very aligned. And it's like being in
143:09 a culture that is super tight and having this competitive dynamic is a thing that's gonna
143:16 make you work hard and create things that are better. But that comes at the cost of human capital, which is like
143:24 you can only do this for so long, and people are definitely burning out.
143:28 I wrote a post on burnout as I've tread in and out of this myself, especially trying to be a manager, full-mode training.
143:36 It's a crazy job doing this. The book Apple in China by Patrick
143:39 McGee, he talked about how hard the Apple engineers worked to set up the supply
143:43 chains in China, and he was like, they had "saving marriage" programs, and
143:47 he told in a podcast, he was like, "People died from this level of working hard." So I think it's just like
143:55 it's a perfect environment for creating progress based on human
143:59 expense, and there's gonna be a lot of... the human expense is the 996 that we started this with, which is like—
144:06 ... people do really grind. - I also read this book. I think they had a code word for if someone had to go home
144:13 to spend time with their family to save the marriage, and it's crazy. Then the
144:16 colleagues said, "Okay, this is like red alert for this situation. We have to let
144:20 that person go home this weekend." But at the same time, I don't
144:24 think they were forced to work. They were so passionate about the product, I guess, that
144:28 you get into that mindset. And I had that sometimes as an academic, but also as an independent person, I have that sometimes. I overwork,
144:36 and it's unhealthy. I had back issues, I had neck issues, because I
144:40 did not take the breaks that I maybe should have taken. But no one forced
144:44 me to; it's because I wanted to work, because it's exciting stuff.
144:46 - That's what OpenAI and Anthropic are like. They want to do this work.
144:49 - Yeah, but there's also a feeling of fervor that's building, especially in Silicon Valley, aligned with the scaling laws idea, where
144:57 there's this hype where the world will be transformed in a scale of
145:01 weeks and you want to be at the center of it. And then, you know,
145:04 I have this great fortune of having conversations with a wide variety of human beings,
145:12 and from there I get to see all these bubbles and echo chambers across
145:16 the world. It's fascinating to see how we humans form them. And I think it's
145:20 fair to say that Silicon Valley is a kind of echo chamber, a kind of silo and
145:27 bubble. I think bubbles are actually really useful and effective. It's not necessarily a negative thing because you could be ultra-productive.
145:35 It could be the Steve Jobs reality distortion field, because you just convince each other that
145:41 breakthroughs are imminent, and by convincing each other of that, you make the breakthroughs imminent.
145:48 - Byrne Hobart wrote a book classifying bubbles. One of them is financial bubbles,
145:53 which is like speculation, which is bad, and the other one is for build-outs,
145:56 because it pushes people to build these things. And I do think AI is in
146:00 this, but I worry about it transitioning to a financial bubble, which is
146:05 - Yeah, but also in the space of ideas, that bubble—you are doing a reality distortion
146:11 field, and that means you are deviating from reality. And if
146:15 you go too far from reality while also working, you know, 996,
146:22 you might miss some fundamental aspects of the human experience,
146:26 including beyond Silicon Valley. This is a common problem in Silicon Valley:
146:30 it's a very specific geographic area. You might not understand the
146:34 Midwest perspective, the full experience of all the other humans in the United States and across the world,
146:40 and you speak a certain way to each other, you convince each other of a certain
146:44 thing, and that can get you into real trouble. Whether AI is a big success and becomes a powerful technology
146:52 or it's not, in either trajectory you can get yourself into trouble. So you have to consider all of that. Here you are, a young
146:59 person trying to decide what you want to do with your life. - The thing that is... I don't even really understand this, but the
147:06 SF AI memes have gotten to the point where "permanent underclass" was
147:10 one of them, which was the idea that the last six months of 2025 was the only time to build durable value in an AI
147:17 startup or model. Otherwise, all the value will be captured by
147:21 existing companies and you will therefore be poor, which... that's an example of the SF thing that goes so far. I still
147:28 think for young people going to be able to tap into it, if you're really
147:32 passionate about wanting to have an impact in AI, being physically in
147:36 SF is the most likely place where you're going to do this. But it has
147:40 has trade-offs. - I think SF is an incredible place, but there is a bit of
147:45 a bubble. And if you go into that bubble, which is extremely
147:49 valuable, just get out also. Read history books, read literature, visit other places in the world.
147:57 Twitter and Substack are not the entire world. - I would say, one of the people I worked with is moving to SF, and it's like, I
148:05 need to get him a copy of Season of the Witch, which is a history of SF from
148:08 1960 to 1985, which goes through the hippie revolution, like all the
148:16 gays taking over the city and that culture emerging, and then the
148:20 HIV/AIDS crisis and other things. And it's just like, that is so recent, and
148:24 so much turmoil and hurt, but also love in SF. And it's like, no one knows about this. It's a great book, Season of the Witch. I recommend it.
148:32 A bunch of my SF friends who get out recommended it to me. And I think that's just like living
148:39 there... I lived there and I didn't appreciate this context, and it's
148:43 just so recent. - Yeah. Okay, let's... We talked a lot about a lot of things. Certainly about
148:54 the things that were exciting last year. But this year, One of the things you guys mentioned that's exciting is the scaling of text
149:02 diffusion models, and just a different exploration of text diffusion. Can you talk
149:05 about what that is and what the possibility it holds? So, different kinds of approaches than the current LMs?
149:13 - Yeah, so we talked a lot about the transformer architecture and the autoregressive
149:17 transformer architecture specifically, like GPT. And it doesn't mean no one
149:21 else is working on anything else. So, people are always on the,
149:25 let's say, lookout for the next big thing. Because I think it would be almost
149:28 stupid not to. Because sure, right now, the transformer architecture is the thing, and it works best, and there's, right now, nothing
149:36 else out there. But, you know, it's always a good idea to not put all your eggs into one
149:40 basket. So, people are developing other alternatives to the autoregressive transformer. One of them would be, for example, text
149:48 diffusion models. And listeners may know diffusion models from image
149:52 generation, like Stable Diffusion popularized it. There was a paper on
149:56 generating images. Back then, people used GANs, Generative Adversarial
150:00 Networks. And then there was this diffusion process where you iteratively
150:03 denoise an image, and that resulted in really good quality images over
150:07 time. Stable Diffusion was a company. Other companies build their own diffusion models.
150:11 And then people are now like, "Okay, can we try this also for text?"
150:15 Doesn't, you know, make intuitive sense yet, because it feels like, okay, it's not
150:19 something continuous like a pixel that we can differentiate. It's discrete text, so how do
150:23 we implement that denoising process? It's kind of similar to the BERT models by
150:30 Google. Like, when you go back to the original transformer, they were the encoder and
150:34 the decoder. The decoder is what we are using right now in GPT and so
150:38 forth. The encoder is more like a parallel technique where you have multiple tokens that you
150:45 fill in in parallel. GPT models, they do autoregressive generation,
150:49 completing the sentence one token at a time. And in BERT models, you have a sentence that has gaps.
150:57 You mask them out, and then one iteration is filling in these gaps. Text diffusion is kind of like that, where you
151:04 are starting with some random text, and then you are filling in the missing parts or refining them iteratively over multiple iterations.
151:12 And the cool thing here is that this can do multiple tokens at the same time. It's like
151:19 the promise of having it more efficient. Now, the trade-off is, of course,
151:23 how good is the quality? It might be faster, and now you have this dimension of the denoising process. The more steps you do, the better the text
151:31 becomes. And people... I mean, you can scale in different ways. They try to see if that is
151:39 maybe a valid alternative to the autoregressive model in terms of giving you the same quality for less compute.
151:46 Right now, there are papers that suggest if you want to get the same
151:50 quality, you have to crank up the denoising steps, and then you end up spending the same compute you would spend on an autoregressive model.
151:58 The other downside is, while being parallel sounds appealing,
152:01 some tasks are not parallel. Like reasoning tasks or tool use,
152:06 maybe where you have to ask a code interpreter to give you an intermediate result. That is tricky with diffusion models. So, there are some
152:13 hybrids, but the main idea is: how can we parallelize it? It's an
152:17 interesting avenue. I think right now, there are mostly research models out there, like LaMDA and some other ones.
152:24 I saw some by startups, some deployed models. There is no big diffusion model at scale yet, like on the Gemini or ChatGPT level.
152:32 But there was an announcement by Google, a site where they said they are launching Gemini Diffusion, and they
152:40 put it into context of their Gemini Nano 2 model, and they said basically: for the same quality on most
152:47 benchmarks, we can generate things much faster. You mentioned what's next. I don't think the text diffusion model is going to replace
152:54 autoregressive LLMs, but it will be something maybe for quick, cheap, at-scale tasks. Maybe the free
153:02 tier in the future will be something like that. - I think there are examples where it's already being used. To paint an example of why this is
153:12 better, for example, when GPT-5 is taking 30 minutes to respond, it's generating one token at a time. And this diffusion idea is
153:19 essentially to generate all of those tokens and the completion in
153:23 one batch, which is why it could be way faster. And I think it could be
153:26 suited for... the startups I'm hearing about are code startups where you have a
153:30 code base, and you have somebody that's effectively "vibe coding," and they say,
153:34 "Make this change." And a code diff is essentially a huge reply from the model, but it doesn't have to have that
153:41 much external context, and you can get it really fast by using these diffusion models.
153:45 One example I've heard is that they use text diffusion to generate
153:49 really long diffs, because doing it with an autoregressive model would take
153:53 minutes, and that time for a user-facing product causes a lot of churn.
153:57 Every second, you lose a lot of users. So, I think it's going to be this
154:01 thing where it's going to— ...grow and have some applications, but I actually thought that different types of models
154:06 were going to be used for different things much sooner than they have been,
154:10 so I kind of trade off. I think the tool-use point is the one that's stopping
154:14 them from being most general purpose because, for Claude Code and ChatGPT search,
154:22 the autoregressive chain is interrupted with some external tool, and
154:25 I don't know how to do that with the diffusion setup. - So what's the future of tool use this year and then in the coming years? Do you
154:32 think there's going to be a lot of developments there, and how that's integrated into the entire
154:36 stack? - I do think right now, it's mostly on the proprietary LLM side,
154:41 but I think we will see more of that in the open-source tooling. And I
154:45 think it is a huge unlock because then you can really outsource certain tasks from just memorization to actual—
154:53 you know, instead of having the LLM memorize what is 23 plus 5,
154:57 just use a calculator. - So do you think that can help solve hallucination?
155:02 - Not solve it, but reduce it. So the LLM still needs to know
155:06 when to ask for a tool call. And the second one is, well, it doesn't mean the internet is always correct. You can do
155:14 a web search, but let's say I asked who won the World Cup in, let's say, 1998; it still needs to find the right website and get the right
155:21 information. You can still go to the incorrect website and give me incorrect
155:24 information. So I don't think it will fully solve that, but it is
155:28 improving it in that sense. And so another cool paper earlier this year—I think it was
155:35 December 31st, so it's not technically 2026, but close—the recursive language model.
155:43 That's a cool idea to kind of take this even a bit further. Just to explain, Nathan, you also mentioned earlier,
155:51 it's harder to do cool research in academia because of the compute budget. If I
155:55 recall correctly, they did everything with GPT-5, so they didn't even use local models,
155:59 but the idea is, let's say you have a long-context task; instead of having the
156:03 LLM solve all of it in one shot or even in a chain, you break it down into sub-tasks. You have the LLM decide
156:10 what is a good sub-task, and then recursively call an LLM to solve that. And I think something like
156:18 that, adding tools—you know, each one maybe you have a huge Q&A task, so each one goes to the web and gathers
156:26 information, and then you pull it together at the end and stitch it back together.
156:29 I think there's going to be a lot of unlock using things like that where
156:35 you don't necessarily improve the LLM itself; you improve how the LLM is
156:39 used and what the LLM can use. One downside right now with tool use
156:43 is you have to give the LLM permission to use tools. And that will take some trust, especially if you want to unlock things
156:51 like having an LLM answer emails for you—or not even answer, but just
156:54 sort them for you or select them for you or something like that. I don't know if I would
156:58 today give an LLM access to my emails, right? I mean, this is a huge risk.
157:03 - I think there's a cool... one last point on the tool use thing. I think that
157:07 you hinted at this, and we've both come at this in our own ways, is that the open versus
157:12 closed models use tools in very different ways, where open models, people go to
157:16 Hugging Face and download the model, and then the person's going to be like, "What tool do I want?"
157:20 I don't know, Exa is my preferred search provider, but somebody else might care
157:24 for a different search startup. Where you release a model, it needs to be useful for multiple
157:28 tools, for multiple use cases, which is really hard because you're making a
157:31 general reasoning engine model, which is actually what gpt-oss-120b is
157:35 good for. But on the closed models, you're deeply integrating
157:40 the specific tool into your experience, and I think that open models will
157:44 struggle to replicate some of the things that I like to do with closed
157:47 models, which will be like, you can reference a mix of public and private
157:51 information. And something that I keep trying every three to six months,
157:55 I try Claude Code on the web, which is just prompting a model to
157:59 make an update to some GitHub repository that I have. And it's just like
158:03 that set of secure cloud environments is just so nice for just sending it off to do this thing and then come back to
158:10 me, and these will probably help define some of the local open and closed
158:18 niches. But I think initially, because there was such a rush to get tool use
158:22 working, the open models were on the back foot, which is kind of inevitable. I think there's so
158:26 much research, so many resources in these frontier labs, but it will be
158:30 fun when the open models solve this because it's going to necessitate a bit
158:33 more flexible and potentially interesting model that might work with this recursive
158:37 idea to be an orchestrator and a tool use model, so hopefully the necessity drives some interesting innovation there.
158:45 - So, continual learning—this is a longstanding topic, important problem. I think that increases in importance as
158:54 the cost of training the models goes up. So can you explain what
158:57 continual learning is and how important it might be this year and in the coming
159:01 years to make progress? - This relates a lot to this kind of SF zeitgeist of, what is
159:06 AGI, which is Artificial General Intelligence, and what is ASI, Artificial Superintelligence, and what are the language models that we
159:14 have today capable of doing? I think the language models can solve a lot
159:17 of tasks, but a key milestone among the AI community is essentially when AI could replace any remote worker,
159:25 taking in information and solving digital tasks and doing them. And
159:29 the limitation that's highlighted by people is that a language model will not learn from feedback the same way that an
159:36 employee does. So if you hire an editor, the editor will mess up, but
159:40 you will tell them. And if you hired a good editor, they don't do it again. But language models don't
159:44 have this ability to modify themselves and learn very quickly. So the idea is, if we are going to
159:49 actually get to something that is a true, general adaptable intelligence that can go into any remote work scenario, it needs to be able to learn
159:56 quickly from feedback and on-the-job learning. I'm personally more bullish on language models being able to just provide
160:04 them with very good context. You said, maybe offline, that you can write extensive documents to models where you say,
160:11 "I have all this information. Here are all the blog posts I've ever written.
160:15 I like this type of writing. My voice is based on this." But many people don't provide
160:19 this to models, and the models weren't designed to take this amount
160:23 of context previously. Agentic models are just starting. So it's
160:27 this kind of trade-off: do we need to update the weights of this model
160:31 with this continual learning thing to make them learn fast? Or the
160:34 counterargument is we just need to provide them with more context and information,
160:38 and they will have the appearance of learning fast by having a lot of context and being
160:42 smart. - So we should mention the terminology here. Continual learning refers to
160:46 changing the weights continuously so that the model adapts and adjusts
160:53 based on the new incoming information, doing so continually, rapidly, and frequently.
161:00 And then the thing you mentioned on the other side of it generally will be referred to as in-context learning. As
161:08 you learn stuff, there's a huge context window. You can just keep loading it with extra information every time you prompt the system,
161:15 which I think both legitimately can be seen as learning. It's just a different place where you're doing the learning.
161:24 - I think, to be honest with you, continual learning — updating
161:28 weights — we already have that in different flavors. If you think about
161:32 how... I think the distinction here is: do you do that on a personalized custom model for each person, or
161:39 do you do it on a global model scale? I think we have that already,
161:43 going from GPT-5 to 5.1 and 5.2. It's maybe not immediate, but it is a curated update, a quick curated update
161:51 where there was feedback about things they couldn't do, feedback by the
161:55 community. They updated the weights, next model, and so forth. So it
161:58 is a flavor of that. Another even finer-grained example is like
162:06 RLVR; you run it, it updates. The problem is you can't just do that for each person because it would be too expensive to update the
162:14 weights for each person, and I think that's the problem. Unless you get...
162:17 Even at OpenAI scale, building the data centers, it would be too
162:21 expensive. I think that is only feasible once you have something
162:25 on the device where the cost is on the consumer. Like what Apple tried to do with the
162:29 Apple Foundation models, putting them on the phone, where they learn from
162:33 experience. - A bit of a related topic, but this kind of, maybe anthropomorphized term: memory.
162:42 What are different ideas for the mechanism of how to add memory to these systems as we're
162:46 increasingly seeing? Personalized memory especially? - Right now, it's mostly basically
162:52 stuffing things into the context and then just recalling that.
162:56 But again, I think it's expensive because you have to—you can cache it, but still
163:04 you spend tokens on that. And the second one is you can only do
163:08 so much. I think it's more like a preference or style. I mean, a lot of people do
163:12 that when they solve math problems. You say it's way so you can add previous knowledge and stuff, but you also give it certain preference
163:20 prompts: "do what I preferred last time," or something like that. But
163:24 it doesn't unlock new capabilities. So for that, one thing people still use is
163:31 LoRA adapters. These are basically, instead of updating the whole weight
163:34 matrix, there are two smaller weight matrices that you kind of have
163:39 in parallel or overlays like the delta. But yeah, you can do that to some extent, but then again,
163:45 it is economics. There were also papers, for example, LoRA learns less but forgets less.
163:53 It's like, there's no free lunch. If you want to learn more, you need to use more
163:57 weights, but it gets more expensive. And then again, if you learn more, you forget
164:01 more, and you have to find that Goldilocks zone basically. - We haven't really mentioned it much, but implied in this discussion is
164:08 context length also. Is there a lot of innovation that's possible there?
164:13 - I think the colloquially accepted thing is that it's a compute and data
164:17 problem where you can... and sometimes small architecture things like attention variants. We talked about
164:24 hybrid attention models, which is essentially if you have what
164:28 looks like a state space model within your transformer. And those are
164:32 better suited because you have to spend less compute to model the
164:37 furthest along token. I think that, those aren't free because they have to be accompanied by a lot of compute or
164:47 the right data. How many sequences of 100,000 tokens do you have in
164:51 the world, and where do you get these? It just ends up being pretty
164:54 expensive to scale them. We've gotten pretty quickly to a million tokens
164:58 of input context length. I would expect it to keep increasing and
165:02 get to 2 million or 5 million this year, but I don't expect it to go to 100
165:06 million. That would be like a true breakthrough, and I think those
165:10 breakthroughs are possible. I think of the continual learning thing as a research problem where
165:14 there could be a breakthrough that just makes transformers work way
165:18 better at this and it's cheap. These things could happen with so much scientific
165:22 attention. But turning the crank, it'll be consistent increases over time.
165:28 - Looking at the extremes, I think there's, again, no free lunch. So, the one
165:31 extreme to make it cheap: you have, let's say, an RNN that has a single state
165:35 where you save everything from the previous stuff. It's like a
165:39 specific fixed-size thing, so you never really grow the memory because you are stuffing everything into one state, but
165:47 then the longer the context gets, the more information you forget because
165:51 you can't compress everything into one state. Then on the other hand, you have the transformers, which try to remember every token, which is great
165:58 sometimes if you want to look up specific information, but very expensive because you have the
166:02 KV cache that grows, the dot product that grows. But then, like you said, the Mamba layers—I mean, they kind of have the same problem.
166:10 Like an RNN, you try to compress everything into one state; you're a bit more selective
166:14 there. But then I think it's like this Goldilocks zone again. With
166:18 Nemotron 3, they found a good ratio of how many attention layers do you need for the global information where everything is accessible
166:25 compared to having these compressed states. And I think that's
166:29 how we will scale more—by finding better, let's say, ratios in the Goldilocks zone, like between
166:37 making computing cheap enough to run, but then also making it powerful enough to be useful. And one more plug here,
166:45 the Recursive Language Model paper, that is one of the papers that tries to kind of address the long context thing. So what they found
166:52 is essentially instead of stuffing everything into this long context
166:57 if you break it up into multiple smaller tasks, so you save memory by having multiple smaller cores, you can actually get
167:04 better accuracy than having the LLM try everything all at once.
167:08 I mean, it's a new paradigm. We will see, you know, there might be other flavors of
167:12 that. So I think with that, we will still make improvement on
167:16 long context, but then also, like Nathan said, I think the problem is for
167:20 pre-training itself, we don't have as many long context documents as other
167:24 documents. So it's harder to study basically how LLMs behave and stuff like that on that level.
167:31 - There are some rules of thumb where essentially you pre-train a language model, like OLMo.
167:35 we pre-trained at like 8K context length and then extended to
167:39 32K with training. And there are some rules of thumb where you're essentially doubling
167:43 the training context length, it takes like 2X compute, and then you can
167:46 normally like 2 to 4X the context length again. So I think a lot of it ends up being kind of compute bound at pre-training,
167:54 which is in this... Like we talked about, everyone talks about this big increase in
167:58 compute for the top labs this year, and that should reflect in some longer context
168:01 windows. But I think on the post-training side, there are some more interesting things. As we have
168:05 agents, the agents are gonna manage this context on their own, where now
168:09 agents, people that use Claude Code a lot dread the compaction, which is when Claude takes its
168:13 entire full 100,000 tokens of work and compacts it into a bulleted
168:17 list. But what the next models will do—and I'm sure people are already working on this—is essentially the model can control when it
168:25 compacts and how. So you can essentially train your RL algorithm where
168:29 compaction is an action- ...where it shortens the history and then the problem formulation will be, "I
168:35 want to keep the maximum evaluation scores that I have gotten
168:39 while the model compacts its history to the minimum length." Because then you
168:43 have the minimum amount of tokens that you need to do this kind of compounding
168:47 autoregressive prediction. So there are actually pretty nice problem setups in this, where
168:51 the... Like these agentic models learn to use their context in a
168:55 different way than just plow forward. - One interesting recent example would be DeepSeek-V3.2,
169:00 where they had a sparse attention mechanism where they have essentially a very efficient, small, lightweight indexer. And
169:08 instead of attending to all tokens, it selects: "What tokens do I actually need?"
169:12 I mean, it almost comes back to the original idea of attention where you are selective, but attention is always on, you have maybe zero
169:20 weight on some of them, but you use them all. But they are even more like, "Let's just mask that out
169:24 or not even do that." And even with sliding window attention in OLMo, that is also kind of that idea.
169:32 You have a rolling window where you keep it fixed, because you don't need everything.
169:35 Occasionally, some layers you might, but it's wasteful. But right now, I think,
169:39 if you use everything, you're on the safe side—it gives you the best bang for the buck
169:43 because you never miss information. And I think this year will be more
169:47 about figuring out, like you said, how to be smarter about that. Right now, people want to have the next state-of-the-art, and the
169:55 state-of-the-art happens to be the brute-force, expensive thing. And then once you have that, as you said, keep that
170:02 accuracy, but let's see how we can do that cheaper now, with tricks.
170:07 - Yeah. All this scaling thing. The reason we get the Claude 4.5 Sonnet model first is
170:12 because you can train it faster and you're not hitting these compute walls as soon.
170:16 They can just try a lot more things and get the model faster, even though the bigger
170:19 model is actually better. - I think we should say that there's a lot of exciting stuff going on in the AI space.
170:25 My mind has recently been really focused on robotics. Today, we almost entirely didn't talk
170:33 about robotics. There's a lot of stuff on image generation, video generation. I think it's fair
170:40 to say that the most exciting research work in terms of the amount, intensity, and fervor is in the
170:48 LLM space, which is why I think it's justified for us to focus on the
170:52 LLMs that we're discussing. But it'd be nice to bring in certain things that might be useful. For example, world models—
170:59 there's growing excitement on that. Do you think there will be any
171:03 use in this coming year for world models in the LLM space? Yes, I do think so. Also with LLMs,
171:10 what's interesting here is that if we unlock more LLM capabilities, it also automatically unlocks all the other fields because
171:18 it makes progress faster. A lot of researchers and engineers use LLMs for
171:24 coding. So even if they work on robotics, if you optimize these LLMs
171:28 that help with coding, it pays off. But then, yes, world models are interesting. It's basically where you
171:36 have the model run a simulation of the world in a sense, like a
171:40 little toy thing of the real thing, which can, again, unlock capabilities regarding data the LLM is
171:47 not aware of. It can simulate things. And I think LLMs happen to work
171:55 well by pre-training and doing next-token prediction. But we could do this even more sophisticatedly in a
172:03 sense. I think there was a paper by Meta, a paper called World Models. So where they
172:10 basically apply the concept of world models to LLMs again, where instead of just having next-token prediction and
172:18 verifiable rewards, checking the answer correctness, they also make sure the
172:21 intermediate variables are correct. You know, it's kind of like
172:25 the model is learning basically a code environment in a sense. And I
172:29 think this makes a lot of sense. It's just expensive to do, but it is making things more sophisticated, like modeling
172:40 the whole thing, not just the result. And so it can add more value. I remember when I was a grad student, there is a...
172:51 competition called CASP, I think, where they do protein structure prediction. They predict the structure of a protein
172:59 that is not solved yet at that point. So in a sense, this is actually great, and I think we need something like that for LLMs also,
173:07 where you do the benchmark, but no one does. You hand in the results, but no one
173:11 knows the solution. And then after the fact, someone reveals that. But,
173:15 AlphaFold, when it came out, it crushed this benchmark. I mean, there were also multiple
173:23 iterations, but I remember the first one. I'm not an expert in
173:27 that subject, but the first one explicitly modeled the physical
173:31 interactions of the... You know, the physics of the molecule.
173:34 Also the angles, impossible angles. And then in the next version, I think they got rid of
173:38 this, and just with brute force, scaling it up. And I think with LLMs, we are
173:42 currently in this brute force scaling because it just happens to work. But I do think at some
173:46 point it might make sense to bring back this thing. And I think with world models, I think that is
173:53 where I think that might be actually quite cool. I mean, yeah.
173:57 And of course, also for robotics, which is completely unrelated to LLMs.
174:03 - Yeah. And robotics is very explicit. So there's the problem of locomotion
174:07 or manipulation. Locomotion is much more solved, especially in the learning domain. But there's a
174:12 lot of value, just like with the initial protein folding systems,
174:14 bringing in the traditional model-based methods. So you don't... it's unlikely
174:21 that you can just learn the manipulation or the whole body, local
174:25 manipulation problem end to end. That's the dream. But then you realize when you look at the magic of the human hand
174:33 and the complexity of the real world, it's really hard to learn this all the
174:37 way through, the way I guess AlphaFold 2 didn't. - I'm excited about the robotic learning space. I think it's collectively
174:44 getting supercharged by all the excitement and investment in
174:48 language models generally, where the infrastructure for training
174:51 transformers, which is a general modeling thing, is becoming
174:55 world-class industrial tooling, where wherever there was a limitation for robotics, it's just way better.
175:03 There's way more compute. And then on top of that, they take these language models
175:07 as kind of central units where you can do interesting explorative work around
175:11 something that already works. And then I see it emerging as, kind of like we talked about, Hugging Face transformers and
175:18 Hugging Face. I think when I was at Hugging Face, I was trying to get this to happen, but it was too early.
175:22 It's like these open robotic models on Hugging Face, and having people be able to contribute data and fine-tune them.
175:29 I think we're much closer now that the investment in robotics and self-driving
175:33 cars is related and it enables this, where once you get to the point
175:37 where you can have this sort of ecosystem where somebody can download a robotics model and
175:41 maybe fine-tune it to their robot or share datasets across the world.
175:45 There's some work in this area like RTX, I think it was a few years ago, where people are starting to do that. But once they have this ecosystem,
175:53 it'll look very different. And then this whole post-ChatGPT boom is putting more resources into that, which I think
176:00 is a very good area for doing research. - This is also resulting in much better, more accurate, and more
176:06 realistic simulators being built, closing the sim-to-real gap in the robotic
176:10 space. But, you know, you mentioned a lot of excitement in the
176:14 robotics space and a lot of investment. The downside of that, which happens in hype cycles, I personally
176:21 believe, and most robotics people believe, that it's not... Robotics is not going to be solved at the
176:28 time scale as being implicitly or explicitly promised. And so what happens when there's all these robotics companies
176:36 that spring up and then they don't have a product that works? Then there's
176:42 going to be this kind of crash of excitement, which is nerve-wracking. Hopefully something else will come in and keep swooping
176:49 in so that the continued development of some of these ideas keeps going.
176:53 - I think it's also related to the continual learning issue, essentially, where
176:58 the real world is so complex. With LLMs, you don't really need to have something learn for the user,
177:05 because there are a lot of things everyone has to do. Everyone maybe wants
177:09 to, I don't know, fix their grammar in their email or code or something like that. It's more constrained, so you can kind of prepare the model for
177:17 that. But preparing the robot for the real world is harder. I mean, you
177:20 have the robotic foundation models, and you can learn certain things like grasping things. But then again,
177:28 everyone's house is different. It's so different, and that is,
177:32 I think, where the robot would have to learn on the job, essentially.
177:36 And that, I guess, is the bottleneck right now: how to, customize it on the fly, essentially.
177:42 - I don't think I can possibly understate the importance of the thing that doesn't get talked about almost at all
177:50 by robotics folks or anyone, which is safety. All the interesting complexities we talk about learning, all the failure modes and
177:57 failure cases, everything we've been talking about with LLMs—sometimes they fail in
178:01 interesting ways. All of that is fun and games in the LLM space.
178:06 In the robotic space, in people's homes, across millions of minutes and billions
178:13 of interactions, you really are almost allowed to fail never. When you have embodied systems that are
178:21 put out there in the real world, you just have to solve so many
178:27 problems you never thought you'd have to solve when just thinking about the general
178:31 robot learning problem. - I'm so bearish on in-home learned robots for consumer purchase. I'm very bullish on self-driving cars, and I'm
178:40 very bullish for robotic automation, e.g., like Amazon distribution
178:44 where Amazon has built whole new distribution centers designed for robots
178:48 first rather than humans. There's a lot of excitement in AI circles about
178:52 AI enabling automation and mass-scale manufacturing, and I do think that the path to robots doing that is more reasonable,
178:58 where it's a thing that is designed and optimized to do a repetitive task that a human could
179:05 conceivably do but doesn't want to. And then I'm much, but it's also going to take a lot longer than people
179:13 probably predict. I think the leap from AI singularity to we can now scale up mass manufacturing in the
179:20 US because we have a massive AI advantage is one that is troubled by a lot of political and other challenging problems.
179:31 - Let's talk about timelines, specifically timelines to AGI or
179:37 ASI. Is it fair, as a starting point, to say that nobody really agrees on the
179:43 definitions of AGI and ASI? - I kind of think there's a lot of disagreement, but
179:49 I've been getting pushback where a lot of people kind of say the same thing, which is like a
179:53 thing that could reproduce most digital economic work. So, the remote worker is a fairly reasonable
180:00 example. And I think OpenAI's definition is somewhat related to that, which is like an AI that can do a lot of
180:08 economically valuable tasks—which I don't really love as a definition, but I think
180:12 it could be a grounding point, because language models today, while immensely powerful,
180:19 are not this remote worker drop-in. And there are things that
180:23 could be done by an AI that are way harder than remote work, which are like finding an unexpected
180:30 scientific discovery that you couldn't even posit, which would be an example of something that
180:34 somebody says is an artificial superintelligence problem. Or,
180:38 taking in all medical records and finding linkages across certain
180:45 illnesses that people didn't know, or figuring out that some common drug can
180:49 treat some niche cancer. They would say that that is a superintelligence
180:52 thing. So these are kind of natural tiers. My problem with it is that it
180:57 becomes deeply entwined with the quest for meaning of AI and these religious
181:01 aspects to it. So there's different paths you can take it. - And I don't even know if the remote worker is a good definition
181:10 because what exactly is that? I actually, I mean, I like... I don't know if you like the originally titled
181:17 AI27 report. They focus more on code and research taste, so the target there is the
181:25 superhuman coder. So they have several milestone systems: Superhuman coders, superhuman AI researcher,
181:31 then superintelligent AI researcher, and then the full ASI, artificial superintelligence. But
181:39 after you develop the superhuman coder, everything else follows quickly.
181:45 There, the task is to have fully autonomous, automated coding.
181:52 So any kind of coding you need to do in order to perform research is fully automated. And from there,
181:59 humans would be doing AI research together with that system, and they will
182:03 quickly be able to develop a system that can actually do the research for you.
182:07 That's the idea. And initially their prediction was 2027, 2028, and now they've
182:15 pushed it back by three to four years to 2031 (mean prediction). Probably my prediction is even
182:21 beyond 2031, but at least you can, in a concrete way, think about how difficult it is to fully automate programming.
182:31 - Yeah, I disagree with some of their presumptions and dynamics on how
182:35 it would play out, but I think they did good work in the scenario-defining milestones that are concrete and tell a useful story, which is
182:43 why the reach for this AI 2027 document transcended Silicon Valley. It's because they told a good story and they
182:50 did a lot of rigorous work to do this. I think the camp that I fall into is that AI is so-called
182:56 "jagged," which will be excellent at some things and really bad at some things.
183:00 I think that when they're close to this automated software engineer, what it will be good at is traditional ML systems
183:08 and frontend, the model is excellent at; but distributed ML,
183:12 the models are actually quite bad at because there's so little training data
183:16 on doing large-scale distributed learning. This is something we already see,
183:20 and I think this will just get amplified. And then it's kind of messier in these
183:23 trade-offs, like how you think AI research works and so on. - So you think basically a superhuman coder is almost unachievable,
183:31 because of the jagged nature of the thing, you're just always going to
183:35 have gaps in capabilities? - I think it's assigning completeness to something where the models are already
183:42 superhuman at some types of code. I think that will continue.
183:46 And people are creative, so they'll utilize these incredible abilities
183:50 to fill in the weaknesses of the models and move really fast. There'll always
183:54 be this dance for a long time between the humans enabling the thing that the model can't do. And the best AI researchers
184:01 are the ones that can enable this superpower. And I think those lines lead to what we already see. I think like Claude Code for building a
184:08 website, you can stand up a beautiful website in a few hours or do data
184:11 going to keep getting better, and we'll pick up some new coding skills
184:15 along the way. Linking to what's happening in big tech, this AI 2027 report leans into the
184:27 singularity idea, whereas I think research is messy, social and largely in the data in ways that AI models can't
184:34 process. But what we do have today is really powerful and these
184:39 tech companies are all collectively buying into this with tens of billions of dollars of
184:42 investment. So we are going to get some much better version of ChatGPT, a much better version of Claude Code than we already have.
184:50 I think it's just hard to predict where that is going, but the
184:55 bright clarity of that future is why some of the most powerful people in the world are
184:59 putting so much money into this. And I think it's just kind of
185:03 small differences between like—we don't actually know what a better version of
185:06 ChatGPT is, but also, can it automate AI research? I would say probably not, at
185:13 least in this timeframe. Big tech is going to spend $100 billion much
185:17 faster than we get an automated AI researcher that enables an AI research singularity.
185:22 - So you think your prediction would be— if this is even a useful milestone or more than 10 years out?
185:31 - I would say less than that on the software side, but I think longer than that on
185:34 things like research. - Well, let's just for fun try to imagine a world where all software writing
185:43 is fully automated. Can you imagine that world? - By the end of this year, the amount of software that'll be automated will be so
185:50 high. But it'll be things like trying to train a model with RL and you need to have multiple bunches of GPUs
185:57 communicating with each other. That'll still be hard, but it'll be much easier.
186:02 - One way to think about this— the full automation of programming—is just thinking of lines of useful code written, the fraction of
186:12 that to the number of humans in the loop. So presumably there'll be
186:16 for a long time humans in the loop of software writing. It'll just be fewer and
186:20 fewer relative to the amount of code written. Right? And the superhuman coder—I think the presumption there is it
186:28 goes to zero, the number of humans in the loop. What does that world
186:32 look like when the number of humans in the loop is in the hundreds, not in the hundreds of thousands?
186:39 - I think software engineering will be driven more to system design
186:43 and goals of outcomes, where I do think software is largely going to
186:46 be. I think this has been happening over the last few weeks, where people
186:50 have gone from a month ago saying, "Oh yeah, agents are kind of slop," which
186:54 is a famous Karpathy quote, to what is a little bit of a meme—the
187:00 industrialization of software when anyone can just create software with their
187:04 fingerprints. I do think we are closer to that side of things, and it takes
187:08 direction and understanding how systems work to extract the best from the language models. I think it's hard to accept the
187:15 gravity of how much is going to change with software development and how many
187:20 more people can do things without ever looking at the code. - I think what's interesting is to think about whether these systems will be
187:26 independent—completely independent in the sense that, while I have no doubt
187:29 that LLMs will kind of at some point solve coding in a sense,
187:33 like calculators solve calculating, right? So at some point, humans
187:37 developed a tool where you never need a human to calculate that number.
187:41 You just type it in, and it's an algorithm. You can do it in
187:45 that sense. And I think that's the same probably for coding. But
187:49 the question isn't... I think what will happen is, you will just say, "Build that website."
187:53 It will make a really good website, and then you maybe refine it. But will it
187:57 do things independently where... Will you still be having humans asking
188:03 the AI to do something? Like will there be a person to say, "Build that
188:07 website?" Or will there be AI that just builds websites or something?
188:12 - I think talking about building websites is— - Too simple. - The problem with websites and the problem with the web,
188:20 you know, HTML and all that kind of stuff, it's very resilient to just ... slop.
188:26 It will show you slop; it's good at showing slop. I would rather
188:30 think of safety-critical systems, like asking AI to end-to-end generate
188:37 something that manages logistics, or manages cars, a fleet of cars—all that kind of stuff. So it end-to-end generates that for you.
188:45 - I think a more intermediate example is take something like Slack or Microsoft
188:49 Word. I think if organizations allow it, AI could very easily
188:54 implement features end-to-end and do a fairly good job for like
188:58 things that you want to try. You want to add a new tab in Slack that you want to use, and I think AI will be able to do that pretty well.
189:06 - Actually, that's a really great example. How far away are we from that?
189:09 - Like this year. - See, I don't know. I don't know. - I guess I don't know how bad production codebases are, but I think that
189:18 within... on the order of a few years, a lot of people are going to be pushed to
189:22 be more of a designer and product manager, where you have multiple of these
189:26 agents that can try things for you and they might take one to two days to implement
189:30 a feature or attempt to fix a bug. And you have these dashboards,
189:34 which I think Slack is actually a good dashboard where your agents will talk to you and
189:38 you'll then give feedback. But things like, if I make a website, like, "Do you want a passable logo?" I think these
189:46 cohesive design things and the style is going to be very hard
189:50 for models and deciding on what to add the next time. - I just... Okay. I hang out with a lot of programmers and some of them
189:57 are a little bit on the skeptical side in general. That's just their vibe. I
190:05 just think there's a lot of complexity involved in adding features to complex
190:09 systems. Like, if you look at the browser, Chrome. If I wanted to add a feature, if I wanted to have tabs
190:17 as opposed to up top, I want them on the left side. Interface-wise, right? I think we're not... This is not a next-year thing.
190:26 - One of the Claude releases this year, one of their tests was: we give it a piece of
190:29 software and leave Claude to run to recreate it entirely, and it could already
190:33 almost rebuild Slack from scratch, just given the parameters of the software and left in a sandbox environment to do that.
190:41 - So the "from scratch" part, I like almost better. - So it might be that smaller and newer companies are advantaged, and they're like,
190:48 "We don't have the bloat and complexity, and therefore this feature exists."
190:53 - And I think this gets to the point you mentioned, that some people you talk
190:57 to are skeptical. I think that's not because the LLM can't do
191:01 X, Y, Z. It's because people don't want it to do it this way.
191:05 - Some of that could be a skill issue on the human side. We have to be
191:09 honest with ourselves. And some of that could be an underspecification issue.
191:13 So, programming, it's like you're just assuming... This is like an
191:20 issue with communication in relationships and friendships. You're assuming the
191:24 LLM is supposed to read your mind. This is where spec-driven design is
191:28 really important. Using natural language to specify what you want.
191:32 - If you talk to people at the labs, they use these in their training
191:36 and production code. Claude Code is built with Claude Code, and they all
191:40 use these things extensively. Dario talks about how much of Claude's
191:44 code... It's like these people are slightly ahead in terms of the capabilities
191:50 they have and what they probably spend on inference. They could spend 10 to 100x as much as we're spending on a lowly
191:57 $100 or $200 a month plan. They truly let it rip. And I think that,
192:03 with the pace of progress that we have, it seems like- a year ago we didn't have Claude Code and we didn't really have reasoning
192:11 models. The difference between sitting here today and what we can do with these
192:15 models is significant, and there's a lot of low-hanging fruit to improve them. The failure modes are pretty dumb. Like-
192:24 "Claude, you tried to use a CLI command I don't have installed 14 times, and then
192:28 I sent you the command to run." That, from a modeling perspective,
192:31 is pretty fixable. So I don't know. - I agree with you. I've been becoming more and more bullish in
192:38 general. Speaking to what you're articulating, I think it is a
192:42 human skill issue. Anthropic is leading the way, along with other companies, in understanding
192:50 how to best use the models for programming; therefore, they're effectively
192:54 using them. There are a lot of programmers on the outskirts who
192:58 don't... I mean, there's not a really good guide on how to use them. People are trying to figure it out, but-
193:04 - It might be very expensive. The entry point might be $2,000 a
193:08 month, which is only for tech companies and rich people. That could be it.
193:13 - But it might be worth it. If the final result is a working
193:17 software system, it might be worth it. By the way, it's funny how we converged from
193:21 the discussion of timeline to AGI to something more pragmatic and
193:25 useful. Is there anything concrete, interesting, useful, and profound to be said about the timeline to AGI and
193:32 ASI? Or are these discussions a bit too detached from the day to day?
193:39 - There are interesting bets. A lot of people are trying to do RLVR—
193:42 Reinforcement Learning with Verifiable Rewards—in real scientific domains,
193:46 where startups with hundreds of millions of funding have wet labs where they're
193:50 having language models propose hypotheses that are tested in the real
193:54 world. I would say that they're early, but with the pace of progress, it's like-
194:00 ...maybe they're early by six months and they make it because they
194:04 were there first, or maybe they're early by eight years; you don't know.
194:08 That type of moonshot to branch this momentum into other sciences
194:15 would be very transformative. If, AlphaFold moments happen in all sorts of other scientific domains
194:23 by a startup solving this. I think there are startups—maybe Harmonic is
194:27 one—where they're going all in on language models plus Lean for
194:31 math. You had another guest where you talked about this recently, and
194:34 we don't know exactly what's going to fall out of spending $100 million on that model. Most of them will
194:42 fail, but a couple might be big breakthroughs that are very different than ChatGPT or Claude Code type software
194:50 experiences. A tool that's only good for a PhD mathematician but makes them 100X effective...
194:58 - I agree. I think this will happen in a lot of domains, especially
195:02 domains that have a lot of resources, like finance, legal, and pharmaceutical companies.
195:09 But then again, is it really AGI? Because we are now specializing it again. Is it really that much different from back
195:17 in the day when we had specialized algorithms? It's just the same thing,
195:21 way more sophisticated, but I don't know, is there a threshold for AGI? I think the real cool thing here is that
195:29 we have foundation models we can specialize. That's like the
195:33 breakthrough. Right now, I think we are not there yet because, first,
195:37 it's too expensive, but also, ChatGPT doesn't just give away their model to customize it. I think
195:44 once that's true... And I can imagine this as a business model,
195:48 where OpenAI says at some point, "Hey, Bank of America, for $100 million we will do your custom model,"
195:55 something like that. I think that will be the huge economic value-add. The other thing, though, is
196:02 also... Companies, I mean, what is the differentiating factor? If everyone uses the same LLM, if everyone uses
196:09 ChatGPT, they will all do the same thing. Well, if everyone is moving in lockstep, but companies want to have a
196:17 competitive advantage, there is no way around using some of their
196:21 private data and specializing. It's gonna be interesting. - Seeing the pace of progress, it does feel like things are coming. I don't
196:30 think the AGI and ASI thresholds are particularly useful. - I think the real question, and this relates to the remote worker
196:39 thing, is: when are we going to see a big, obvious leap in economic impact?
196:48 Because currently there's not been an obvious leap in the economic impact of LLM models, for example.
196:55 And that's, you know, aside from AGI or ASI, all that stuff, there's a real question of, "When are we going to see a GDP..." "...jump?"
197:06 - Yeah, what is the GDP made up of? A lot of it is financial
197:10 services, so I don't know what this is. - Right, GDP is a- - It's just hard for me to think about the GDP bump, but
197:18 I would say that software development becomes valuable in a different
197:22 way, when you no longer have to look at the code anymore. When Claude Code will make you a small business.
197:29 Which is essentially, Claude can set up your website, your bank account, your email, and your
197:33 whatever else. And you just have to express what you're trying
197:37 to put into the world. That's not just an enterprise market, but it is hard. I don't know how you get people to try doing
197:45 that. I guess if ChatGPT can do it—people are trying ChatGPT.
197:49 - I think it boils down to the scientific question of, "How hard is tool use
197:53 to solve?" Because a lot of the stuff you're implying, the remote work stuff, is tool use. It's like... computer
198:01 use, like how you have an LLM that goes out there, this agentic system, and does something in the world, and
198:09 only screws up 1% of the time. - Computer use- - Or less. - ...is a good example of what labs care about and we haven't seen a lot of progress on.
198:16 We saw multiple demos in 2025 of, like, Claude can use your computer, or OpenAI had operator, and they all suck.
198:25 They're investing money in this, and I think that'll be a good
198:28 example. Whereas actually, something where it just seems like taking over the whole screen seems a lot harder
198:36 than having an API that they can call in the back end. For some of that, you have to set up a different environment for them all to work in. They're
198:43 not working on your MacBook; they are individually interfacing with
198:47 Google and Amazon and Slack, and they handle all these things in a very
198:51 different way than humans do. So some of this might be structural blockers.
198:55 - Also, specification-wise, I think the problem is for arbitrary tasks, well, you still have to specify what you
199:03 want your LLM to do. And how do you do that? What is the environment? How do you specify? You can say what the end goal is, but
199:11 if it can't solve the end goal... with LLMs, if you ask it for text, it
199:15 can always clarify or do sub-steps. How do you put that information into a system that, let's say, books a travel trip for you? You can say,
199:23 "You screwed up my credit card information," but even to get it to that point,
199:27 even to get it to that point, how do you, as a user, guide the
199:31 model before it can even attempt that? I think the interface is really hard.
199:36 - Yeah, it has to learn a lot about you specifically. And this goes to continual learning, about the general mistakes
199:43 that are made throughout, and then mistakes that are made through you.
199:48 - All the AI interfaces are getting set up to ask humans for input.
199:51 I think Claude Code we talked about a lot. It asks feedback and questions. If it doesn't have enough specification
199:58 on your plan or your desired goal, it starts to ask questions, "Would you rather?"
200:02 We talked about Memory, which saves across chats. Its first implementation is kind of odd, where it'll mention my
200:10 dog's name or something in a chat. I'm like, "You don't need to be subtle about this. I don't care."
200:15 But things that are emerging, ChatGPT has the Pulse feature.
200:19 Which is like a curated couple of paragraphs with links to something to look at, and people talk about how models
200:27 are going to ask you questions. Which I think is a very... It's probably going to work. The language model knows
200:34 you had a doctor appointment and asks, "Hey, how are you feeling after that?"
200:37 Which again goes into the territory where humans are very susceptible
200:41 to this, and there's a lot of social change to come. But also,
200:45 they're experimenting with having the models engage. Some people like this
200:49 Pulse feature, which processes your chats and automatically searches
200:53 for information and puts it in the app. So there are a lot of things coming.
200:59 - I used that feature before, and I always feel bad because it does
201:02 that every day, and I rarely check it out. It's like, how much compute
201:06 is burned on something I don't even look at, you know? It's kind of like, "Oh..."
201:12 - There's also a lot of idle compute in the world, so don't feel too bad.
201:16 - Okay. Do you think new ideas might be needed? Is it possible that the path to AGI, however we define that,
201:24 to solve computer use more generally, to solve biology and chemistry
201:30 and physics—sort of the Dario Amodei definition of AGI? Do you think it's
201:37 possible that totally new ideas are needed? Non-LLM, non-RL ideas?
201:45 What might they look like? We're going into philosophy land a bit.
201:50 - For something like a singularity to happen, I would say yes.
201:54 The new ideas could be architectures or training algorithms,
201:58 fundamental deep learning things. But in that nature, they're pretty hard to predict. I think we won't get very
202:06 far even without those advances. We might get the software solution, but it might stop at software and not do computer use
202:14 without more innovation. So I think that a lot of progress will be
202:17 coming, but if you're gonna zoom out, there's still ideas in the
202:21 next 30 years that are gonna look like that was a major scientific innovation that enabled the
202:28 next chapter of this. And I don't know if it comes in one year or in 15 years.
202:33 - Yeah. I wonder if the bitter lesson holds true for the next 100 years, what that looks
202:37 like. - If scaling laws are fundamental in deep learning, I think the bitter lesson
202:41 will always apply, which is compute will become more abundant, but even within abundant compute, the ones
202:49 that have a steeper scaling law slope or a better offset— like, this is a 2D plot of performance and compute—and like
202:56 even if there's more compute available, the ones that get 100x out of it will win.
203:01 - It might be something like literally computer clusters orbiting Earth
203:07 with solar panels. - The problem with that is heat dissipation. You get all the radiation from the sun
203:13 and don't have any air to dissipate heat. But there is a lot of space to put clusters.
203:17 There's a lot of solar energy there and you could figure out the heat dissipation,
203:21 as there is a lot of energy and there probably could be engineering will to solve
203:25 the heat problem— so there could be. - Is it possible—and we should say that it definitely is possible—
203:30 that we're basically going to be plateauing this year? Not in terms of—
203:39 the system capabilities, but what they actually mean for human civilization. So on the coding front, really nice websites will be built.
203:50 Very nice auto-complete. Very nice way to understand code bases and maybe help debug, but really just a
204:01 very nice helper on the coding front. It can help research mathematicians do some math. It can help you with
204:08 shopping. It's a nice helper. It's Clippy on steroids. What else? It may be a good education tool and all that kind of stuff,
204:19 but computer use turns out extremely difficult to solve. So I'm trying to frame the cynical
204:27 case in all these domains where there's not a really huge economic impact, but realize how costly it is to train these
204:35 systems at every level, both the pre-training and the inference,
204:39 how costly the inference is, the reasoning, all of that. Like, is that possible? And how likely is that, do you think?
204:47 - When you look at the models, there are so many obvious things to improve and
204:51 it takes a long time to train these models and to do this art,
204:55 and it'll take us with the ideas that we have multiple years
204:59 to actually saturate in terms of whatever benchmark or performance we are searching for. It might serve very narrow
205:07 niches; like the average ChatGPT 800 million user might not get a lot of benefit out of this, but it is going to serve different
205:15 populations by getting better at different things. - But I think what everybody's chasing now is a
205:22 general system that's useful to everybody. So, okay, so if that's not... That can plateau, right?
205:28 - I think that dream is actually kind of dying. As you talked about with the
205:31 specialized models where it's like... And multimodal is often... Video generation is a totally different thing. Thing.
205:39 - "That dream is kind of dying" is a big statement, because I don't know if it's dying. If you ask the actual frontier
205:46 lab people, they... I mean, they're still chasing it, right?
205:48 - I do think they are still rushing to get the next model out,
205:52 which will be much better than the... "Much" is a relative term, but it will
205:56 be better than the previous one. And I can't see them slowing down.
206:00 I just think the gains will be made or felt more through not only scaling the model, but
206:07 now... I feel like there's a lot of tech debt. It's like, "Well, let's just put the
206:11 better model in there." Better model, better model. And now people are like, "Okay, let's also
206:17 at the same time improve everything around it too." Like the engineering of the context and inference scaling.
206:23 The big labs will still keep doing that. And now also the smaller labs will catch up, because now they are hiring more.
206:31 There will be more people and LLMs. It's kind of like a circle.
206:35 They also make them more productive and it's just... It's like amplification.
206:39 I think what we can expect is amplification, but not like a change
206:43 of any... not like a paradigm change. I don't think that is true, but
206:47 everything will be just amplified and amplified, and I can see that
206:50 continuing for a long time, you know? - Yeah. I guess my statement that the dream is dying depends on exactly
206:56 what you think it's gonna be doing. Like, Claude Code is a general model
207:00 that can do a lot of things, but it's not necessarily... It depends a lot on integrations. I bet Claude Code could do a fairly good job
207:09 of doing your email, and the hardest part is figuring out how to give information
207:13 to it and how to get it to be able to send your emails. But that's just kind of like... I think it goes back to what is the
207:21 "one model to rule everything" ethos, which is just like a thing in the
207:25 cloud that handles your entire digital life and is way smarter than everybody.
207:29 It's like it's operating in a... So it's an interesting leap of faith to go from "Claude Code becomes that,"
207:39 which in some ways is... There are some avenues for that, but I do think that the
207:46 rhetoric of the industry is a little bit different. - I think the immediate thing we will feel next as a normal
207:52 person using LLMs will probably be related to something trivial, like making figures. Right now, LLMs are terrible at
208:00 making figures. Is it because we are getting served the cheap models
208:04 with much less inference compute than behind the scenes? Maybe some.
208:10 Like, there are some ways to get better figures, but if you ask today,
208:13 ..."Draw a flowchart of X, Y, Z," it's most of the time terrible.
208:18 And it is a very simple task for a human. I think it's almost
208:22 easier sometimes to draw something than to write something. - Yeah, the multimodal understanding does feel like something that is odd...
208:29 ...that it's not better solved. - I think we're not saying one obvious thing that we're not realizing,
208:35 that's a gigantic thing that's hard to measure, which is making all of
208:39 human knowledge accessible— —to the entire world. One thing that is hard to articulate is the huge difference between
208:50 Google Search and an LLM. I feel like I can basically ask an LLM anything and get an answer, and it's doing less and less hallucination.
209:04 And that means understanding my own life, figuring out a career trajectory, solving the problems all around me,
209:12 learning about anything through human history. I feel like nobody's really talking about that,
209:22 because they just immediately take it for granted that this is awesome.
209:25 That's why everybody's using it: because you get answers for stuff.
209:29 Think about the impact across time. This is not just in the United States; it's all across the world.
209:37 Kids throughout the world being able to learn these ideas— the impact that has across time is probably... ...That's the real impact.
209:48 Talk about GDP; it won't be like a leap. It'll be... ...that's how we get to Mars, that's how we build these things,
209:56 that's how we have a million new OpenAIs and all the innovation from there.
210:00 It's this quiet force that permeates everything: human knowledge.
210:06 - I agree with you. In a sense, it makes knowledge more accessible,
210:10 but it also depends on what the topic is. For something like math, you can ask it questions and it answers,
210:21 but if you want to learn a topic from scratch, the sweet spot is still elsewhere.
210:28 There are really good math textbooks laid out linearly, and that is a proven strategy to learn a topic.
210:36 It makes sense, if you start from zero, to use information-dense text to soak it up,
210:43 but then you use the LLM to make infinite exercises. Like, you have problems in a certain area or have questions that something's-
210:51 uncertain about certain things, you ask it to generate example problems, you solve them,
210:59 and you need more background knowledge, you ask it to generate
211:02 that. But then... it won't give you anything, let's say, that is not in the textbook. It's just packaging it differently,
211:12 if that makes sense. But then there are things I feel like where it also adds value in a
211:17 more timely sense, where there is no good alternative besides a human doing it on the fly. For example, if
211:25 you're planning to go to Disneyland and you try to figure out which tickets to buy for which park when, well,
211:32 there is no textbook on that. There is no information-dense resource.
211:36 There's only the sparse internet, and then there is a lot of value in the
211:40 LLM. You just ask it. You have constraints on traveling these days.
211:44 I want to go there and there. Please figure out what I need, when and
211:48 from where, what it costs and stuff like that, and it is a very
211:51 customized, on-the-fly package. And this is like one of a thousand examples of personalized-
211:59 Personalization is essentially pulling information from the sparse internet, the
212:05 non-information-dense thing where there's no better version that exists. It just doesn't exist. You make it
212:11 almost from scratch. - And if it does exist, it's full of- speaking of Disney World, full of-
212:16 what would you call it? Ad slop. It's impossible. Take any city in the world,
212:25 what are the top 10 things to do? An LLM is just way better to ask
212:29 than anything on the internet. - Well, for now, that's because they're subsidized and they're gonna be paid for
212:34 by ads. - Oh my goodness. - It's coming. - No. No. I mean, I'm hoping there's a very clear indication what's an ad
212:44 and what's not an ad in that context. - That's something I mentioned a few years ago. If, I don't know,
212:50 if you are looking for a new running shoe, well, is it a coincidence that Nike maybe comes up first? Maybe, maybe not.
212:58 But I think there are clear laws. You have to be clear about that.
213:02 I think that's what everyone fears. It's the subtle message in there,
213:07 but that also brings us to the topic of ads, where I think this was a thing. Hopefully, I think for-
213:15 in 2025, just because I think it's they're still not making money in other ways right now.
213:22 Having ad spots in there... but the thing is, they couldn't,
213:26 because there are alternatives without ads and people would just flock-
213:31 to the other products. It's also just crazy how- yeah, how they're one-upping each other, spending
213:39 so much money to just get the users. - I think so. Like, some Instagram ads— I don't use Instagram,
213:44 but I understand the appeal of paying a platform to find users who will
213:51 genuinely like your product, and that is the best case of things like Instagram ads.
213:56 But there are also plenty of cases where advertising is very awful for
214:00 incentives, and I think that a world where the power of AI can integrate with that positive view of, "I am a person and
214:07 I have a small business and I want to make the best, I don't
214:11 know, damn steak knives in the world, and I want to sell them to somebody
214:15 who needs them." And if AI can make that sort of advertising thing
214:19 work even better, that's very good for the world, especially with
214:23 digital infrastructure, because that's how the modern web has been
214:27 built. But that's not to say that addicting feeds so that you can show people more content
214:34 is a good thing. So, I think that's even what OpenAI would say, is they want to
214:38 find a way that can make the monetization upside of ads while still giving their users agency.
214:45 And I personally would think that Google is probably going to be better at figuring out
214:49 how to do this, because they already have ad supply and if they figure out how to turn this demand in their Gemini app into
214:56 useful ads, then they can turn it on. And somebody will figure it out—I don't know if it's this year, but there will be
215:04 experiments with it. - I do think what holds companies back right now is really just that the
215:09 competition is not doing it. It's more like a reputation thing.
215:13 It's just, I think people are just afraid right now of ruining or
215:17 losing their reputation, losing users, because it would make headlines if someone launched these ads. But—
215:23 - Unless they were great, but the first ads won't be great because it's a hard problem
215:27 that we don't know how to solve. - Yeah, I think also the first version of that will likely be something like on X,
215:32 like the timeline where you have a promoted post sometimes in between.
215:35 It'll be something where it will say "promoted" or something small, and then there will be an
215:39 image. I think right now the problem is: who makes the first move?
215:43 - If we go 10 years out, the proposition for ads is that you will make so much money on ads by having so many users
215:50 that you can use this to fund better R&D and make better models, which is why
215:54 YouTube is dominating the market for any— Netflix is scared of YouTube. They have the ads,
216:02 they make—I pay $28 a month for Premium. They make at least $28 a month
216:06 off of me and many other people. And they're just creating such a dominant position in video. So I think that's the proposition: that
216:15 ads can make you have a sustained advantage. in what you're spending per user. But there's so much
216:21 money in it right now that somebody starting that flywheel is scary because it's a long-term bet.
216:29 - Do you think there'll be some crazy big moves this year business-wise? Like Google or Apple acquiring Anthropic or something like this?
216:40 - Dario will never sell, but we are starting to see some types of
216:43 consolidation with Groq for $20 billion and Scale AI for almost $30 billion and countless
216:50 other deals like this that are structured in a way that is detrimental to the Silicon Valley ecosystem, which is this
216:58 licensing deal where not everybody gets brought along, rather than a full acquisition that benefits the rank-and-file employee
217:06 by getting their stock vested. That's a big issue for culture to
217:10 address because the startup ecosystem is the lifeblood where,
217:14 if you join a startup, even if it's not successful, it might get acquired on a
217:20 cheap premium and you'll get paid out for this equity. These licensing deals are taking the top talent a lot of the time.
217:27 The deal for Groq to NVIDIA is rumored to be better to the employees, but it is still this antitrust-avoiding thing.
217:35 But I think that this trend of consolidation will continue. Me and many smart people I respect have been expecting
217:43 consolidation to have happened sooner, but it seems like some of these things are starting to turn,
217:49 but at the same time, companies are raising ridiculous amounts of money for
217:55 reasons where I'm like, "I don't know why you're taking that money."
217:59 So it's mixed this year, but some consolidation pressure is starting.
218:04 - What kind of surprising consolidation will we see? You say Anthropic
218:08 is a "never." I mean, Groq is a big one. Groq with a Q, by the way.
218:12 - Yeah. There's just a lot of startups and a very high premium on AI startups.
218:16 So there could be a lot of - that kind of stuff, yeah. - $10 billion range acquisitions, which is really big for a startup that was
218:23 maybe founded a year ago. I think Manus.ai... this company based in Singapore that Meta-founded was
218:30 founded eight months ago and then had a $2 billion exit. I think there will
218:34 be some other multi-billion dollar acquisitions, like Perplexity.
218:39 - Like Perplexity, right? - Yeah, people rumor them to Apple. I think there's a lot of
218:43 of pressure and liquidity in AI. There's pressure on big companies to have outcomes and-
218:50 I would guess that a big acquisition gives people leeway to then tell the next chapter of that story.
218:56 - I guess Cursor—we've been talking about code—somebody acquires Cursor.
219:00 if somebody acquires Cursor... - They're in such a good position by having so much user data.
219:05 And we talked about continual learning. They had one of the most interesting sentences
219:09 in a blog post, which is that they had their new Composer model, which was
219:14 a fine-tune of one of these large Mixture of Expert models from China. You can
219:18 know that by asking it or because the model sometimes responds in Chinese—
219:22 ...which none of the American models do. And they had a blog post where they're like,
219:26 "We're updating the model weights every 90 minutes based on real-world feedback
219:30 from people using it." Which is like the closest thing to real-world RL
219:34 happening on a model, and it's just mentioned in one of their blog posts—
219:36 - That's incredible. - which is super cool. - And by the way, I should say I use Composer a lot because one of the benefits it
219:42 has is that it's fast. - I need to try it 'cause everybody says this.
219:45 - And there'll be some IPOs potentially. You think Anthropic, OpenAI, xAI.
219:51 - They can all raise so much money so easily that they don't feel a need to. So long as fundraising is easy, they're not
219:57 going to IPO because public markets apply pressure. I think we're seeing in China that the ecosystem's a little different with both
220:03 MiniMax and Z.ai applying for, filing IPO paperwork, which will be interesting to see how the Chinese market reacts.
220:11 I actually would guess that it's going to be similarly hypey to
220:15 the US, so long as all this is going and not based on the reality that they're both
220:19 losing a ton of money. I wish more of the gigantic American AI startups were public because it would be very interesting to see how they're spending money
220:27 and have more insight. And also just to give people access to investing in these, because I think they're some of the most
220:35 formidable companies—they're the companies of the era. And the tradition is now for so many of the
220:41 big startups in the US to not go public. It's like we're still waiting for Stripe and
220:45 the IPO, but Databricks definitely didn't. They raised like a Series G
220:49 or something. And I just feel like it's kind of a weird equilibrium for the market where it's like,
220:56 I would like to see these companies go public and evolve in that way that a company can.
221:01 - Do you think 10 years from now some of the frontier model companies are
221:05 still around? Anthropic, OpenAI? - I definitely don't see it as a winner-takes-all unless there truly
221:12 is some algorithmic secret that one of them finds that lets this
221:16 flywheel. Because the development path is so similar for all of them. Google
221:20 and OpenAI have all the same products, and then Anthropic's more
221:24 focused, but when you talk to people it sounds like they're solving a lot of the same
221:28 problems. So I think... and there's offerings that'll spread out. There's a lot
221:31 of... it's a very big cake being made that people are going to take money out of.
221:36 - I don't want to trivialize it, but OpenAI and Anthropic are primarily LLM service
221:43 providers. And some of the other companies like Google and xAI, linked to X, do other stuff
221:51 too. And so it's very possible, if AI becomes more commodified,
221:56 that the companies just providing LLMs will die. - I think the advantage they have is a lot of users, and I think they will
222:04 just pivot. Like Anthropic, I think, pivoted. I don't think they
222:11 originally planned to work on code, but they found, "Okay, this
222:15 is a nice niche, and now we are comfortable and we push on this niche." I can see
222:19 the same thing... Let's say hypothetically, I'm not sure if it will be true, but let's say Google takes all the market share of the general
222:27 chatbot. Maybe OpenAI will then focus on some other sub-topic.
222:31 They have too many users to go away in the foreseeable future.
222:37 - I think Google is always ready to say, "Hold my beer," with AI models.
222:40 - I think the question is if the companies can support the valuations. I see the AI companies being
222:47 looked at in some ways like AWS, Azure, and GCP are, all competing in the same space and all very successful businesses. There's a
222:55 chance that the API market is so unprofitable that they go up and down the stack to products and hardware. They have so much cash that they can build
223:03 power plants and data centers, which is a durable advantage now. But there's
223:07 also a reasonable outcome that these APIs are so valuable and
223:11 so flexible for developers that they become something like AWS. But AWS and Azure are also going to have these APIs, so
223:21 having five or six people competing in the API market is hard. So maybe that's why
223:25 they get squeezed out. - You mentioned "RIP Llama." Is there a path to winning for Meta?
223:32 - I think nobody knows. They're moving a lot, so they're signing licensing deals
223:38 with Black Forest Labs, which is an image generation company,
223:42 or Midjourney. So I think in some ways on the product and consumer-facing AI front, it's too
223:50 early to tell. I think they have some people who are excellent
223:54 and very motivated being close to Zuckerberg. So I think there's still a
223:58 story to unfold there. Llama is a bit different, where Llama was the
224:04 most focused expression of the organization. And I don't see Llama
224:08 being supported to that extent. I think it was a very successful
224:12 brand for them. So they still might participate in the open ecosystem or continue the Llama brand into a different
224:19 service, because people know what Llama is. - You think there's a Llama 5?
224:24 - Not an open-weight one. - It's interesting. I think Llama was the pioneering open-weight model. With Llama 1,
224:34 2, and 3, there was a lot of love. But I think then, hypothesizing or speculating, I think the leaders
224:42 at Meta, like the upper executives, they... I think they got
224:46 very excited about Llama because they saw how popular it was in the community. And then
224:50 I think the problem was trying to, let's say, monetize the open—or not
224:54 monetize the open source, but use it to make a bigger splash. It felt almost forced, like developing these very big Llama 4 models
225:05 to be on top of the benchmarks. But I don't think the goal of Llama models is to be on top
225:12 of the benchmarks beating, let's say, ChatGPT or other models. I think the goal was to
225:16 have a model that people can use, trust, modify, and understand.
225:20 So that includes having smaller models. They don't have to be the best models.
225:23 And what happened was, these models were, of course... the benchmarks
225:28 suggested that they were better than they were because they had specific
225:32 models trained on preferences so that they performed well on benchmarks. That's kind of,
225:36 like, this overfitting thing to force it to be the best. But then at the same
225:39 time, they didn't do the small models that people could use. And I think
225:43 that no one could run these big models then. And then there was kind of a
225:47 weird thing. I think it's just because people got too excited
225:50 about headlines pushing the frontier. I think that's it. - And too much on the benchmarking side.
225:56 - It's too much work. - I think it imploded under internal political fighting and misaligned incentives. The researchers want to
226:04 build the best models, but there's a layer of organization— ...and management that is trying to demonstrate that they do these things. And then
226:11 there are rumors about how, for example, some horrible technical decision was made.
226:19 It just seems like it got so bad that it all just crashed out.
226:24 - Yeah, but we should also give huge props to Mark Zuckerberg. I think it comes from Mark, actually, from
226:32 Mark Zuckerberg, from the top of the leadership, saying open source is
226:35 important. The fact that that leadership exists means there could be a Llama 5, where they learn the lessons from
226:43 benchmarking and say, "We're going to be GPT-OSS—" "...and provide a really awesome library of open source."
226:51 - What people say is that there's a debate between Mark and Alexandr Wang, who is very bright, but much more against open
226:59 source. And to the extent that he has a lot of influence over the AI org, it
227:03 seems much less likely, because it seems like Mark brought him in for a
227:07 fresh leadership eye in directing AI. And if being open or closed is no longer the defining nature of the model, I
227:14 don't expect that to be a defining argument between Mark and
227:18 Alex. They're both very bright, but I just have a hard time understanding all of it because Mark wrote this piece in July of
227:27 2024, which was probably the best blog post at the time, saying "The Case for Open Source AI."
227:36 And then July 2025 came around and it was, "We're reevaluating our relationship with
227:40 open source." So it's just kind of... - But I think also the problem... Not the problem, but I think,
227:46 well, we may have been a bit too harsh, and that caused some of that.
227:50 Because I mean, we as open source developers or the community...
227:54 Even though the model was maybe not what everyone hoped for, it got a lot of backlash. And I think that was unfortunate
228:02 because I can see that as a company, they were hoping for positive headlines. And instead of just getting
228:09 no headlines or positive headlines, in turn they got negative
228:13 headlines. And then it kind of reflected bad on the company.
228:17 I think that is also something where it's maybe a spite reaction,
228:21 almost like, "Okay, we tried to do something nice, we tried to
228:25 give you something cool, like an open source model, and now you are
228:29 kind of being negative about us, even for the company." So in that sense, it looks like, "Well, maybe then we'll change our mind."
228:36 I guess. I don't know. - Yeah, that's where the dynamics of discourse on
228:44 X can lead us, as a community, astray. Because sometimes it feels random. People pick the thing they like and don't
228:51 like. I mean, you can see the same thing with Grok 4.1 and Grok Code Fast 1.0.
228:59 I don't think, vibe-wise, people love it publicly. But a lot of people use it.
229:09 So if you look to Reddit and X, they don't really give it praise from the programming community, but they use it.
229:17 And the same thing with probably Llama. I don't understand the dynamics of
229:21 either positive hype or negative hype. I don't understand it.
229:25 - I mean, one of the stories of 2025 is the US filling the gap of Llama,
229:29 which is the rise of these Chinese open-weight models, models-
229:34 to the point where that was the single issue I've spent a lot of energy on lately,
229:38 trying to do policy work to get the US to invest in this. - So just tell me the story of ADAM.
229:43 - The ADAM Project started as me calling it the American DeepSeek
229:47 Project, which doesn't really work for DC audiences, but it's the story of
229:51 the most impactful thing I can do with my career, which is that these Chinese open-weight models are cultivating a lot of power,
229:58 and there is a lot of demand for building on these open models,
230:02 especially in enterprises in the US that are very cagey about Chinese models.
230:06 - The ADAM Project, American Truly Open Models, is a US-based initiative to build and host high-quality,
230:14 genuinely open-weight AI models and supporting infrastructure
230:17 explicitly aimed at competing with and catching up to China's rapidly advancing open-source AI ecosystem.
230:25 - I think the one-sentence summary would be that... or two sentences.
230:29 One is a proposition that open models are going to be an engine for AI
230:33 research because that is what people start with; therefore, it's important to
230:37 own them. And the second one is, therefore, the US should be building the best models so that the best research happens in the
230:45 US, and those US companies take the value from being the home of where AI research is happening. And without more
230:52 investment in open models—we have plots on the website where it's like, "Qwen,
230:56 Qwen, Qwen, Qwen"—it's all these models that are excellent from these Chinese companies that are cultivating influence
231:04 internationally. I think the US is spending way more on AI, and the ability to create open models that
231:12 are a generation beyond what the cutting edge of closed labs
231:16 costs roughly $100 million, which is a lot of money, but not a lot of money to these companies. Therefore, we need a
231:24 centralizing force of people who want to do this. And I think
231:29 we got engagement from people pretty much across the full stack, whether
231:33 it's policy. - So there has been support from the administration?
231:36 - I don't think anyone technically in government has signed it publicly, but I know people that have worked in AI
231:44 policy, in both the Biden and Trump administrations, are very supportive
231:48 of promoting open-source models in the US. I think, for example,
231:52 AI2 got a grant from the NSF for $100 million over four years, which is the biggest CS grant the
232:00 NSF has ever awarded, and it's for AI2 to attempt this. It's a starting point.
232:05 But the best thing happens when there are multiple organizations building models,
232:09 because they can cross-pollinate ideas and build this ecosystem.
232:13 I don't think it works if it's just Llama releasing models, because Llama could go away. The same thing applies for AI2;
232:21 I can't be the only one building models. It becomes a lot of time spent on
232:29 talking to people, whether in policy... I know NVIDIA is very excited about this. I think Jensen Huang has been talking about the urgency
232:37 for this, and they've done a lot more in 2025, where the Nemotron models are more of a focus. They've started releasing some
232:45 data along with NVIDIA's open models, and very few companies do this, especially of NVIDIA's size, so there
232:52 are signs of progress. We hear about Reflection AI, where they say their
232:57 two billion dollar fundraise is dedicated to building US open models, and I feel
233:01 and their announcement tweet reads like a blog post, right? I think that cultural tide is starting to turn. In July,
233:12 four or five DeepSeek-caliber Chinese open-weight models and
233:16 and zero from the US. That's the moment where I realized, like, "Oh, I guess I have to spend energy on this because nobody else is gonna do it."
233:24 So it takes a lot of people contributing together, and I don't say that,
233:28 the Adam Project isn't the thing that's helping to move the ecosystem, but it's
233:31 people like me doing this sort of thing to get the word out.
233:35 - Do you like the 2025 America's AI Action Plan? That includes open source stuff. The White House AI Action Plan includes
233:43 a dedicated section titled "Encourage Open-Source and Open-Weight
233:47 AI," defining such models and arguing they have unique value for
233:51 innovation and startups. - Yeah. I mean, the AI Action Plan is a plan,
233:55 but largely, I think it's maybe the most coherent policy document that has come out of the administration, and I hope
234:04 that it largely succeeds. I know people that have worked on the AI Action Plan and
234:08 the challenges of taking policy and making it real. I have no idea how
234:12 to do this as an AI researcher, but largely a lot of things in
234:15 that were very real, and there's a huge build-out of AI in the
234:19 country. There are a lot of issues that people are hearing about, from water use to
234:23 whatever, and we should be able to build things in this country, but also,
234:27 we need to not ruin places in our country in the process of building it, and it's worthwhile to spend energy on. I
234:35 think that's a role the federal government plays. They set the agenda.
234:38 And with AI, setting the agenda that open-weight should be a
234:42 first consideration is a large part of what they can do and then people think about it.
234:49 - Also, for education and talent for these companies, it's very
234:52 important because otherwise, if there are only closed models, how do you get the next generation of
234:59 people contributing at some point? Because otherwise, you will
235:02 point only be able to learn after you joined a company. But at that point, how do you hire
235:10 talented people? How do you identify talented people? I think open source
235:14 is essential for a lot of things, but also even just for educating the population and training the next generation of
235:21 researchers. It's the way, or the only way. - The way that I could've gotten this to go more viral was to tell a
235:27 story of Chinese AI integrating with an authoritarian state,
235:31 being ASI and taking over the world, and therefore we need our own American models. But it's very
235:35 intentional why I talk about innovation and science in the US
235:39 because I think it's both more realistic as an outcome, but also
235:43 it's a world that I would like to manifest. - I would say, though, also even any open-weight
235:51 model, I do think, is a valuable model. - Yeah. And my argument is that we should be in a leading position. But I think it's
236:00 worth saying it simply because there are still voices in the AI ecosystem
236:04 that say we should consider banning the release of open models due to safety risks.
236:09 And I think it's worth adding that, effectively, that's impossible without making the US have its own great firewall, which is
236:17 also known to not work that well because the cost for training these models, whether it's one to a hundred million dollars, is
236:24 attainable to a huge amount of people in the world that want to have influence, so these models will be trained all over the
236:32 world. And we want the models, especially when, like, I mean, there are safety concerns, but we want this information and tools to flow
236:42 freely across the world and into the US so that people can use them and
236:46 learn from them. Stopping that would be such a restructuring of our internet
236:50 that it seems impossible. - Do you think maybe in that case the big open-weight models from China are
236:55 actually a good thing in a sense, like, for the US companies? Because maybe
236:59 the US companies you mentioned earlier are usually one generation behind
237:03 in terms of what they release open source versus what they are using? For example,
237:07 gpt-oss might not be the cutting-edge model. Gemini 3 might not
237:10 be, but they do that because they know this is safe to release. But then when they
237:14 see, these companies see, for example, there is DeepSeek-V3.2,
237:18 which is really awesome, and it gets used and there is no backlash, there is no security risk, that could then, again, encourage them to
237:26 release better models. Maybe that, in a sense, is a very positive thing.
237:30 - A hundred percent. These Chinese companies have set things into motion that I
237:33 think would potentially not have happened if they were not all releasing models.
237:38 So I think it was like I'm almost sure that those discussions have been had
237:44 by leadership. - Is there a possible future where the dominant AI models in the
237:49 world are all open source? - Depends on the trajectory of progress that you predict. If you think saturation in
237:54 progress is coming within a few years, so essentially, within the time where financial support is still very good, then open models
238:02 will be so optimized and so much cheaper to run that they'll win out. This
238:06 goes back to open source ideas where so many more people will be
238:10 putting money into optimizing the serving of these open-weight common architectures that they will become standards, and
238:17 then you could have chips dedicated to them, and it'll be way cheaper than
238:21 the offerings from these closed companies that are custom. - We should say that the AI27 report kinda predicts one
238:29 of the things it does from a narrative perspective is that there will be a lot of
238:32 centralization. As the AI systems get smarter and smarter, national security concerns will arise, and
238:39 you'll centralize the labs, and they'll become super secretive, and there'll
238:43 be this whole race - ...from a military perspective of how do you... between China and the US.
238:48 And so all of these fun conversations we're having about LLMs... the generals and the soldiers will
238:56 come into the room and be like, "All right. We're now in the Manhattan
239:00 Project stage of this whole thing." - I think in 2025, '26, '27, I don't think
239:06 something like that is even remotely possible. You can make the same argument
239:10 for computers, right? You can say, "Computers are capable and we don't want
239:14 the general public to get them." Or chips, even AI chips, but you see how Huawei
239:21 makes chips now. It took a few years, but... and I don't think there is a way you can contain knowledge like that.
239:29 I think in this day and age, it is impossible, like the internet. I don't think this is a possibility.
239:37 - On the Manhattan Project thing, I think that a Manhattan Project-like thing for open models would be pretty reasonable, because
239:45 it wouldn't cost that much. But I think that will come. It seems like culturally, the companies are changing. But I agree with Sebastian
239:52 on all of that. I don't see it happening nor being helpful. - Yeah. The motivating force behind the Manhattan Project was
240:02 civilizational risk. It's harder to motivate that for open-source models.
240:08 - There's no civilizational risk. - On the hardware side, we mentioned NVIDIA a
240:14 bunch of times. Do you think Jensen and NVIDIA will keep winning?
240:18 - I think they have to iterate and manufacture a lot. And I think they probably... what they're doing,
240:26 they do innovate, but I think there's always the chance that someone does something fundamentally
240:34 different, gets very lucky, and then does something. But the
240:38 problem is adoption. The moat of NVIDIA is probably not just the GPU. It's more like the CUDA ecosystem, and that
240:45 has evolved over two decades. Even back when I was a grad student, I was in a lab doing biophysical simulations,
240:53 molecular dynamics, and we had a Tesla GPU back then just for the
240:57 computations. It was about 15 years ago now. And they built this up for a long time, and that's the moat, I think.
241:05 It's not the chip itself, although they have the money to iterate, build, and scale. But then it's really
241:13 about compatibility. If you're at that scale, why would you go with something risky where
241:19 there are only a few chips they can make per year? You go with the big one. But then I do think with LLMs
241:25 now, it will be easier to design something like CUDA. It took 15 years because it was hard, but now that we have LLMs, we can
241:33 maybe replicate CUDA. - And I wonder if there will be a separation of training and inference
241:40 compute as we stabilize, and more compute is needed for inference.
241:47 - That's supposed to be the point of the Groq acquisition. And that's why
241:51 part of what Vera Rubin is- where they have a new chip with no high-bandwidth memory, which is one of the-
241:56 or very little, which is one of the most expensive pieces. It's
241:59 designed for pre-fill, which is the part of inference where you essentially do a lot of matrix multiplications. And then you only need
242:07 the memory when you're doing this autoregressive generation, and you have the KV cache
242:11 swaps. So they have this new GPU that's designed for that specific
242:15 use case, and then the cost of ownership per FLOP or whatever is actually way lower. But I think that
242:23 NVIDIA's fate lies in the diffusion of AI still. Their biggest clients
242:26 are still these hyperscale companies. Like, Google obviously
242:30 can make TPUs. Amazon is making Trainium. Microsoft will try to do its own things. And so long as the pace of AI progress is
242:40 high, NVIDIA's platform is the most flexible and people will want that. But if
242:44 there's stagnation, then creating bespoke chips, there's more time to do it.
242:50 - It's interesting that NVIDIA is quite active in trying to develop all kinds of different products.
242:55 - They try to create areas of commercial value that will use a lot of GPUs.
243:01 - Mm-hmm. But they keep innovating and they're doing a lot of
243:05 incredible research, so... - Everyone says the company's super oriented around Jensen and how
243:11 operationally plugged in he is. And it sounds so unlike many other big
243:15 companies that I've heard about. And so long as that's the culture, I think that
243:19 we can expect that to keep progress happening. And it's like he's still in
243:22 the Steve Jobs era of Apple. So long as that is how it operates, I'm pretty optimistic for their
243:31 situation because it's like, it is their top-order problem, and I don't
243:35 know if making these chips for the whole ecosystem is the top goal of all
243:38 these other companies. They'll do a good job, but it might not be as good of a job.
243:43 - Since you mentioned Jensen, I've been reading a lot about history and about singular figures in history. What do you guys think about the
243:51 single man/woman view of history? How important are individuals for steering the direction of history in the tech sector?
243:58 So, you know, what's NVIDIA without Jensen? You mentioned Steve Jobs. What's Apple without Steve Jobs? What's xAI
244:07 without Elon or DeepMind without Demis? - People make things earlier and faster, whereas scientifically,
244:17 many great scientists credit being in the right place at the right time and still making
244:21 the innovation, where eventually someone else will still have the
244:25 idea. So I think that in that way, Jensen is helping manifest this GPU revolution
244:33 much faster and much more focused than it would happen without having a person there. And this is making the whole AI build-out faster. But I
244:41 do still think that eventually, something like ChatGPT would have
244:45 happened and a build-out like this would have happened, but it probably would not have
244:48 been as fast. I think that's the sort of flavor that is applied.
244:55 - These individual people, there are people who are placing bets on something.
244:58 Some get lucky, some don't. But if you don't have these people at the helm, it would be more
245:02 diffused. It's almost like investing in an ETF versus individual
245:06 stocks. Individual stocks might go up or down more heavily than an ETF, which is more balanced. It will eventually go up over time. We'll
245:14 get there. But it's just like, you know, the focus I think is the thing. Passion and focus.
245:19 - Isn't there a real case to be made that without Jensen, there's not a
245:23 reinvigoration of the deep learning revolution? - It could've been 20 years later, is what I would say.
245:30 Or like another AI winter could have come if GPUs weren't around.
245:35 - That could change history completely because you could think of all the other
245:39 technologies that could've come in the meantime, and the focus of human civilization would get...
245:44 Silicon Valley would be captured by different hype. - But I do think there's certainly an aspect where it was all
245:52 planned, the GPU trajectory. But on the other hand, it's also a lot of lucky
245:55 coincidences or good intuition. Like the investment into, let's say, biophysical simulations. I mean, I think it
246:03 started with video games and then it just happened to be good at linear algebra
246:07 because video games require a lot of linear algebra. And then you have the biophysical
246:11 simulations. But still, I don't think the master plan was AI. I think it happened to be Alex
246:19 Krizhevsky. So someone took these GPUs and said, "Hey, let's
246:23 try to train a neural network on that." It happened to work really well, and
246:26 I think it only happened because you could purchase those GPUs.
246:30 - Gaming would've created a demand for faster processors if NVIDIA had gone out of business in the early days.
246:37 That's what I would think. I think that the GPUs would've been different, but I think GPUs would still exist
246:46 at the time of AlexNet and at the time of the Transformer. It was just hard to
246:49 know if it would be one company as successful or multiple smaller
246:53 companies with worse chips. But I don't think that's a 100-year delay. It might be a decade delay.
247:01 - Well, it could be one, two, three, four, five-decade delay. I just can't
247:05 see Intel or AMD doing what NVIDIA did. - I don't think it would be a company that exists.
247:11 I think it would be a different company that would rise. - Like Silicon Graphics or something.
247:15 - So yeah, some company that has died would have done it. - But just looking at it, it seems like these
247:22 singular figures, these leaders, have a huge impact on the trajectory of the world. Obviously, there are incredible teams behind them.
247:31 But, you know, having that kind of very singular, almost dogmatic focus-
247:37 -is necessary to make progress. - Yeah, I mean, even with GPT, it wouldn't exist if there wasn't a person,
247:44 Ilya, who pushed for this scaling, right? - Yeah, Dario Amodei was also deeply involved in that.
247:50 If you read some of the histories from OpenAI, it seems wild thinking about
247:54 how early these people were like, "We need to hook up 10,000 GPUs and take all of
247:57 OpenAI's compute and train one model." There were a lot of people who didn't want
248:01 to do that. - Which is an insane thing to believe. To believe in scaling
248:06 before scaling has any indication that it's going to materialize. Again, singular figures. Speaking of which, 100 years from now, this
248:17 is presumably post-singularity, whatever singularity is. When historians look back at our time now, what
248:24 technological breakthroughs would they really emphasize as the breakthroughs that led to the singularity?
248:32 So far we have Turing to today, 80 years. - I think it would still be computing, like the umbrella term "computing."
248:40 I don't necessarily think that in 100 or 200 years it would be AI. It could still very well be
248:48 computers. We are now taking better advantage of them, but the fact of computing remains.
248:53 - It's basically a Moore's Law discussion. Even the details of CUDA and GPUs won't even be remembered, nor
249:01 will all this software turmoil. It'll just be, obviously, compute.
249:07 - I generally agree, but is the connectivity of the internet and compute able to be merged? Or is it both of them?
249:17 - I think the internet will probably be related to communication.
249:21 It could be a phone, the internet, or satellites. Compute is more like the scaling aspect of it.
249:29 - It's possible that the internet is completely forgotten- -that the internet is wrapped into phone networks, like communication networks.
249:38 This is just another manifestation of that, and the real breakthrough
249:42 comes from increased compute, or Moore's Law, broadly defined.
249:46 - Well, I think the connection of people is very fundamental to it.
249:50 it's like, you can talk to anyone. You want to find the best person in the world for something, they are somewhere in the world. And being able to have
249:58 that flow of information—the AIs will also rely on this. I've been fixating on when I said the dream was dead about
250:06 the one central model. The thing that is evolving is people having many agents
250:10 for different tasks. People already started doing this with different clouds.
250:14 It's described as many AGIs in the data center where each one manages and they talk to each other. And that is
250:22 reliant on networking and the free flow of information. on top of compute. But networking, especially with
250:29 GPUs, is such a part of scaling of compute. The GPUs and the data centers need to talk to each other.
250:36 - Anything about neural networks will be remembered? Like, do you think
250:40 there's something very specific and singular to the fact that it's neural
250:44 networks that's seen as a breakthrough, like a genius, that you're
250:47 basically replicating, in a very crude way, the human mind? The
250:51 structure of the human brain, the human mind? - I think without the human mind, we probably wouldn't have neural
250:58 networks, because it just was an inspiration for that. But on the
251:02 other end, I think it's just so, so different. I mean, it's digital
251:06 versus biological, that I do think it will probably be more grouped as an algorithm.
251:11 - That's massively parallelizable... ...On this particular kind of compute?
251:15 - It could have been like genetic computing; genetic algorithms just
251:19 parallelized. It just happens that this is more efficient and works better.
251:23 - And it very well could be that the LLM, the neural networks, the way
251:26 we architect them now is just a small component of the system that leads to singularity.
251:33 - If you think of it in 100 years, I think society can be changed more with more compute and intelligence because of
251:41 autonomy. But looking at this, what are the things from the Industrial Revolution that we remember? We remember the engine, which is
251:48 probably the equivalent of the computer in this. But there's a lot of other physical transformations that people are aware
251:55 of, like the cotton gin and all these things, these machines that are still known: air conditioning, refrigerators.
252:04 Some of these things from AI will still be known. The word "transformer" could still be known. I would guess that deep
252:12 learning is definitely still known, but the transformer might be evolved away
252:16 from in 100 years with AGI researchers everywhere. But I think deep
252:24 learning is likely to be a term that is remembered. - And I wonder what the air conditioning and refrigeration of the future is that AI brings.
252:32 If we travel forward 100 years from now, we transport there right now,
252:36 what do you think is different? How do you think the world looks different?
252:40 First of all, do you think there are humans? Do you think there are robots
252:44 everywhere walking around? - I do think specialized robots, for sure, for certain tasks.
252:49 - Humanoid form? - Maybe half-humanoid. We'll see. I think for certain things, yes, there will be humanoid robots because it's
252:57 just amenable for the environment. But for certain tasks, it might make sense. What's harder to imagine is how we interact
253:05 with the devices and what humans do with devices. Well, I mean,
253:09 I'm pretty sure it will probably not be the cellphone or the laptop.
253:13 Will it be implants? - I mean, it has to be brain-computer interfaces, right?
253:18 I mean, 100 years from now, given the progress we're seeing now—
253:22 there has to be... unless there's legitimately a complete alteration of how we interact with reality.
253:33 - On the other hand, cars are older than 100 years, right? And
253:37 it's still the same interface. We haven't replaced cars with
253:40 something else. We just made them better, but it's still a steering wheel,
253:44 still wheels, you know? - I think we'll still carry around a physical brick of compute
253:48 because people want some ability to have a private... Like, you
253:52 might not engage with it as much as a phone, but having private information
253:56 that is yours as an interface between the rest of the internet, I think that will still exist. It might not
254:03 look like an iPhone and it might be used a lot less, but I still expect
254:06 people to carry things around. - Why do you think the smartphone is the embodiment of private?
254:11 There's a camera on it. There's— - Private for you, like encrypted messages, encrypted photos...
254:19 know what your life is. I guess it's a question of how optimistic on brain-machine interfaces you are.
254:26 Is all that just going to be stored in the cloud? Your whole calendar?
254:30 It's hard to think about processing all the information that we can process visually
254:37 through brain-machine interfaces presenting something like a
254:41 calendar or something to you. It's hard to just think about knowing, without looking, your email inbox.
254:49 Like you signal to a computer and then you just know your email inbox.
254:53 Is that something that the human brain can handle being piped
254:56 into it non-visually? I don't know exactly how those transformations happen.
255:03 Humans aren't changing in 100 years. I think agency and community are things that people actually want.
255:09 - A local community, yeah. - People you are close to, being able to do things with them
255:14 and being able to ascribe meaning to your life and being able to do things.
255:22 In 100 years, I don't think that human biology is changing away from those on a time scale that we can discuss.
255:30 And I think that UBI does not solve agency. I do expect mass wealth, and I hope that it has spread
255:38 so that the average life looks very different in 100 years. But that's still a lot to happen. If you think about countries
255:46 that are early in their development process to getting access to computing and internet, to build all the infrastructure and have
255:56 policy that shares one nation's wealth with another is... I think it's an optimistic view to see
256:04 all that happening in 100 years- ...while they are still independent entities and not just
256:10 like absorbed into some international order by force. - But there could be just better, more elaborate, more effective
256:17 ... social support systems that help alleviate some levels of basic suffering from the world. You know, the
256:25 transformation of society where a lot of jobs are lost in the short term, I
256:29 think we have to really remember that each individual job that's lost
256:33 is a human being who's suffering. That's like a ... When jobs are lost, the scale is a real tragedy. You can
256:41 make all kinds of arguments about economics or how it's all going to
256:45 be okay. It's good for the GDP, there's going to be new jobs
256:49 created. Fundamentally at the individual level for that human being, that's real suffering. That's a real
256:57 personal sort of tragedy. And we have to not forget that as the technologies are being developed. And also my hope
257:05 for all the AI slop we're seeing is that there will be a greater and greater premium for the fundamental
257:14 aspects of the human experience that are in-person. The things that we all... Like seeing each other, talking together in-person.
257:23 - The next few years are definitely going to be an increased value on
257:27 physical goods and events— ...and even more pressure on slop.
257:32 So it'll be... the slop is only starting. The next few years will be more and more diverse ...versions of slop.
257:38 - They would be drowning in slop. Is that what— - So I'm hoping that society drowns in slop enough to snap
257:44 out of it and be like, "We can't deal with it. It just doesn't matter."
257:48 And then, the physical has such a higher premium on it. - Even like classic examples, I honestly think this is true,
257:57 and I think we will get tired of it. We are already kind of tired of it.
258:01 I mean, even art. I don't think art will go away. You have paintings,
258:05 physical paintings. There's more value, not just monetary value, but just more value appreciation for the actual
258:13 painting than a photocopy of that painting. It could be a perfect digital
258:16 reprint, but there is something when you go to a museum and you look at that art and you see the
258:20 real thing and you just think, "Okay. A human." It's like a craft.
258:24 You have like an appreciation for that. And I think the same is true for writing, for
258:28 talking, for any type of experience... I do unfortunately think it will be like a dichotomy,
258:36 like a fork where some things will be automated. Like, you know, there are not as many paintings as there used to be, you know, 200 years ago.
258:44 There are more photographs, more photocopies. But at the same time, it won't go away. There will be value in that.
258:52 I think the difference will just be, you know, what's the proportion of
258:55 that. But personally, I have a hard time reading things where I obviously see
259:01 it's obviously AI generated. I'm sorry. It might—it might be really good information there, but I'm just like, "Nah, not for me."
259:09 - I think eventually they'll fool you, and it'll be on platforms that
259:12 give ways of verifying or building trust. So you will trust that Lex is not AI generated, having been here. So then you have trust in this
259:21 channel. But it's harder for new people who don't have that trust.
259:25 - Well, that will get interesting because I think fundamentally it's a
259:29 solvable problem by having trust in certain outlets that they won't do it, but it's all going to be trust-based.
259:37 There will be systems to authorize, "Okay, this is real. This is not real." There will be some
259:41 telltale signs where you can obviously tell this is AI generated and
259:45 this is not. But some will be so good that it's hard to tell, and
259:49 then you have to trust. And well, that will get interesting and a bit problematic.
259:54 - The extreme case of this is to watermark all human content.
259:58 So all photos that we take on our own have some watermark until they are edited
260:02 or something like this. And software can manage communications with the
260:07 device manufacturer- device manufacturer to maintain human editing—
260:11 which is the opposite of the discussion to try to watermark AI images. And then
260:16 you can make a Google image that has a watermark and use a different Google tool to remove it.
260:20 - Yep. It's going to be an arms race, basically. - And we've been mostly focusing on the positive aspects of
260:27 AI. All the capabilities that we've been talking about can be used to destabilize human civilization with even just
260:35 relatively dumb AI applied at scale, and then further, superintelligent
260:41 AI systems. Of course, there's the sort of doomer take that's important to consider as we develop these
260:49 technologies. What gives you hope about the future of human civilization, given everything we've been talking about? Are we going to be okay?
260:59 - I think we will. I'm definitely a worrier, both about AI and non-AI things. But humans do tend to find a way. I think that's what
261:10 humans are built for: to have community and find a way to figure out
261:14 problems. That's what has gotten us to this point. And to think that
261:19 the AI opportunity and related technologies is really big. And I think that there's big social and political problems to
261:28 help everybody understand that. And I think that's what we're staring at a
261:32 lot of right now, is like the world is a scary place, and AI is a very
261:35 uncertain thing. And it takes a lot of work that is not necessarily building things. It's like telling people and
261:43 understanding people, that the people building AI are historically
261:47 not motivated or wanting to do. But it is something that is probably doable. It just will
261:54 take longer than people want. And we have to go through that
261:57 long period of like hard, distraught AI discussions if we want to have the lasting benefits.
262:04 - Yeah. Through that process, I'm especially excited that we get a
262:07 chance to better understand ourselves, us at the individual level as humans and at the civilization
262:15 level, and answer some of the big mysteries, like what is this whole consciousness thing going on
262:22 here? It seems to be truly special. Like, there's a real miracle in
262:26 our mind. And AI puts a mirror to ourselves and we get to answer some of the big questions about like, what is this
262:34 whole thing going on here? - Well, one thing about that is also what I do think makes us very different
262:39 from AI and why I don't worry about AI taking over is, like you said, consciousness. We humans, we decide what we want to
262:47 do. AI in its current implementation, I can't see it changing. You have to tell it what to do. And so you
262:55 have still the agency. It doesn't take the agency from you because
262:59 it becomes a tool. You can think of it as a tool. You tell it what to
263:03 do. It will be more automatic than other previous tools. It's certainly more powerful than a hammer, it can figure things out, but
263:10 it's still you in charge, right? So the AI is not in charge, you're in
263:14 charge. You tell the AI what to do and it's doing it for you.
263:17 - So in the post-singularity, post-apocalyptic war between humans and machines, you're saying humans are worth fighting for?
263:27 - 100%. I mean, this is... The movie Terminator, they made in
263:31 the '80s, essentially, and I do think, well, the only thing I can see going wrong is, of course, if
263:39 things are explicitly programmed to do the thing that is harmful, basically.
263:43 - I think actually in that, in a Terminator type of setup, I think humans win.
263:49 I think we're too clever. It's hard to explain how we figure it out, but we do.
263:56 And we'll probably be using local LLMs, open source LLMs to help fight the machines.
264:04 I apologize for the ridiculousness. Like I said, Nathan already knows I've been a big fan of his for a long time. Been
264:11 a big fan of yours, Sebastian, for a long time, so it's an honor to finally meet
264:15 you. Thank you for everything you put out into the world. Thank you for the excellent books you're
264:19 writing. Thank you for teaching us. And thank you for talking today. This was fun.
264:26 - Thank you for inviting us here and having this human connection, which is actually-
264:30 - Extremely valuable- human connection. Thanks for listening to this conversation with Sebastian Raschka and Nathan
264:37 Lambert. To support this podcast, please check out our sponsors in the
264:41 description, where you can also find links to contact me, ask
264:44 questions, give feedback and so on. And now let me leave you with some words from Albert Einstein.
264:52 "It is not that I'm so smart, but I stay with the questions much longer." Thank you for listening, and hope to see you next time.
$

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

@lexfridman 4:25:13 26 chapters
[AI agents and automation][content creation and YouTube][developer tools and coding]
// chapters
// description

Nathan Lambert and Sebastian Raschka are machine learning researchers, engineers, and educators. Nathan is the post-training lead at the Allen Institute for AI (Ai2) and the author of The RLHF Book. Sebastian Raschka is the author of Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch). Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep490-sb See below for timestamps, transcript, and to give feedback, submit questions, contact L

now: 0:00
// tags
[AI agents and automation][content creation and YouTube][developer tools and coding]