0:05 Let's talk about timelines. Uh specifically timelines to AGI or ASI.
0:13 Is it fair like as a starting point to say that nobody really agrees on the
0:18 definitions of AGI and ASI? I kind of think there's a lot of disagreement, but
0:21 among I've been getting push back where a lot of people kind of say the same thing,
0:26 which is like a a thing that could reproduce most digital economic work. So
0:31 like the remote remote worker is a fairly reasonable example. And I think
0:36 OpenAI's definition is somewhat um related to that which is like an AI that
0:40 can do a lot of econ like a certain number of economically valuable tasks
0:43 which I don't really love as a definition but I think it could be a
0:48 grounding point because um language models today while immensely powerful
0:53 are are not this remote worker drop in and there are things that you could
0:57 think of that are could be done by an AI that are way harder than remote work
1:01 which are like solving a finding an unexpected scientific ific discovery
1:04 that you couldn't even posit. Which would be an example of something that
1:07 somebody says is like an artificial super intelligence problem or like
1:14 taking in all medical records and finding linkages across certain
1:18 illnesses that people didn't know or figuring out that some common drug can
1:22 treat some niche cancer. Like they would say that that is like a super
1:25 intelligence thing. So these are kind of natural tears. My problem with it is
1:30 that it becomes deeply entwined with like the quest for meaning of AI and
1:35 this religious aspects to it. So there's kind of different there's different
1:38 paths you can take it. And I don't even know if the remote work is a good
1:43 definition cuz what exactly is that? It's like perfect tool use. I actually I
1:47 mean I like I don't know if you like the originally titled AI 27 report. They
1:54 focus more on code and research taste. So the the target there is the
1:59 superhuman coder. So they have several several milestone systems. Superhuman
2:04 coder, superhuman AI researcher, then super intelligent AI researcher and uh
2:09 then the full ASI artificial super intelligence. But the after you develop
2:14 the superhuman coder, everything else falls quickly. There the task is to have a fully
2:23 autonomous like automate coding. So any kind of coding you need to do in
2:28 order to perform research is fully automated and from there humans would be
2:33 doing AI research together with that system and they will quickly be able to
2:38 develop a system that's actually can do the research for you. That's the idea
2:43 and then and initially their prediction was 2027 28 and now they've pushed it
2:50 back by 3 to four years to to uh 2031 mean prediction. probably my prediction
2:55 is even beyond 2031. But at least you can in a concrete way think about how difficult it is to fully
3:02 automate programming. >> Yeah, I disagree with some of their
3:07 presumptions and dynamics on how it would play out. Um, but I think they did
3:10 a good like they did good work in the scenario defining milestones that are
3:14 concrete and to tell a useful story which is why the reach for this AI 2027
3:20 document well transcended Silicon Valley is because they told a good story and
3:23 they did a lot of rigorous work to do this. I think the camp that I fall into
3:28 is that like AI is like so-called jagged which will be excellent at some things
3:32 and really bad at some things. I think that when they're close to this
3:37 automated software engineer, what it will be good at is that traditional ML
3:41 systems and front end the model is excellent at. But the distributed ML,
3:45 the models are actually really quite bad at because there's so little training
3:48 data on doing large scale distributed learning and things and this is
3:51 something that we already see and I think this will just get amplified and
3:55 then it's kind of messier in these trade-offs and then there's like how do
3:59 you think AI research works and so on. So you think basically superhuman coder
4:04 is almost unachievable meaning like because of the jagged nature of the
4:08 thing you're just always going to have gaps in capabilities. I
4:12 >> I think it's assigning completeness to something where the models are kind of
4:15 superhuman at some types of code and I think that will continue and people are
4:19 creative so they'll utilize this like incredible abilities and like to fill in
4:24 the weaknesses of the models and move really fast. There'll always kind of be
4:27 this I've perceived for a long time this dance between the humans are enabling
4:31 this thing that the model can't do and the best the best AI researchers the
4:35 ones that can enable this superpower and I think this aligns like to what we
4:39 already see I think like cloud code for building a website you can stand up a
4:42 beautiful website in a few hours or do data analysis and I don't think it's
4:45 going to keep getting better at these things and it'll pick up some new code
4:49 skills and stuff that it'll get along the way and kind of linking to what's
4:56 happening in in big tech is This AI 2027 report is like it leans into the
5:00 singularity idea where I think research is messy and social and largely in the
5:05 data in ways that AI models can't process. But like what we do have today
5:10 is really powerful and these tech companies are all collectively
5:13 buying into this with tens of billions of dollars of investment. So like we are
5:18 going to get some much better version of chat GBT a much better version of cloud
5:22 code than we already have. I think that it's just like hard to predict where
5:26 that is going. But the like bright clarity of that future is why
5:30 some of the most powerful people in the world are putting so much money into
5:34 this. And I think it's just kind of small differences between like we don't
5:38 actually know what a better version of TIGBT is, but also like can it automate
5:44 AI research? I would say probably not at least in this time frame. like big tech
5:48 is going to spend $und00 billion much faster than we get a automated AI
5:53 researcher that enables a AI research singularity. >> So you think your prediction would be
5:58 what like if this is even a useful milestone we're more than 10 years out.
6:03 >> I would say less than that on the software side but I think longer than
6:08 that on the things like research. Let's just like for fun try to imagine a world where all
6:15 software writing is fully automated. Like can you imagine that world?
6:19 >> By the end of this year, the amount of software that'll be automated will be so
6:23 high. But it's like it'll be the things of like you're trying to train a model
6:27 with RL and you need to have multiple bunches of GPUs communicating with each
6:31 other. That'll still be hard, but I think it'll be much easier. One of the
6:35 ways to think about this, the full automation of programming
6:41 is just think of like lines of useful code written. The fraction of that to
6:46 the number of humans in the loop. So presumably there'll be for a long time
6:50 humans in the loop of software writing is just be fewer and fewer relative to
6:55 the amount of code written, right? And the the the SC superhuman coder I think
7:00 the the presumption there is it goes to zero the number of humans in the loop.
7:05 What does that world look like when the number of humans in the loop is in the
7:09 hundreds not in the hundreds of thousands? >> I think software engineering will be
7:15 driven more to system design and goals of outcomes where I do think software is
7:19 largely going to be come on. I think this has been happening over the last
7:23 few weeks where people have gone from a month ago of like oh AI agents are kind
7:27 of slopp which is a famous carpenty quote to like the what is a little bit
7:33 of a meme of like the industrialization of software when anyone can just create
7:37 software at their fingerprints like I do think we are closer to that side of
7:41 things and it takes direction and like understanding how the systems work to
7:45 extract that best from the language models and I think it's hard to like
7:49 accept the gravity of how much is going to change with software development and
7:52 how many more people can do things without ever looking at it. I think
7:55 what's interesting is to think about whether these systems will be um
7:59 independent like completely independent in the sense that well I have no doubt
8:03 that LMS will kind of at some point solve coding in a sense like calculators
8:08 solve calculating right so at some point humans developed a tool that you know
8:11 you never [clears throat] need a human to calculate that number you just type
8:15 it in and it's an algorithm you you can do it in a in that sense and I I think
8:20 that's the same probably for coding but the question is so I think what will
8:24 happen is yeah you will just say build that website, it will make a really good
8:27 website and then you maybe refine it. But will it do things independently
8:34 where so will you be still having humans asking the AI to do something like will
8:39 there be a person say build that website or will there be AI that just builds
8:43 websites or something or whatever? >> I think using talking about building
8:47 websites is the >> too simple. It's just like there there's
8:50 the the the problem with websites and the problem with the web, you know, HTML
8:54 and all that kind of stuff. >> It's very resilient to just
8:58 >> slop. It will show you slop as good as showing slop.
9:02 >> I would rather like think of like safety critical systems like
9:09 >> uh asking AI to end to end generate something that manages logistics
9:14 >> or manages cars, a fleet of cars, all that kind of stuff. So end to end
9:17 generates that for you. I think a more intermediate example is take something
9:22 like Slack or Microsoft Word. I think if the organizations allow it, AI could
9:27 very easily implement features end to end and do a fairly good job for like
9:31 things that you want to try. You want to add a new like tab in Slack that you
9:36 want to use. And I think AI will be able to do that pretty well.
9:38 >> Actually, that's a really great example. How far away are we from that?
9:42 >> Like this year. >> See, I don't I don't know. I don't know.
9:47 I I guess I don't know how bad production code bases are, but I think
9:49 that >> within like on the order of low years, a lot of people are going to be pushed to
9:55 be more of like a designer and product manager where you have multiple of these
10:00 agents that can try things for you and they might take one to two days to
10:03 implement a feature or attempt to fix a bug and you have these dashboards which
10:07 I think Slack is actually a good dashboard where your agents will talk to
10:11 you and you'll then give feedback. But things like like I make a website, it's
10:16 like you want to make a logo that's passable. Like I think these like
10:20 cohesive design things and this style is going to be very hard for models and
10:25 deciding on what to add the next time. >> I just Okay, so I hang out with a lot of
10:30 programmers and some of them are a little bit on the skeptical side in
10:34 general. That's just vibe wise they're like that. I just think there's a lot of complexity
10:40 involved in adding features to complex systems. Like if you look at the browser
10:45 Chrome, >> if I wanted to add a feature, if I wanted to have tabs as as opposed to up
10:52 top, I want them on the left side. >> Mhm. >> interface, right? You I think we're not
10:57 this is not a next year thing. One of the Claude releases this year, one of
11:01 their tests was we give it a piece of software and leave claude to run to
11:05 recreate it entirely and it could almost rebuild scr like slack from scratch just
11:10 given the parameters of the software and left in a sandbox environment.
11:15 >> The scratch part I like almost better. >> So it might be that the smaller newer
11:18 companies are advantaged and they're like we don't have to have the bloat and
11:24 complexity and therefore this future exists. And I think this gets to the
11:27 point that you mentioned that some people are you talk to are skeptical and
11:32 I think that's not because the LM can't do XYZ. It's because people don't want
11:37 it to do it this way. Some of that could be a skill issue on the human side.
11:41 Unfortunately, we have to be honest with ourselves and some of that could be an
11:45 under specification issue. So programming like you're like you're just assuming
11:52 this is like in in relationships and friendships communication type of issue.
11:56 You're assuming the LM somehow is supposed to read your mind. I think this
11:59 is where spec driven design is really important. Like you just using natural
12:03 language specify like what you want. >> I think that's like if you talk to
12:06 people at the labs they use these in their training in production code. Like
12:11 cloud code is built with cloud code and they all use these things extensively
12:15 and Daario talks about how much clouds code own and it's like these people are
12:20 slightly ahead in terms of the capabilities they have and they probably
12:26 spend on inference they could spend 10 to 100 plus x as much as we're spending
12:30 like we're on a lowly$100 or $200 a month plan like they truly let it rip
12:35 and I think that that like with the pace of progress that we have it it seems
12:40 like like where a year ago we didn't have cloud code and we didn't really
12:43 have reasoning models and it's like the difference between sitting here today
12:47 and what we can do with these models and it seems like there's a lot of like
12:51 there's a lot of low hanging fruit to improve them the failure modes are
12:55 pretty dumb. It's like Claude you tried to use the CLI command I don't have
12:59 installed 14 times and then then I sent you the command to run. It's like that
13:03 thing from a modeling perspective is pretty fixable. >> So I I
13:08 >> I agree with you. be been uh becoming more and more bullish in general.
13:13 Speaking to what you're articulating, I think it is a human skill issue. So,
13:19 Anthropic is leading the way in or other companies in understanding how to best
13:24 use the models for programming. Therefore, they're effectively using
13:27 them. I think there's a lot of programmers on the outskirts. They're
13:32 like they don't I mean, there's not a really good guide on how to use them.
13:36 people trying to figure it out. Exactly. >> It might be very expensive. Like it
13:39 might be that the entry point for that is $2,000 a month, which is only tech
13:43 companies and rich people, which is like like that could be it,
13:46 >> but it might be worth it. I mean, if if if the final result is is a working
13:50 software system that might be worth it. But by the way, it's funny how we
13:53 converge from the discussion of timeline to AGI to something more pragmatic and
13:58 and useful. Is there anything concrete and interesting and useful and profound
14:04 to be said about timeline to AGI and ASI or are these discussions a bit too
14:10 detached from the day-to-day? There's interesting bets. So there's a lot of
14:13 people trying to do reinforcement learning with verifiable rewards but in
14:17 real scientific domains where there's startups that are spending like they
14:19 have hundreds of millions of dollars of funding and they have wet labs where
14:23 they're having language models propose hypotheses that are tested in the real
14:27 world. And I I would say that I think they're very early or they're early, but
14:31 with the pace of progress, it's like >> yeah, >> maybe they're early by six months and
14:35 they make it because they were there first or maybe they're early by eight
14:38 years. You don't really know. So I think that that type of moonshot to um branch
14:46 this momentum into other other sciences is like okay like that would be very
14:51 transformative if like alphafold moments happen in all sorts of other scientific
14:56 domains by like a startup solving this. I think there are startups I think maybe
15:00 harmonic is one where they're going all in on language models plus lean for
15:03 math. I think you had another podcast guest where we talked about this
15:07 recently and it's like we don't know exactly what's going to fall out of
15:12 spending $100 million on that model and most of them will fail but a couple of
15:17 them might be big breakthroughs that are very different than try GPT or cloud
15:23 code type software experiences like a tool that's only good for a PhD
15:28 mathematician but makes them 100x effective like >> okay I agree I think this will happen in
15:34 a lot of domains especially also like domains that have a lot of um you know
15:40 resources like finance and legal and pharmaceutical companies but then again
15:44 is it really AGI again because we are now specializing it again and then again
15:48 is it really that much different from back in the day how we had specialized
15:52 algorithms I think it's just the same thing more way more sophisticated but I
15:57 don't know is there a threshold when we call it AGI I guess I think the the real
16:01 cool thing is here that we have like the foundation models that we can
16:04 specialize. I think that that's like the breakthrough at some point. Right now, I
16:08 think we're not there yet because well, first it's too expensive, but also you
16:12 know like chd doesn't just give away their chbd to customize it. I think once
16:18 that's going to be true in some way and I think I can imagine this as a business
16:22 model that JPD open says at some point like hey you know bank of America for
16:26 100 million we will do your custom model or something like that and I think that
16:31 will be the huge economic uh value ad. The other thing though is also companies
16:36 I mean right now what is the differentiating factor I mean if
16:40 everyone uses the same LLM if everyone uses JPD they will all do the same thing
16:45 again I mean then well it's everyone is moving in lock step but usually
16:48 companies they want to have a competitive advantage and I think
16:52 there's no way around using some of their private data and experimenting and
16:56 maybe specializing it's going to be interesting yeah >> sitting in the pace of progress it does
17:01 just feel like things are coming. I don't think the AGI and ASI thresholds
17:06 are particularly useful. I I think I guess the real question and this takes us to the remote
17:14 worker thing is when are we going to see a a big obvious leap in e economic
17:21 impact cuz currently there's not been an obvious leap in economic impact of LLM
17:28 models for example and that's you know aside from AGI or ASI or all that kind
17:32 of stuff there's a real question of like when are we going to see a GDP like
17:36 >> Mhm. Mhm. >> junk. >> Yeah. It's like what is the GDP made up
17:41 of? Like a lot of it is like financial services. So like I don't I don't know
17:46 what this is. It's just hard for me to think about the >> GDP bump. But like I say that software
17:53 development becomes valuable in a different way when you no longer have to
17:58 look at the code anymore. So when when it is like cloud will make you a small
18:02 business which is essentially cloud can set up your website your bank account
18:05 your email and your whatever else and like you just have to express like what
18:10 you're trying to put into the world like that's not just a enterprise market but
18:15 it is a hard like I don't know how you get people to try doing that I guess if
18:20 chatbt can do it like people are trying >> I think it boils down to the the
18:24 scientific question of how hard is tool use to solve Because a lot of the stuff you're
18:30 implying, the remote work stuff is to tool use. It's like how computer use
18:35 like how you have an LM that goes out there, this agentic system, and does
18:41 something in the world and only screws up 1% of the time. >> Computer use is a good example of what
18:46 labs care about and we haven't seen a lot of progress on. We saw multiple
18:52 demos in 2025 of like Claude can use your computer or OpenAI had KUA and they
18:56 all suck. >> Yeah. >> So like they're also investing money in
19:00 this. And I think that would be a good example where that's actually something
19:03 where it just seems pretty like taking over the whole screen
19:09 seems a lot harder than having an API that they can call in the back end. And
19:12 some of that is you have to then set up a different environment for the model to
19:15 work in. Like they're not working on your MacBook. they are individually
19:21 interfacing with Google and Amazon and Slack and they handle all these things
19:24 in a very different way than humans do. So some of those might be structural
19:29 blockers also like specification wise I think the problem is also for you know
19:34 arbitrary tasks uh well you still have to specify what you want your LLM to do
19:38 and how do you do that in a what is the environment how do you
19:41 specify you can say what the end goal is but if it can't solve the end goal with
19:46 LLMs if you ask it for text you can always you know clarify do substeps what
19:51 is how how do you put that information into a system that let's say books a
19:54 travel trip for you you can say, "Well, you screwed up my credit card
19:58 information." But even to get it to that point, like how do you like as a user
20:04 guide the model before like it can't even attempt that? I think the interface
20:07 is really hard. >> Yeah, it has to learn a lot about you
20:10 specifically >> and about this goes to continue learning
20:16 about the general mistakes that are made throughout and then mistakes that are
20:20 made through you. All the AI interfaces are getting set up to ask humans for
20:25 input. I think cloud code we talk about a lot. It asks feedback on questions. If
20:28 it doesn't have enough specification on your plan or your desired, it starts to
20:33 ask questions. Would you rather? Um, we talked about memory which saves across
20:36 chats which >> it's first implementation is kind of odd
20:41 where it be like it'll mention my dog's name or something like in a chat. I'm
20:45 like you didn't need to be subtle about this like I don't care. But the like
20:48 things that are emerging are chat GBT has the pulse feature >> which is like um a curated couple
20:55 paragraphs with links to something to look at or to talk about and people talk
20:58 about how the language models are going to ask you questions which I think is a
21:04 very it's probably going to work. The language model is like it knows you had
21:07 a doctor appointment or something. It's like, hey, how are you feeling after
21:10 that? Which is like >> um again goes into the territory of
21:14 humans are very susceptible to this and there's a lot of social change to come.
21:18 But also like they're experimenting with having the models engage. Some people
21:22 really like this pulse feature which is it processes your chats and
21:25 automatically searches for information and puts it in the chat GBT app. So
21:30 there's a lot of things coming. >> I used that feature before and I always
21:34 feel bad because [laughter] it does that every day and I rarely check it out.
21:39 It's like how much money like I mean comput is burned on something I don't
21:42 even look at you know where it's like it's kind of >> there's also a lot of idle comput in the
21:46 world so don't feel too