// transcript — 958 segments
0:00 Introduction to Nicole Forsgren
0:01 A lot of companies are trying to measure productivity for their teams.
0:05 >> Most productivity metrics are a lie. If the goal is more lines of code, I can
0:08 prompt something to write the longest piece of code ever. It's just too easy
0:12 to gain that system. >> How do I know if my edge team is moving
0:16 fast enough if they can move faster if they're just not performing as well as
0:18 they can? >> Most teams can move faster, but faster for what? We can ship trash faster every
0:25 single day. We need strategy and really smart decisions to know what to ship.
0:28 One of the biggest issues we're going to probably have with AI is learning how
0:32 much to trust code that it generates. >> We can't just put in a command and get
0:36 something back and accept it. We really need to evaluate it. You know, are we
0:38 seeing hallucinations? What's the reliability? Does it meet the style that
0:42 we would typically write? >> So much of the time is now going to be
0:45 spent reviewing code versus writing code. >> There's some real opportunity there to
0:48 not just rethink workflows, but rethink how we structure our days and how we
0:51 structure our work. Now, we can also make a 45minute work block useful
0:55 because getting into the flow is actually kind of handed off at least in
0:59 part to the machine or the machine can help us get back into the flow by
1:02 reminding us of context and generating diagrams of the system.
1:05 >> What's just like one thing that you think ange team, a product team can do
1:09 this week, next week to get more done. >> Honestly, I think the best thing you can
1:12 do today, my guest is Nicole Forsrren. With so much talk about how AI is increasing
1:19 developer productivity, more and more people are asking, "How do we measure
1:23 this productivity gain? And are these AI tools actually helping us or hurting how
1:28 our developers work?" Nicole has been at the forefront of this space longer than
1:32 anyone. She created the most used frameworks for measuring developer
1:36 experience called Dora and Space. She wrote the most important book in the
1:40 space called Accelerate and is about to publish her newest book called
1:44 Frictionless, which gives you a guide to helping your team move faster and do
1:48 more in this emerging AI world. Her core thesis is that AI indeed accelerates
1:53 coding. But developers aren't speeding up as much as you think because they
1:56 still have to deal with broken builds and unreliable tools and processes and a
2:00 bunch of new bottlenecks that are emerging. In our conversation, we chat
2:04 about her current, best, and very specific advice for how to measure
2:09 productivity gains from AI, signs that your team could be moving faster, what
2:13 companies get wrong when trying to measure engineering productivity, how AI
2:17 tools are both helping and hurting engineers, including getting into flow
2:21 states, her seven-step process for setting up a developer experience team
2:24 at your company, how to get buyin and measure the impact of a team like this,
2:28 and a ton more. This episode is for anyone looking to improve the
2:32 performance of their engineering teams. If you enjoy this podcast, don't forget
2:35 to subscribe and follow it in your favorite podcasting app or YouTube. It
2:40 helps tremendously. Also, if you become an annual subscriber of my newsletter,
2:45 you get a year free of 15 incredible products including lovable, replet,
2:49 bolt, nadam, linear, superhuman, dscript, whisper flow, gamma,
2:52 perplexity, warp, granola, magic patterns, raycast, chapard, and mobin.
2:56 Head on over to lennisnewsletter.com and click product pass. With that, I bring
3:01 you Nicole Forscreen. This episode is brought to you by Mercury. I've been banking with Mercury
3:07 for years and honestly, I can't imagine banking any other way at this point. I
3:12 switched from Chase and holy moly, what a difference. Sending wires, tracking
3:16 spend, giving people on my team access to move money around. So freaking easy.
3:21 Where most traditional banking websites and apps are clunky and hard to use,
3:25 Mercury is meticulously designed to be an intuitive and simple experience. And
3:29 Mercury brings all the ways that you use money into a single product, including
3:34 credit cards, invoicing, bill pay, reimbursements for your teammates, and
3:38 capital. Whether you're a funded tech startup looking for ways to pay
3:42 contractors and earn yield on your idle cash, or an agency that needs to invoice
3:46 customers and keep them current, or an e-commerce brand that needs to stay on
3:50 top of cash flow and excess capital, Mercury can be tailored to help your
3:53 business perform at its highest level. See what over 200,000 entrepreneurs love
3:59 about Mercury. Visit mercury.com to apply online in 10 minutes. Mercury is a
4:04 fintech, not a bank. banking services provided through Mercury's FDICins
4:08 insured partner banks. For more details, check out the show notes. Here's a
4:12 puzzle for you. What do OpenAI, Cursor, Perplexity, Verscell, Platt, and
4:16 hundreds of other winning companies have in common? The answer is they're all
4:21 powered by today's sponsor, Work OS. If you're building software for
4:24 enterprises, you've probably felt the pain of integrating single signon, skim,
4:29 arbback, audit logs, and other features required by big customers. Work OS turns
4:34 those deal blockers into drop-in APIs with a modern developer platform built
4:38 specifically for B2B SAS. Whether you're a seedstage startup trying to land your
4:42 first enterprise customer or a unicorn expanding globally, work OS is the
4:46 fastest path to becoming enterprise ready and unlocking growth. They're
4:50 essentially Stripe for enterprise features. Visit workos.com to get
4:54 started or just hit up their Slack support where they have real engineers
4:58 in there who answer your questions super fast. Work OS allows you to build like
5:02 the best with delightful APIs, comprehensive docs, and a smooth
5:07 developer experience. Go to work os.com Nicole, thank you so much for being here
5:09 The concept of developer experience (DevEx)
5:16 and welcome to the podcast. >> Thank you. It's so good to be here.
5:20 >> It's so good to have you back. I was just watching our first episode, which
5:24 we did two and a half years ago. I was watching it and I was both shocked and
5:29 not shocked that we barely talked about AI. The episode was called how to
5:33 measure and improve developer productivity and we got to AI barely
5:38 like an hour in and we're just like hm I wonder what's going to happen with AI
5:40 and productivity. Does that just blow your mind? >> Yeah, because I mean it was just hitting
5:46 the scene. It was the topic of so much conversation and at the same time so
5:52 many things don't change, right? So many things are still important. So many
5:55 things are the same. Um, yeah, it's also a little wild that it's been two and a
5:59 half years. Where's where go? Time is a social construct. >> Yeah, it was like most of our
6:04 conversation was just like questions like, well, how might this impact
6:08 people? How will we change the way we build product? And now it basically was
6:12 not it was barely a thing back then. Now it's the only thing that I imagine
6:15 people want to talk about when they talk about engineering productivity. That's
6:18 what we're spending a lot of our time focusing on today. The reason I'm
6:21 excited about this conversation, it feels like there's been so much money
6:26 poured into AI tools, increasing productivity. All the fastest growing
6:29 companies in the world are these engineering AI tools. And now more and
6:32 more people are just asking this question of just like what gains are we
6:36 getting out of this? How much is this actually helping us be more productive?
6:40 How do we become more productive? You've been at the center of this world for
6:44 longer than anyone. You've uh invented so many of the frameworks that people
6:47 rely on now. So I'm really excited to have you back and to talk about this
6:50 stuff. I want to talk I want to start with just like this term Devx which is
6:54 uh something that comes up a lot in this in this whole space. So we're gonna and
6:57 we're going to hear this term a bunch in this conversation. Can you just explain
7:00 what is Devx? This term Devx. >> So Devx is uh developer experience. And
7:07 when we think about developer experience it's we're really talking about what
7:12 it's like to build software daytoday for a developer. Right? So the friction that
7:16 they face, the workflows that they have to go through, um any support that they
7:22 have and it's important because when DevX is poor, everything else just isn't
7:26 going to help, right? The best processes, the best tools, the best
7:31 whatever magic you have, right? If if the DevX is bad, everything kind of
7:34 takes. >> And so within DevX is productivity. And I think the key insight that you had and
7:40 other folks in the space of about it is not just like productivity, but there's
7:43 also engineering happiness and and we're going to get into a lot of these parts,
7:46 but just maybe speak to there's productivity and there's broader
7:50 components to engineers being successful at a company. >> Yeah. And I love that point, right?
7:54 Because productivity, first of all, it's hard to define anyway. But if if you're
7:58 just looking at like output, you can get there a lot of different ways, but if
8:02 you're getting there in ways that are high toil or high friction, then at some
8:06 point a developer is going to burn out. Or if it's, you know, super high
8:09 cognitive load, if it's hard to even think about what you're doing because
8:13 you're concentrating on like the the mechanics of, you know, the plumbing of
8:17 something, then you don't have the brain space left to come up with like really
8:21 innovative solutions and questions. And so I love that it's kind of this
8:25 self-reinforcing loop in terms of you do more work, you do better work, and
8:31 it it's better for people, it's better for, you know, the systems, it's better
8:33 Flow state and cognitive load in the age of AI
8:34 for our customers. Something I was going to get to this later, but I want to
8:37 actually get to this right now. This idea of flow state for engineers. So I
8:40 was an engineer actually early in my career. I went to school for computer
8:44 science. I was an engineer for 10 years. The best part of the job is for me was
8:47 just this flow state you enter when you're coding and building and just
8:51 things feel like so fun. It feels like AI is making that harder in a lot of
8:54 ways because there's all these agents you're working with now. There's all
8:57 this code that's kind of being written for you. Talk about just the importance
9:02 of flow state to a developer happiness, developer productivity, and just what
9:05 you've seen AI impacting, how you've seen AI impacting that.
9:08 >> A lot of times, well, there are lots of different ways to talk about DevX,
9:12 right? One way to talk about it is kind of three key things that
9:17 have components that are important of themselves. They also kind of reinforce
9:19 each other. So flow state is one of them, cognitive load is another, and
9:23 then feedback loops are another. And so I think, you know, when you touch on
9:26 this, your question about flow state is a really good one. And I'll I'll admit,
9:30 you know, we're just a few years into this. We're still figuring out what the
9:38 best flow state and and like cognitive uh requirements are for people in this
9:41 because to your point, sometimes we're getting interrupted all the time, right?
9:45 you don't just get in the flow and lock down and write a whole bunch of code and
9:49 do like the typing of a whole bunch of code as much anymore. Instead, you're
9:54 kind of creating a prompt, uh, getting some code back and reviewing the code,
9:57 trying to integrate what's happening in the system. Um, and that can really
10:03 interrupt. At the same time though, it can contribute to flow if and I've I've
10:07 seen some senior jars pull together seen tool chains that are really incredible
10:10 where they figured out how to kind of keep the flow going, right? The fast
10:14 feedback loops really really work work well for them. They can kind of assign
10:19 out different pieces uh to agents. It helps them keep in the flow in terms of
10:22 instead of details and line by line writing, they're in the flow in terms of
10:26 what's my goal, what are the pieces that I need to get there, how quickly can I
10:28 get there, so then I can step back and kind of evaluate everything and then
10:33 dive back in and, you know, fix some pieces. >> Is there anything more you could say
10:36 about this engineer that figured out this really cool workflow about just
10:39 what that looks like? >> So, I've spoken with a handful of them
10:42 and I've kind of watched them work. I haven't built it myself yet. It's on my
10:47 list. Um, so they've been able to set up this like really incredible workspace
10:52 and workflow where like right now a lot of us, you know, play around with tools
10:54 and we'll like put in a prompt and we'll get a few lines back or maybe we'll put
10:57 in a prompt and we'll get like whole programs back. Well, what they can do is
11:02 they can many times I'll see them say uh to kind of helped help prime it, you
11:07 know, this is what I want to build. It needs to have these basic architectural
11:10 components. it needs to have this kind of a stack. It needs to follow like this
11:15 general workflow. Help me think that through and it'll kind of design it for
11:18 it. And then for each piece, it'll assign an agent to go work on each piece
11:22 in parallel. And then it'll say, "Oh, and up front, you know, these need to be
11:25 able to work together. Make sure it's architected correctly. Make sure we use,
11:29 you know, appropriate uh APIs and conventions." Then at the end uh and
11:34 then they can like let it run for a few minutes and they can think through
11:36 something else that's interesting or they anticipate is going to be hairy and
11:41 they come back to something that's I mean probably a little better than
11:45 vibe coded, right? Because like because they were so systematic about it up
11:49 front, they're much closer to something that looks like production code. So,
11:52 what I'm hearing is spending a little more time up front planning what all
11:58 these AIA engineers are doing versus just like powering through and just
12:02 Challenges in measuring productivity with AI
12:02 figuring out as you go. >> Yeah. >> Okay, cool. Let me get to this quite a
12:06 core question that I think on is a lot of people's minds. A lot of companies
12:11 are trying to measure productivity for their teams. Is this improving our
12:14 productivity? Is this hurting our productivity? So, let me just start with
12:18 this question. How are people doing this wrong currently when they try to measure
12:23 their productivity gains with AI? >> I will say most productivity metrics are
12:30 a lie. Uh you know it's it's really tricky because historically now look
12:34 lines of code has always been a bad metric, right? But many folks still use
12:38 lines of code as some proxy. Yeah. As some proxy for >> uh output or productivity or complexity
12:46 or something, right? Well, now for many of the systems that they would sometimes
12:49 like whisper and not super talk about that uses lines of code, it's just blown
12:54 out of the water because what do you mean by lines of code? Uh, if the goal
12:59 is more lines of code, I can prompt something to write you the longest piece
13:03 of code ever and add tons of comments. And, you know, we know that agents and
13:09 and LLMs tend to be very verbose uh, by definition. And so it's just too easy to
13:14 gain that system and then introduce complexity and technical debt into all
13:19 of the work that you're doing. I will say there are some things that we can
13:24 kind of watch and pay attention to because so lines of code as a productivity metric
13:30 isn't great, right? It's pretty bad. But now it's kind of more relevant if we can
13:34 tease out which code came from people and which code came from AI because now
13:40 we can answer downstream questions. What is the code survivability rate? What is
13:44 the quality of our code? >> Is our code being fed back into train
13:47 systems and for that code that's that's retraining systems later especially if
13:51 we're doing like fine-tuning and local tuning. How much of that is fishing
13:55 generated? Right? what types of loops is that creating and what types of patterns
14:00 or biases might it be in invertly introducing. So on the one hand like
14:04 it's not good as a productivity metric but it can be useful right and I'll I'll
14:08 even say the same for Dora right so I have done Dora metrics their speed
14:12 metrics their stability metrics if that's all you're looking at it's it's
14:16 not going to be sufficient anymore because AI has now changed the way we
14:21 think about feedback loops right they need to be much faster now what door is
14:24 meant for you know kind of assessing the pipeline overall in terms of speed and
14:28 stability it's still that works but we can't just blindly apply the exist
14:31 existing metrics we've used before because we'll miss super important
14:36 phenomenon and and changes in the way people work. >> Interesting. So, so you you invented
14:42 Dora that was kind of the main framework people used for a long time to measure
14:46 productivity and then there's space there's uh core 4 there's probably
14:50 others. So what I'm hearing here is all these are kind of out of date now where
14:54 AI is contributing large portions of code. I will say if it is a prescriptive
14:59 metric it needs to be used only in the way it was prescribed. Now Dora 4 there
15:05 are four key metrics there's uh two speed metrics uh deployment frequency
15:10 and lead time so code commit to code deploy there's stability metrics uh mtr
15:15 and change fail rate if those are used to assess the speed of the pipeline and
15:20 the general performance of the pipeline that's great if you're trying to use
15:24 those to understand because implied in that is feedback loops right because you
15:26 used to kind of like to get feedback from customers um but we can't just use that blindly now
15:33 when we're using AI as an example because we have feedback loops much
15:38 earlier and not even just at like the local build and test phase. We have
15:42 feedback loops throughout and even sometimes in the middle of some of the
15:45 pipeline that we really want to leverage in ways that weren't
15:51 as useful before. I won't say they weren't possible, but like we just
15:54 didn't really focus there. So those are prescriptive metrics. When we think
15:58 about space, space is a framework. It doesn't tell you what metric to use. So
16:01 I'll say sometimes people get real frustrated because I didn't tell them
16:06 what to measure, right? Uh but now I think that's the power of it. We're
16:10 actually seeing that space applies fairly well in these new emerging
16:15 contexts like AI because we still want to look at so space is an acronym,
16:17 right? So we still want to look at satisfaction. We still want to look at
16:22 performance. What's the outcome? Um we still want to look at activity. Yes, you
16:26 know, in some ways lines of code and number of PRs can be useful for
16:31 something, right? Uh or number of alerts or number of things, activities or
16:35 accounts. C is communication and collaboration. This is also super
16:38 important and useful because it's how our systems communicate with each other
16:41 and also how our people do. You know, what proportion of work is being
16:45 offloaded to a chatbot versus talking to a senior engineer on the team. More
16:49 isn't always better, less isn't always better. Depends. And then efficiency and
16:52 flow. Can people get in the flow? how much time does it take to do things what
16:56 is the flow like through our system and here I would probably add a couple
17:00 dimensions right so uh chatting with some of the early authors to say you
17:04 know trust not to say trust wasn't important before but now it is very very
17:08 front of mind right before you you know build your code like if the if the
17:11 compile comes back you're fine and like that's the way it is LLMs are
17:17 nondeterministic right now we can't just put in a command and get something back
17:20 and accept it we really need to evaluate it so you know are we seeing
17:23 hallucinations. What's the reliability? Does it meet like the style that that we
17:27 would typically write? And if it doesn't meet, is that fine?
17:33 So, that's my kind of kind of it depends answer. Prescriptive, you got to make
17:37 sure you're using it fit for purpose, right? >> We're going to get to your current
17:42 thinking on the best way to do this stuff. You have a book coming out that
17:47 explains the how to do this. Well, so we're going to get to that. One thing I
17:50 wanted to highlight in our last chat that we had, you uh you highlighted one
17:55 of the biggest issues we're going to ex probably have with AI is trust.
18:00 Understanding and learning how much to trust code that it generates and also
18:04 how much you said this two and a half years ago that so much of the time is
18:07 now going to be spent reviewing code versus writing code. That's exactly what
18:10 I'm hearing. >> I think it'll be interesting to see how
18:14 that impacts the way we structure work moving forward. You know, we were
18:17 talking about flow state and cognitive load. Now that our attention has to
18:22 focus on things at certain times and and it's broken up from how we used to do
18:25 it. Um I think there's some real opportunity there to not just rethink
18:29 workflows but rethink how we structure our days and how we structure our work.
18:32 >> Can you say more about that? Just what is that what do you what are you
18:35 thinking will be happening? Where do you think things go? What are you seeing
18:39 working? this uh so purely speculative um but for example uh Gloria Mark has
18:44 done some really good work on attention and deep work and Cubans can get about
18:50 four hours of good deep work a day and like that's about it.
18:53 >> Yeah, I feel that. >> And that that's like kind of the upper
18:57 limit for the most part and I'm sure people are going to be like well I am
18:59 super human. I can do >> what if you take 20 grams of creatine
19:03 >> right micro dose. >> Yeah, exactly. >> Yeah. So in the context of knowing we
19:11 have about four hours of good deep work and I'm sure many of us have probably
19:15 hit this right where like we have good periods like maybe it's morning maybe
19:17 it's afternoon for folks and then you hit a time where you're like I'm going
19:21 to clean out my inbox because that is all I can do right now right like I can
19:24 be functional but I'm not going to come up with my best innovative problemolving
19:31 authoring codew writing work a lot of times the way to do that and to
19:36 get into it is to have these long chunks to get into flow and to get
19:41 that deep work, right? And it's usually, this is I'm making I'm like hand waving,
19:45 right? Two hours minimum, right? Like an hour can be tricky because it could take
19:49 time to get into that state. Okay. Well, when we think about what it used to be
19:52 like back in the olden days, three years ago, three and a half years ago, we
19:56 could block off four hours of time and we could probably get two or three hours
20:00 of really good work done now because we were just focused, right? There were no
20:05 interruptions, minimal interruptions. Now the nature of writing code in
20:10 systems itself is interrupt driven or or full of interruptions at least right
20:14 because you start something and then it interjects and so how do we think about
20:18 that? Does that mean that a 4-hour work block is still useful? I mean, probably.
20:23 But does that mean that now we can also make a 45minute work block useful?
20:29 Because getting into the flow is actually kind of handed off at least in
20:32 part to the machine or the machine can help us get back into the flow by
20:36 reminding us of context and generating diagrams of the system and you know all
20:41 the things. And so I think that's a really really interesting area that's
20:45 just ripe for questions and opportunity and and please folks do this research
20:49 and and come back to me cuz it might not make my list but it's such a great
20:52 question. >> That is so interesting. Essentially everyone every engineer is turning into
20:58 an EM engineering manager coordinating all of these junior AI engineers. And so
21:02 your point is even if you have like a 30-hour block, you can't get deep into
21:07 code, but you can unblock all these AI engineers that are running off doing
21:11 tasks. Plus, your point is they give you they remind you of just like here's
21:14 where you left off. Okay, you can just jump into this code, maybe make some
21:17 tweaks. >> Yeah. >> So interesting. Let me zoom out a little
21:19 The importance of developer experience for business value
21:23 bit and before we get into your framework for how to approach developer
21:26 experience, the latest thinking you've got beyond just like obviously
21:31 engineering engineers doing more is great. What's what's your best pitch for
21:35 why companies should really really really focus on developer experience?
21:39 >> I hate to say return on investment but like the business value is the
21:43 opportunity here is huge, right? In general, we write software well for fun
21:47 and for hobbies, right? But we also have software because it meets a business
21:53 need. It helps us with market share. It helps us uh attract and retain
21:56 customers. It helps us do all of these things. And you know, I think DevX is
22:02 important because it enables all of that software creation. It
22:06 enables all of that problem solving. enables the super rapid experimentation
22:10 with customers that before, you know, you you'd need a while for a prototype
22:13 and maybe a little bit longer to actually flight it through an AB test on
22:18 a production system. I mean, you can do it in hours right now.
22:20 Common issues and solutions in developer experience
22:22 >> Getting maybe the opposite end of the spectrum, getting very tactical before
22:27 we get into the larger framework. What's just like one thing that you think an
22:31 edge team, a product team can do this week, next week to help their developer
22:35 experience maybe get more done? Honestly, I think the best thing you can
22:39 do is go talk to people and listen. And I love that, you know, the audience of
22:42 this podcast is primarily PMs because they tend to be really good at this. And
22:47 I would say start with listening and not with tools and automation. So many times
22:50 companies are like, well, I'm just going to build this tool or going to I'm going
22:55 to build this thing. Often you build a thing that you yourself have had a challenge with or
23:01 that like is easy to do, easy to automate. And if you just go talk to
23:07 people and ask the developers like think of think of yesterday, what did you do
23:12 yesterday? Walk me through it. What were the points that were just delightful?
23:15 What were the points that were really difficult? Where did you get frustrated?
23:19 Where did you get slowed down? Where was there friction? And if you go talk to a
23:24 handful of people, a lot of times you can surface a handful of things that are
23:30 a relatively low lift and still have impact. um or you can identify a process
23:34 that's unnecessarily complex and slow. >> So the listening tour here most is you
23:41 want to help your teams move faster, be happier teams, your advice is just
23:45 before you do anything, just like go ask them what is bothering you.
23:49 >> Go ask them. Yeah. And trust me, like most developers are going to be more
23:52 than happy to tell you what's broken and what's bad. >> And I mean I will I'll say there was one
23:58 company that I had worked with. I remember they they had a process that
24:02 was like really difficult and it was on an old mainframe system and they were
24:05 going to have to like replplat the whole thing and so they never went to work on
24:09 it or talk about it. Uh everyone hated it cuz it was this huge delay. I mean
24:14 all they had to do was change a process. Sometimes all you have to do is change a
24:17 process and they changed it so that instead of I think it was someone had to
24:20 like print it out and walk it down three or four flights and then get approval
24:24 and then someone else had to like walk it back up and so it was just that
24:28 interim. They didn't replat anything. They didn't redesign anything major.
24:31 They just had it send an email. >> Let me push on that. And I'm curious
24:35 just what are the most common things people do? Like if you were just
24:38 starting on, okay, we need to focus on engineering experience. What do you find
24:42 are the most like I don't know two or three most common improvements companies
24:45 need to make? >> I will say, you know, I'll kind of echo
24:49 that process. there's almost always a process that can be improved and that
24:53 can be approved improved without a lot of engineering lift or a lot of
24:58 engineering headcount, right? Uh most large companies in particular have
25:04 something that is several several steps. It's the way it is because it's the way
25:06 it is, but that's no longer the way it is, right? Um and even small companies
25:10 sometimes it's just a little too yolo and you don't know what it is and you're
25:13 kind of chasing everyone around. So if you can create a very lightweight
25:16 process, that can also be helpful. that can be one of the best places to start
25:21 especially if you have limited uh exposure to the whole rest of
25:25 the or right sometimes just a team process can help um I will say from a business leader
25:33 standpoint a lot of what you can do is provide structure and support for this
25:37 organizational change communicate what you're doing communicate what the
25:40 priorities are communicate why this is important celebrate wins because
25:48 if folks try to do this just like a oneoff side fully isolated project. It's
25:53 really challenging to get some good momentum and get people to care and to
25:57 get them stay involved, right? Because it feels like just another interim internal project
26:04 that isn't going to matter or that isn't going to get celebrated, but it has
26:09 these huge uh upside uh potential returns for the business. >> It's interesting what I'm hearing here
26:14 is nothing about tools or technologies. It's not like move to this cloud. It's
26:20 not like install this new deployment system. It's processes and people and
26:24 organ morale. >> Yeah. Now, there will be technical pieces that are very important, right?
26:31 Uh especially now with AI, right? We're we're rethinking how build and test
26:35 systems work. We're rethinking feedback to users so that it's very very
26:39 customized in terms of what is shared and when it is shared. There are a lot
26:44 of technical pieces that are involved. But that's not the only thing, right?
26:48 It's necessary but not sufficient. And that doesn't have to be the place that
26:49 Signs your eng team is moving too slow
26:50 you start. >> I'm going to ask you, I have a hard question I want to ask you that I
26:53 thought of as you were talking. I feel like this is the question that most
26:59 founders and heads think about. And the question is just like, how do I know if
27:03 my edge team is moving fast enough, if they can move faster, if they're just
27:08 not performing as well as they can? What are just maybe smells, signs that tell
27:12 you, yeah, my team should be moving faster versus like this is just the way
27:16 it works. This is as fast as they can move. >> Most teams can move faster, right? So,
27:23 and also uh given what we know about cognitive load, not all speed gains are
27:31 necessarily good, right? Um or the upside is going to be kind of
27:34 limited, right? Once you hit kind of a certain point, most people are not even
27:37 near that point. Uh, I don't know of a single team, frankly. But how do you
27:43 know? Uh, you know, if you're always hearing about bills breaking, flaky tests, uh, overly
27:54 long processes, if you have to request a new system or if you need to provision a new
28:01 environment or if it's really really hard to switch tasks or switch projects,
28:05 right? Right? So if someone has an opportunity to go work in another part
28:09 of an org and they don't for reasons that are unclear and like not political
28:14 and anyone says anything about the system, that's usually a pretty good smell uh
28:20 that there's friction somewhere because once you finally figure out your system
28:24 and you're able to get work done, you don't the switching costs can often be
28:29 really really high to go anywhere else. And so sometimes people will do that but
28:33 you know I've worked with companies where switching orgs within the company
28:41 uh you had to basically pay the same tax as a new hire because the systems were
28:46 so different and they were so full of friction and it was so difficult to do
28:49 so many things. >> I love the first part of your answer
28:52 especially which is you can always move faster. I think every founder is going
28:55 to love hearing that. Uh to your point that there's diminishing returns over time.
29:00 >> Yeah. and and you don't know about the quality, right? So, like I think that's
29:02 the other side is that you can always move faster, but faster for what? Are we
29:07 making the right business decisions? And I think, you know, that's especially
29:10 where PMs come in. You could we can ship trash faster every single day. We need
29:15 strategy and really smart decisions to know what to ship, what to experiment uh
29:21 with, what what features we want to do in what order, and what roll out, right?
29:25 The strategy is the core piece. And then think about speeding that up. If we
29:28 don't have the other pieces in place, I mean, garbage in, garbage out.
29:31 >> I want to follow that thread, but before I do that, just to mirror back what you
29:34 shared. So, signs that your team, there's a lot of low hanging fruit to
29:38 improve this the productivity of your team is builds are always breaking.
29:42 There's flaky tests that are constantly incorrect, false positives. It's hard to
29:46 context switch between different projects. The system, you just hear
29:49 people talking about the system is just really hard to work with. Is that
29:52 How AI is improving productivity
29:52 roughly right? >> Yep. >> Cool. Okay. So going back to the point
29:57 you just made, there's a sense that AI is making teams so much faster because
30:01 it's writing all this code for them. You're going to have all these
30:03 asynchronous agents, engineers working for you. Feels like a core part of your
30:07 message is that's just a one part of engineering work. There's so much more
30:11 including figuring out what to build alignment internally. Maybe just speak
30:15 to just like there is a lot of opportunity to improve engineering
30:19 performance, productivity, but there's so many other elements that are not
30:22 improved through AI. >> Yes. uh or or could be in the future,
30:27 right? Like I think there are a lot of ways that we can pull in AI tools to
30:32 help us refine our strategy, refine our message, think about the experimentation
30:37 methods or you know uh targets of experimentation or think about our total
30:40 addressable market, right? But we need to have that strategy and plan fairly well aligned,
30:48 right? or at least have like two or three alternatives that you want to test
30:51 because now the engineering can go or at least the prototyping especially can go
30:54 much much faster like right we can throw out prototypes uh we can run AB tests
31:00 and experiments that are customerf facing right assuming that you know we
31:04 have the infrastructure in place which allows us to learn and progress much
31:09 faster before right like it it some places it used to take you know months
31:12 to get something through production to do AB testing and get feedback we can do
31:17 this in a day or two definitely under a week, but we want to make sure that we're
31:22 building and testing the right things. Are we partnering with the right? Do we
31:25 have the data that we need? Right? And I will say AI can actually be a pretty
31:29 good partner there if you keep if you have like a good conversation with it
31:32 and then also check with your experts, right? What type of data should I be
31:36 looking at? What type of instrumentation do I need? What type of analysis can I
31:40 do? Uh because then you can also go to your data science team and say like I'm
31:44 planning on doing this. I'd like to because let's not just yell AB tests,
31:48 right? Because that can be it. It's a shame to do a large test and end up
31:54 disrupting users or disrupting customers or breaking privacy or security uh
31:59 protocols and also end up with data that's unusable, right? Because you just
32:02 can't get the signal that you're looking for. But now I'm also seeing people kind
32:07 of accelerate that into a few days versus a few weeks. and so they can kind
32:11 of start those key stakeholder discussions from uh much more informed kind of filled out
32:17 space. >> Today's episode is brought to you by KOD. I personally use KOD every single
32:23 day to manage my podcast and also to manage my community. It's where I put
32:27 the questions that I plan to ask every guest that's coming on the podcast. It's
32:31 where I put my community resources. It's how I manage my workflows. Here's how
32:35 KOD can help you. Imagine starting a project at work and your vision is
32:39 clear. You know exactly who's doing what and where to find the data that you need
32:43 to do your part. In fact, you don't have to waste time searching for anything
32:46 because everything your team needs from project trackers and OKRs to documents
32:51 and spreadsheets lives in one tab all in KOD. With KOD's collaborative all-in-one
32:55 workspace, you get the flexibility of docs, the structure of spreadsheets, the
33:00 power of applications, and the intelligence of AI all in one easy to
33:05 organize tab. Like I mentioned earlier, I use Kota every single day. And more
33:10 than 50,000 teams trust KOD to keep them more aligned and focused. If you're a
33:13 startup team looking to increase alignment and agility, KOD can help you
33:17 move from planning to execution in record time. To try it for yourself, go
33:21 to kod.io/lenny today and get 6 months free of the team plan for startups. That's cooda.io/lenny
33:30 to get started for free and get 6 months of the team plan. kod.io/lenny.
33:32 Real examples of productivity improvements
33:35 I love that you work with a bunch of different companies and a bunch of
33:40 different types of businesses. I think a very few people get to see inside a lot
33:44 of different places. What kind of gains are you just seeing in terms of
33:47 increased productivity with AI? Like how real like how big of a gain have you
33:49 seen? >> I'd say it's real. And I would also say we don't have great measures for it yet.
33:56 We're still trying to figure out what to measure and what that looks like. Uh one
34:00 of the best is going to be velocity right all the way through the system.
34:05 How quickly can you get uh a feature or a product or something through the
34:08 system so that you can then experiment and test right either from like idea to
34:13 like final end or even kind of a a feature and a piece through the system
34:17 so we can test. That's really good. Now that's also hard to tie back directly to
34:24 like a particular AI tool in the hands of a particular developer. Um, but there
34:28 are some other things that that we can look at and we can see. Um, and and that
34:33 I've seen is again this kind of rapid prototyping. Um, I hate lines of code, but I'm going to
34:40 use lines of code. Um, we do see I I know I worked with uh some
34:45 folks who who had kind of a whole set of companies they were looking at and they
34:49 found that uh AI was generating like significantly more code for the people who were using
34:57 it regularly. But then they also found that for folks who were like you know
35:02 regular users of AI coding environments AI idees the the tool kind of gave them more code
35:10 and then the engineers themselves the increase was double what the uh
35:15 coding agent had given them and so one I'd say probably it's kind of a
35:19 secondary or knockon or or just a smell right is it can unblock you it can speed
35:23 up the work that you would already do right I know sometimes when I work. It's
35:27 like the first few minutes it's hard for me to start, but once I get started, I'm
35:30 there. And so they're really good at unblocking and unlocking that.
35:33 >> Something I've seen people on Twitter sharing is how good uh OpenAI Codeex
35:38 especially is at finding really gnarly bugs. And I think it was Karpathy that
35:44 shared he was like so stuck on a bug and no AI tool could figure it out and then
35:48 the latest version of Codex spent like an hour or something looking into it and
35:51 found it for him. >> Yeah, I'm I'm hearing incredible things
35:55 like that. Right. Well, and even also, you know, writing unit tests and
35:58 spinning up unit tests and >> uh creating documentation and cleaning
36:02 up documentation because >> I know now people are like, "Oh, well,
36:06 we have agents. I don't need to >> I don't need to read the docs because
36:10 there's the code there." Turns out, uh, agents rely on good data, right? Because
36:15 it's it's all about how they've been trained or how they've been grounded.
36:19 Um, and better data gives you better outcomes. And some of that data includes
36:23 documentation and comments. And the better documentation, the better
36:26 comments you have, the better performance you're going to get out of
36:29 your AI tools. >> And AI can help you write that documentation. I've been working with
36:33 Devon a little bit and it's really good at that stuff. >> Yeah.
36:35 Introducing her new book, Frictionless
36:38 >> Okay. Let's talk about this framework, this book. So you're publishing a book
36:42 called Frictionless, which sounds like a dream. How do you create a a dev team
36:46 that's frictionless? This is called frictionless seven steps to remove
36:49 barriers along value and outpace your competition in the age of AI. There's a
36:54 sevenst step process to this. Walk us through this. Maybe give us just context
36:57 on this book. What who is meant for what problem it solves and then the seven
37:00 steps. >> So I will say I uh also wrote this with Abby Nota who has just uh of DX. He has
37:06 incredible experience in the space, right? He's worked with hundreds of
37:10 companies. And so it was kind of nice bouncing ideas off of him. And you know,
37:15 also thanks to all of the uh engineering leads and devs leads and CTO's and
37:19 engineers that we talked to to kind of make sure that we our smells were right.
37:25 Right. And so who is this before? >> Let me actually take a let me take a
37:28 tangent on Obby and DX since you mentioned him. Uh this is super
37:31 interesting and I think it connects so directly with this conversation. So,
37:34 Abby started this company called DX, which is such a great name for a company
37:39 around developer experience. They just sold the company for a billion dollars
37:43 to Atlassian. Uh, it's a very high multiple on their AR. It to me shows
37:48 exactly why this conversation is so valuable, just how much value companies
37:53 are putting into improving developer experience. At Lastian would spend a
37:56 billion dollars on this. It's like a early stageish startup that was doing
38:00 really well and people loved it, but it was like early stageish. a billion
38:04 dollars. Uh, and now and the idea is they have all these companies working
38:08 using Jara and all their products. Uh, they're all trying to figure out how do
38:11 we measure productivity. Uh, it's worth a lot of money to them. So, and I know
38:15 you were an early adviser to them, too. So, >> yeah. >> It just shows us how important this is.
38:19 >> Yeah. Well, and I think it also shows us um how much value you can get out of
38:25 this, right? Like there's so much low hanging fruit. There's so much unlocked
38:28 potential and it's hard to know where to start. A lot of times even in I've been
38:32 at large companies that have a lot of expertise and a lot of really really
38:35 smart people. But if you haven't kind of been in this space and thinking about it
38:38 this way, it's hard to know where to start or it's easy to make simple mistakes up front
38:44 that mean like you kind of need to start over later. Um so I guess which kind of
38:48 also brings us back to you know who is this book for? It's for anyone that
38:55 cares about DevX, right? So definitely technology leaders um anyone who's
39:00 trying to kick off a DevX program or is working on a DevX Devx improvement
39:05 program um I think is particular particularly relevant for PMs because if
39:11 you're PMing something that involves software and building and creating
39:15 software improving Devex will only help your team and also uh you have
39:23 key skills and insights and instincts that are so important important to DevX
39:28 that many times I will say I've seen engineering teams just miss.
39:33 >> Okay. What's the framework? What are the steps? Where do people start?
39:37 >> Uh so the book goes through uh sevenstep process and then also kind of provides
39:42 some some key kind of uh principles at the end. Step one is to start the journey, right?
39:49 So assuming you're kicking off, you can start the journey. And this involves
39:52 what we have already talked about, right? Go talk to people, have a listening
39:58 tour, synthesize what you learn, uh visualize the workflow and tools, right?
40:02 Like get a handle on kind of what the current state is. Uh step two is to get
40:07 a quick win, right? So start small, get a quick win. Uh pick the right projects,
40:13 share out what you've done. Um step three is using data to optimize the
40:18 work, right? So kind of establish some of your data foundation, find the data
40:22 that's there, start collecting new data. um use some surveys for some really fast
40:27 insights and we include uh example surveys. Uh step four then is to decide
40:33 strategy and priority. Once you have some data, then you need to know of all
40:36 the things that are potentially broken and you've already gotten your quick win
40:39 of all the things that are left. What should I do next? And so we walk through
40:43 some evaluation frameworks there. Step five is to sell your strategy. Once
40:48 you've decided, now you have to kind of convince everyone else, right? So now
40:51 you want to get feedback. you want to share why this is the right strategy
40:56 right now. Uh step six is to drive change at your scale. So here we address
41:01 folks that have local scope of control, right? If you're starting on just a dev
41:04 team, you want to do it yourself kind of grassroots effort. Uh or global scope of
41:09 control, right? If you're, you know, the the VP of developer experience or
41:12 something like there are some things that you can leverage for a top down. Um
41:16 and then how do you drive change when you're kind of somewhere in the middle
41:19 because you can leverage both both types of strategies. Uh and then step seven is
41:24 to evaluate your progress and show value and then kind of uh loop back around.
41:29 And I will say that we wrote this so that you could kind of jump into any
41:34 step where wherever you are right now, right? Like if you're kicking off a
41:37 team, you'll or an initiative, you'll probably want to start at step one. You
41:40 should definitely start at step one. If you're joining an existing initiative,
41:44 you could jump into uh picking the priority or uh implementing the changes.
41:52 So those are so those are seven steps. Um there are a few practices that we
41:56 also recommend. So thinking about resourcing it, uh change management, uh
42:04 making technology sustainable and then also bringing a PM lens to this, right?
42:09 How can we think about developer experience as a product and how do we
42:12 think about the metrics that we have as a product? >> Awesome. Okay, I have questions. Point
42:16 people to the book real quick. What what's the URL? How do they get it? When
42:18 does it come out? >> Yeah, developerexperiencebook.com.
42:23 So, right now you can sign up for the mailing list. We'll we'll let you know
42:26 when it's out on pre-order and we'll also be sharing pieces of the workbook.
42:31 So, we've got almost a 100page workbook that goes along with the book. Um, and
42:36 then it should be out by end of year. Okay. So, one piece of this is just this
42:40 term developer experience feels very intentional in that it's not developer
42:45 productivity, developer uh work, it's how do we make developer experiences
42:49 better at our company which includes they get more done but also they're
42:53 happier things like that. So, I think that's an important element of the rate.
42:56 >> Yeah. >> Absolutely. Because >> again, it's not just about productivity,
43:02 right? We talked about this from from kind of the frame and the lens of we
43:05 need to be building the right thing and you want to be productive but you also
43:09 want to be thinking about and this is what engineers are also just really
43:13 incredibly good at. give them a problem and don't tell them how to solve it and
43:17 then they can solve it better right they have uh the freedom they have the
43:20 innovation they have the creativity so that they can solve this problem if it's
43:23 only about productivity then it's just like lines of code or number of PRs or
43:27 whatever right but we really want to talk about value and how do we unlock
43:31 value and how do we get value faster and that involves yes making them more
43:36 productive and removing friction because then they have the flow and the
43:40 How to get started building a DevEx team
43:40 cognitive load and the things that we kind of talked about >> awesome Okay. And then say someone wants
43:45 to start this team. What does it usually look like at Airbnb? I remember this
43:48 team forming and it was just like an engineer or two getting it started and
43:53 taking charge. What do you recommend as the the pilot team and then what does it
43:56 look like as it grows? >> So there are there are a few ways to do
43:59 this, right? So, if you're doing it yourself, you could do it with a couple engineers,
44:07 uh, maybe a a PM or a or a PGM or a TPM, um, to kind of help communicate because
44:13 really comms plans are just so important here. Uh, on a small on a small scale,
44:19 right? What we want to do is look for those quick wins. Look for things that
44:23 you can do at small scale. Are there, you know, some folks call them things
44:26 like paper cuts. Are there small things that you can do to help people see the
44:32 value and feel the benefit themselves, right? How can a developer's work get
44:36 better? How can their day-to-day work get better? Kind of build momentum from
44:40 there. If you're working from a top- down structure and you have the remit,
44:46 you still want uh some some quick wins, but those quick wins can look a little
44:50 more global in scale because you have the the infrastructure or the backing to
44:54 make, you know, different types of changes that aren't only local. So, you
44:58 know, an example of a small local change could be just cleaning up your tests in
45:01 your test suites, right? Any team could do that. Any team could do that. uh more
45:08 more global scale might be changing uh organizationwide process that is just
45:12 overly cumbersome or throwing some resourcing into you know cleaning up the
45:15 The impact of forming developer experience teams
45:15 provisioning environment. >> What kind of impact have you seen from
45:19 teams like this forming on the engineering teams at their companies? I
45:24 I'll say I've seen a huge impact right uh for for smaller companies hundreds of
45:29 thousands of dollars for large companies in the billions because well also we
45:34 need to learn how to communicate that right like what does the math look like
45:37 are we many times we can look at saving time we can look at saving costs um we
45:41 can look at a lot of different things we could look at speed to value as speed to
45:46 market we can look at risk reduction um but the gains really are there I I will
45:49 mention that it tends to follow something like J curve All right. So,
45:52 like you'll have a couple quick wins and it'll look like a big big win and then
45:56 you'll hit a kind of a little divot where suddenly the really obvious
46:01 projects the lowhanging fruit are handled. So now we need to do a little
46:04 bit of work, right? We we might need to build out a little bit more
46:06 infrastructure. We might need to build out a little more telemetry so that we
46:09 can capture the things we want to capture. And then once we get that done,
46:15 How to measure the impact of DevEx teams
46:15 uh then we start to see those benefits really compound. >> So going back to that measurement
46:19 number, what do you recommend? How do people find these numbers? Because I
46:23 think that's so much of the power of this is like we saved a million dollars
46:27 doing this. What do you look at to to figure that out? >> You know, I think there are a few
46:32 different things to keep in mind, right? Who who is our key audience? And we
46:36 usually have a few key audiences, right? We really want to be able to speak to
46:39 developers because they're the ones that are going to be using the systems.
46:43 They'll be partnering with you on either building them or at least providing
46:46 feedback about what you're doing. Um and so for them we often want to frame this
46:52 in terms of things they care about. So time savings right if something gets
46:56 faster they can save time they can you know they don't spend time doing setup
47:00 when they don't need to anymore related to that is reduced toil right so
47:04 compliance and security are super important also many times it requires
47:10 several several manual steps that I don't say they're not value ad they are
47:14 not value ad from an individual human perspective right if we can automate as
47:19 much as possible that's great um and you know improve focus So that's from the developer side of
47:27 view. Leadership often cares about they care about those things, right? But they
47:31 often care more about other things. So we could talk about uh usually costs and
47:36 dollars, right? So can we accelerate revenue? What is our uh time to value
47:41 look like? Uh what does our velocity? How quickly can we get feedback from
47:45 customers? Um, and for folks and organizations that are in really
47:48 competitive environments, that can be really compelling because it's all about
47:53 speed. We can talk about saving money, right? So, here we can look at maybe
47:57 quantifying savings. So, you know, one example is uh test and build. If we can
48:01 clean up a test and build suite to a developer, they really want to hear
48:08 about uh time saved and more reliable systems, right? there's
48:10 less toil because they don't have to keep rerunning tests or kind of go clean
48:14 up test suites. From the business perspective, cleaning up a test in a
48:18 build suite can be uh cloud cost savings because all of those tests are running somewhere on a
48:23 cloud and if they always fail or they're if it's just kind of a waste of spend
48:27 that can be useful, right? Uh recovering some capacity, right? Uh we can always
48:32 talk about time and productivity gains, right? So how much uh
48:39 equivalent developer time are we losing on things that are not necessarily value
48:43 ad right and then sometimes we can correlate to business outcomes and
48:46 correlate is usually the best we can we can do here but there can be some pretty
48:50 compelling correlations in terms of speeding up time to value and increase
48:53 Measuring the impact of AI tools on productivity
48:54 market share for example >> so let me pull that thread and come back
48:57 to this what I think is the biggest question people have right now with AI
49:02 and productivity which and I don't I don't think anyone No one has the answer
49:04 yet, but I'm curious to get your take of just what should people do today? What's
49:10 the best approach to understanding what impact AI tools are having on their
49:15 productivity because they're spending all this money on there like I don't
49:17 know what are we getting out of this and I guess things are moving faster but I
49:20 don't know. So if someone had to just like okay here's what I should probably
49:24 try to do. What would be your best advice here for measuring the impact of
49:28 AI tools on productivity? >> I would say it depends. Um, and in part
49:34 it depends on what your leadership chain really cares about,
49:38 right? Like we we're usually pretty good at like figuring out what matters to
49:41 developers and we could communicate that to them. But if we're trying to just
49:44 identify two or three data points to really kind of focus on because when
49:47 we're first starting with data, sometimes it can be challenging.
49:51 What do they care about? Think about the messaging you've been hearing. Have they
49:55 been talking about market share, right? Losing market share or or
49:59 competitiveness in the marketplace. If that's it, focus on speed. Think about
50:04 ways that you can capture metrics for speed for from like feature to
50:07 production or feature to customer or feature to experiment and what that
50:12 feedback loop looks like. If they're talking about uh profit margin all the time, right now we
50:20 always talk about money, right? because this is business. But if that seems to
50:24 be an overarching narrative, look for ways that you can save money and then
50:28 translate that into uh recovered and recouped uh headcount cost, right? Or sometimes you'll kind of
50:36 like reinvent, change a process and then you no longer need as many vendors,
50:39 right? So so reductions in vendor spend can also help there. Um and I I say also
50:45 it depends because sometimes something will they'll say something, right? like
50:49 leadership will say something and it it kind of comes up as a theme. If you can
50:53 solve a problem that they have or it's like something that they're focused on,
50:57 if you can slightly reframe it even, right? Like if they're calling
51:00 everything developer productivity, go ahead and call it productivity. If
51:03 they're calling it velocity and velocity is what matters to them, think about how
51:06 to frame this in terms of velocity. If they're talking about transformation or
51:10 disruption, right? How does this help with the disruption? Because then it
51:14 will resonate with them. We don't want to make them work to understand what it
51:18 is that we're doing and the value that we provide. >> That is such good advice. So just to
51:26 reflect back the advice here is if your company's trying to figure out what sort
51:29 of impact our AI tools having on our company first is just like what what
51:32 does the company care about most? What do leaders care about most? Could be
51:36 market share, could be profit margin, could be velocity. We need higher
51:40 velocity or we need to transform transformation. So your advice there is
51:45 like figure that out based on words and phrases you're hearing then figure out
51:49 ways to measure that ways to measure market share growing uh profit margin
51:54 increasing. So it could be uh I love these examples like time from feature
51:59 idea to production or to experiment. So maybe start tracking that. If it's
52:04 margin it's like money saved by fewer tests failing or some vendor you don't
52:08 have to pay for things like that. And then velocity. Velocity. I imagine
52:12 that's where things like Dora come in of just like speed of engineering shipping
52:15 or what would you think about there for velocity? >> I would say it's actually, you know, one
52:20 of those uh I would pick as as broad a swamp as you can. So if you can go from
52:24 idea to customer or idea to experiment, how long does that take? How long does
52:28 it typically take and how long can it take and and does it take now with uh
52:33 improved use of AI tooling and reduction in friction? Right? And that's where I
52:36 will say we talk about this a little bit in the book. Uh you know how do we deal
52:41 with attribution challenges? So like what was responsible for this? Was it
52:45 the DevX or was it the AI? Um go ahead and disclose that, right? Say yes, we
52:48 rolled out AI tools. We also had this effort in DevX. They partnered very
52:53 closely together. Both of them probably contributed to this, right? Like if we
52:56 had AI tools without the DevX improvements, we probably would have had
52:59 some improvements but not nearly as much, right? If people were starting to
53:04 do this today, say they're just like, I want to start measuring developer
53:07 experience. Are there like a two or three metrics everybody basically needs
53:10 that you should just start measuring ASAP? >> If you're just starting today and if you
53:14 have nothing at all, uh talk to people obviously after that. I would do surveys
53:20 uh because surveys can give you a nice kind of overall view of the landscape
53:25 quickly so that you know where the the big kind of challenges are. And I say
53:28 that because if if you're just starting, you you might not have instrumentation
53:33 through your system, all the metrics. And if you do already, it might not be
53:37 what you think you want, right? Metrics that were designed without purpose,
53:42 questionable metrics that were designed for another purpose, they might work for
53:46 what you want, but they might not. So, we can't just assume we have them. So,
53:50 that's one reason I like surveys. And uh we included an example in the book. You
53:53 can just ask a few questions, right? How satisfied are you? uh what are the
53:59 biggest barriers to your productivity or what are the biggest challenges to
54:02 getting work done? Um and let them pick you know either from a set of tools or
54:08 maybe like a set of uh processes and then say and like let them pick three
54:15 just three of those three how often does this affect you? Right? Is this hourly?
54:19 Is this daily? Is this weekly? Is this quarterly? Right? Because sometimes it
54:22 hits you every single day and you're just mad about it. Sometimes it only
54:25 hits you once a quarter because it's end of quarter, but it's so ownorous, right?
54:30 And then kind of open text, right? Like, is there anything else we should know?
54:35 um that that can give you incredible signal because by making folks
54:41 prioritize the top three things if you let them pick everything like it makes
54:45 the data super super messy but three things and how often you can just come
54:49 up with a score or a weighted score if you want and then go kind of dig into
54:54 where should that data where should that data be what data do we need but also
54:58 then you've got at least some kind of baseline baseline right it'll be a
55:02 subjective baseline but now you'll know what the biggest challenges are.
55:06 >> I love how all this just comes back to starting by talking to people and asking
55:09 them these things which is very similar to product management and just building
55:12 great products is have you talk to your customers and everyone thinks they're
55:15 doing this but most people are not doing this enough. >> Yeah. And I will say like one thing
55:16 Survey design for developer experience
55:19 that's challenging when you're start when you think about getting data right
55:23 so interviews are data and that's important. Surveys are a little more
55:27 quantified, right? Because we can we can turn into counts. But that's where we
55:31 also want to be careful, right? A lot of folks go to write a survey question and
55:36 they'll say something like, "Were the build and test system slow or
55:40 complicated in the last week?" You're asking four different questions there.
55:43 If someone answers yes, was it the build? Was it the test? Was it slow? Or
55:47 was it like flaky or complicated or something, right? So, it can be really
55:52 difficult to untangle what the signal is you're actually getting there. And so, it is
55:59 worth time uh chatting with with someone who's familiar with survey design. Um
56:05 having a conversation with Claude or Gemini or Chad GPT around uh here are
56:10 the survey questions or can you propose them? And then make sure you take a
56:13 couple rounds. Is this a good survey question? what s what questions can I
56:17 answer from the data that I get? What problems could I solve? If you can't answer a
56:22 question with data, don't get it. >> And you have example surveys in your
56:26 book for folks that want to just copy and paste and not have to think about
56:27 this. >> Example surveys. Um, a lot of example questions. We even recommend like what
56:34 the format like how what the flow should look like, how long it should be, how
56:38 long it should not be. One thing that I was reading uh is that you don't love
56:42 happiness surveys specifically asking engineers how happy they are. Is that
56:45 true? If so, why is that? >> I don't. Now I will I will say I I don't
56:51 love a happiness survey because there are too many things that
56:55 contribute to happiness. Happiness is a lot, right? So happiness is work.
57:00 Happiness is family. Happiness is hobbies. Happiness is weekends.
57:03 Happiness there are so many things that contribute to happiness. Now, that
57:06 doesn't mean I don't care about happiness. I think happiness surveys are
57:11 not particularly useful here. What can be helpful is satisfaction. And people
57:14 are like, "It's the same thing." It's not because you can ask, "Are you
57:19 satisfied with this tool?" Right? And then ask some follow-up questions.
57:24 Now, those two are related because the more satisfied you are with your job and
57:28 your tools and the work and your team, it contributes to happiness. And I used
57:32 to joke, remember the old commercials like Happy Cows make Happy Cheese? out
57:37 of California. That was the best. Um, >> happy devs make happy code. They they
57:42 write better programs. They do better work. Um, they're they're better team
57:47 members and collaborators. But but capturing and trying to directly
57:52 influence happiness, like that's that's that's not what we're here for, right?
57:56 And it's just it's too challenging. It's too all-encompassing. Satisfaction can
57:59 Popular AI tools for developers
57:59 give us some signal >> in a totally different direction. Uh in
58:04 terms of just tools you see people using. Are there any that just like oh
58:10 yeah this one's really commonly uh great for people? This is just like a tool
58:12 people are finding a lot of success with. Like there's the common ones.
58:16 Copilot cursor. Uh I don't know. Is there anything that stands out that you
58:19 you want to share? Just like hey you should check this tool out. People seem
58:21 to love it. >> I I think the use right co-pilot cursor
58:25 Gemini >> claw code. >> Yep. Clock code. I love claw code.
58:32 I bet I have a whole post coming on ways to use cloud code for non-engineering
58:36 use cases. It's so nice. >> It's so interesting. For example,
58:40 >> cloud code uh find ways to clean up storage on my laptop
58:45 >> and it just tells you here's a bunch of files. It's just like Chad GPT running
58:48 on your computer and it could do all kinds of crazy stuff on your computer
58:53 for you like a little mini mini god. >> Well, I'm going to do that now. This is
58:55 great. >> It's so good. It's Yeah, that's why I'm writing this. I had a Dan Shipper on the
59:01 podcast and he said clock code is the most underrated AI tool out there
59:04 because people don't realize what it's capable of. It's not just for coding and
59:08 Bringing a product mindset to DevEx improvements
59:09 that's trying to explore more and more. Okay. Is there anything else that you
59:12 think would be valuable for people to hear for to help people improve their
59:18 developer experience, help them adapt to this new world of AI uh and engineering
59:23 that we haven't covered. I think something that's important to think
59:27 about in general is to bring a product mindset to any type of DevX improvements
59:33 that are happening and also the metrics that we kind of collect and capture. And
59:38 by that I mean we want to identify a problem, right? Make sure we're solving
59:43 a problem for a set of users. We want to think about creating MVPs and
59:49 experiments and get fast feedback. You know, some do some like rapid iteration.
59:54 We want to have a strategy. We want to know who our addressable market is. We
59:58 want to know what success is. We want to basically have a go to market function,
60:02 right? We we need to have comms. We need to get continuous feedback from our
60:06 customers. We want to keep improving. Um and at some point we want to think
60:09 about, you know, suns setting something, right? Is it in maintenance mode?
60:13 sunsetting. And I think that's important in general, but I think it's extra
60:17 important now because when we have AI tools, we're using AI tools, we're
60:22 embedding AI into our products, things are changing so rapidly that it
60:27 it can be really important to take like half a beat and say, "Okay, what's the
60:31 problem I'm trying to solve right here? Is this metric that we've had for the
60:35 last 10 years still important, or should this be sunset?" Because it's it's not
60:37 really important anymore. It's not driving the types of decisions and
1:00:40 AI corner
60:40 actions that I need. Before we get to our exciting lightning round, I want to
60:44 take us to AI corner, which is a recurring segment on this podcast. Is
60:49 there some way that you've found a use for an AI tool in your life, in your
60:52 work that you think might be fun to share, that you think might be useful to
60:55 other people? >> So, I uh have been kind of working on
61:02 some home design and like like redecorating rooms and stuff. Um, I'm
61:05 working with a designer because I know what I like, but I don't know how to get
61:08 there. I'm not good at this. But I've really been loving Chad GPT and Gemini
61:14 especially to render pictures for me, right? So I can give it the floor plan.
61:17 I can give it one shot of the room that's like definitely not what it's
61:20 supposed to look like and then I can give it a pictures of a couple different
61:23 things and then I can just tell it change the walls or change the furniture
61:28 layout or change something. And it helps me and it's relatively quick. Um it
61:32 helps me kind of visualize the things again. I know what I like but I don't
61:35 know how to get there. So, I know if I like it or not, which is probably a very
61:41 random use, but it's fun for now. >> My wife does exactly the same thing.
61:44 She's sending me constantly, here's what this rug will look like in her living
61:48 room. Here's this water feature. Uh, it's so good and it keeps getting
61:51 better. It's just like, that's exactly our house with this new rug.
61:53 >> Yeah. >> Uh, and all you do is just upload these
61:56 two photos and just like, how would this look in that room? >> I've been impressed a couple times. I
62:00 mean, definitely the machines are listening to us. um it's given me a
62:03 mockup of of a room or something and then it throws in a dog bed because I
62:07 have I have dogs and I'm like I did not tell you to do that but yeah that's
62:11 probably the Kylin color and style of a dog bed that I should have in this room.
62:14 >> Have Speaking of that, have you tried this use case? Ask JJBT uh generate an
62:20 image of what you think my house looks like based on everything you know about
62:22 me. >> I have it >> because it is memory. It remembers
62:27 everything you've talked about and it's hilarious. You got to do it. Okay,
62:31 that's that's on my to-do list. >> There we go. Uh, a bonus use case,
1:02:33 Lightning round and final thoughts
62:35 Nicole. With that, we've reached our very exciting lightning round. I've got
62:38 five questions for you. You ready? >> Awesome. Let's go.
62:41 >> What are two or three books that you find yourself recommending most to other
62:43 people? >> Outlive by uh Peter Aia is fantastic. Another one that's maybe related. I hurt
62:53 my back. Um, so like it's not great. Back mechanic by Stuart McIll is
62:58 incredible. So, shout out to anyone who has uh hurt lower back. Um, it's for a
63:02 lay person to read through and like figure out how to fix lower back
63:06 problems. It's kind of a random one. I will say I love how big things get done.
63:10 I can't pronounce the name. I think one's they're Scandinavian. One is um
63:15 but it's it kind of dissects really large projects through recentish history
63:22 and where they failed and why. And I think it's really interesting for us to
63:25 think about especially now in this AI moment where basically all of our at
63:29 least software systems are going to be changing. So how do we think about
63:32 approaching what is essentially going to be a very large project. Um and then
63:36 sorry I'm going to throw in a bonus one. The Undoing Project by Michael Lewis. Uh
63:41 Matt Velos recommended it to me and it's so good. >> Yes. Uh I
63:47 >> gasped at the last sentence. Oh, the book. It was Yeah, it was not.
63:50 >> I read that and I I do not remember that last sentence. So, man. Okay, cool. Uh,
63:55 next question. Do you have a favorite movie or TV show you recently watched
63:57 and enjoyed? >> I will say I watched Love is Blind if I
64:01 got to like shut down at the end of the day. Love is Blind is fun.
64:03 >> There's a new season out. >> Yeah, >> very excited. Um, and Shrinking.
64:07 Shrinking. Have you seen Shrinking? >> No, I I think I started the therapists
64:12 and Yeah, I gave it a shot. Okay. Okay. Sweet. Is there a product
64:17 you've recently discovered that you really love? Could be an app, could be
64:20 some kitchen gadget, some clothing. >> Uh, yeah. The Ninja Creamy is.
64:27 >> Did you say this last time? >> I don't know. I mean,
64:29 >> somebody said this and I still remember it. It's like a
64:32 >> You make ice cream and stuff with it, right? >> Yeah. You can basically freeze a protein
64:36 shake and then it turns it into ice cream. >> Oh man. >> Which is delicious. Um.
64:40 >> Oh. Uhhuh. >> Another one is a Jura coffee maker. I'd
64:45 love good coffee and I'm not great at making it. So, I can just push the
64:48 button and it'll give me anything I want, including like lattes,
64:51 cappuccinos, or anything. So, that's kind of fun. >> Sweet. Okay. Uh, do you have
64:55 >> sugar and caffeine? I just need a power through the day.
64:58 >> There's the there's the engineering productivity 101.
65:01 >> Yeah. >> Oh, man. Okay, two more questions. Do
65:05 you have a favorite life motto that you often find useful in work or life and
65:09 come back to in various ways? >> Yeah, I think one that's come up a
65:13 couple times. It's not like a verbatim thing. It's I think it's more the vibe,
65:16 but like hindsight is 2020, but it's also really dumb, right? I think if we
65:22 made the best decision we could at the time with the information that we had
65:27 available, like then it is what it is, right? If if you make a bad decision
65:31 because you made a bad decision and you you knew better and you had the
65:36 information, not great. But I I don't think we give ourselves or
65:40 other people enough grace because we always end up finding more information
65:42 out later. >> Here here final question. I was going to
65:46 ask you something else, but uh as we were preparing for this, you share that
65:49 you have a new role at Google. Maybe just talk about that. What you're up to
65:52 there, why you joined Google, anything folks should know. >> Sure. So I am senior director of
65:58 developer intelligence in core developer. So, it's super exciting and
66:03 super fun because of all of these things we've been talking about, right? It's
66:07 like focused on Google and all their properties and and their kind of
66:10 underlying infrastructure. How can we improve developer experience, developer
66:14 productivity, velocity, all of these things we've been talking about? And
66:18 because I'm kind of the numbers person, right? How do we want to think about
66:21 measuring it? How does measurement change? How do feedback loops change?
66:24 How can we improve the experience throughout and then kind of drive that
66:28 change through an organization in ways that are meaningful and impactful and
66:33 faster than they've been before? >> Nice job, Google getting Nicole. What a
66:38 win. I need to get some more Google stock ASAP. Okay, two follow questions.
66:41 Where can folks find you online if they want to and find your book online if
66:45 they want to dig deeper and how can listeners be useful to you?
66:49 >> So online you can find the book at developerexperiencebook.com.
66:53 Um, I'm at af.com and LinkedIn occasionally. Sometimes it's a mess. I I try to wade wave
67:01 through all of the uh noise I get there. Um, to be useful, um, sign up for the
67:06 book and the workbooks. The workbooks are free. I'd love to get, you know, any
67:11 kind of feedback on what works, what doesn't. Uh, I always love hearing those
67:15 kind of stories. >> Nicole, thank you so much for being
67:17 here. >> Thanks for having me, Lenny. >> My pleasure. Thanks again. Bye everyone.
67:25 Thank you so much for listening. If you found this valuable, you can subscribe
67:29 to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also,
67:33 please consider giving us a rating or leaving a review as that really helps
67:37 other listeners find the podcast. You can find all past episodes or learn more
67:43 about the show at lennispodcast.com.