How to measure AI developer productivity in 2025 | Nicole Forsgren

Measuring Productivity in Development Teams

The Challenge of Productivity Metrics

• Many companies are attempting to measure team productivity, but most metrics can be misleading.

• Focusing solely on the quantity of code (e.g., lines of code) can lead to gaming the system, as it's easy to produce large amounts of code without quality.

• The question arises: how do we assess if teams are truly moving fast enough? There is a risk that they could perform better but aren't due to various factors.

Quality Over Speed

• Rapid shipping of low-quality products ("shipping trash") is not beneficial; strategic decision-making is essential for determining what should be released.

• A significant amount of time may shift from writing code to reviewing it, necessitating a reevaluation of workflows and daily structures.

AI's Role in Development

• AI tools can assist developers by helping them regain focus and context, potentially enhancing productivity during work blocks.

• Nicole Forsrren discusses the impact of AI on developer productivity and raises questions about measuring these gains effectively.

Insights from Nicole Forsrren

Background and Expertise

• Nicole Forsrren has extensive experience in measuring developer experience through frameworks like Dora and Space.

• She authored "Accelerate" and is set to release "Frictionless," which aims to guide teams in navigating the evolving landscape influenced by AI.

Key Thesis on AI Impact

• While AI has the potential to accelerate coding processes, developers face challenges such as broken builds and unreliable tools that hinder speed.

Measuring Productivity Gains

• The discussion includes practical advice for assessing productivity improvements from AI integration within teams.

Common Pitfalls in Measuring Engineering Productivity

Missteps Companies Make

• Companies often misinterpret engineering productivity metrics, leading to ineffective strategies for improvement.

Balancing Benefits and Drawbacks of AI Tools

• The conversation highlights both positive impacts (like achieving flow states with assistance from AI tools) and negative aspects (new bottlenecks created).

Setting Up Developer Experience Teams

Seven-Step Process Overview

• Nicole outlines a seven-step process for establishing effective developer experience teams within organizations.

Gaining Buy-in for Initiatives

• Strategies for obtaining support from stakeholders when implementing new processes or teams are discussed.

Conclusion & Call-to-Actions

Engaging with Content

• Listeners are encouraged to subscribe to the podcast for more insights into improving engineering team performance.

Additional Resources

AI and Developer Productivity: A Deep Dive

The Impact of AI on Developer Productivity

• Discussion begins on the intersection of AI and developer productivity, highlighting the rapid emergence of AI as a focal point in conversations about engineering.

• Reflecting on the past two and a half years, there's an acknowledgment that while many things remain unchanged, the conversation around AI has intensified significantly.

• The speaker notes a surge in investment towards AI tools aimed at enhancing productivity, with a growing curiosity about their actual effectiveness in improving work output.

Understanding Developer Experience (DevX)

• Introduction to "DevX," or developer experience, which encompasses daily software building experiences for developers including friction points and workflows.

• Emphasis on the importance of good DevX; poor experiences can negate even the best tools or processes available to developers.

Components of Productivity and Happiness

• Discussion highlights that productivity is not solely about output but also involves broader aspects like engineering happiness and overall success within companies.

• The complexity of defining productivity is acknowledged; high toil or cognitive load can lead to burnout among developers.

Flow State in Engineering

• The concept of "flow state" is introduced as crucial for developer happiness and productivity; interruptions from new technologies like AI may disrupt this state.

• Personal anecdotes illustrate how flow state enhances job satisfaction during coding, contrasting it with challenges posed by modern tools.

Key Elements Affecting Developer Experience

• Three critical components affecting DevX are identified: flow state, cognitive load, and feedback loops. Each element reinforces one another in shaping developer experiences.

Workflow and Productivity in AI Development

Innovative Workflow for AI Development

• The speaker discusses a new workflow observed among AI engineers, emphasizing the creation of an organized workspace that enhances productivity.

• Engineers utilize prompts to define project requirements, including architectural components and technology stacks, allowing the system to design solutions effectively.

• The process involves assigning tasks to agents that work in parallel while ensuring compatibility and adherence to APIs and conventions.

• This systematic approach leads to code that is closer to production quality compared to traditional coding methods, which often lack planning.

Measuring Productivity with AI

• A core question arises regarding how companies measure productivity gains from AI; the speaker asserts that most current metrics are misleading.

• Lines of code have historically been a poor metric for productivity but continue to be used as a proxy for output or complexity despite their inadequacy.

• The ease of generating verbose code through prompts complicates the assessment of true productivity, leading to potential technical debt.

Reevaluating Metrics in Light of AI

• While lines of code are not effective as a standalone productivity metric, distinguishing between human-generated and AI-generated code can provide insights into quality and survivability rates.

• Understanding how much generated code contributes back into training systems is crucial for assessing biases and patterns introduced by AI tools.

Limitations of Existing Frameworks

• Traditional frameworks like Dora need reevaluation as they may not capture the rapid feedback loops enabled by AI technologies effectively.

• Dora's four key metrics (deployment frequency, lead time, mean time to recovery, change fail rate) remain relevant but must be adapted for modern workflows influenced by AI.

Adapting New Frameworks

• The speaker highlights that while Dora provides prescriptive metrics, it should not be applied blindly without considering changes brought about by AI integration.

Communication and Collaboration in AI Systems

Importance of Communication and Collaboration

• Emphasizes the significance of communication and collaboration within systems, highlighting how it affects both human interactions and system efficiency.

• Discusses the balance between offloading tasks to chatbots versus engaging with senior engineers, noting that neither extreme is inherently better.

Efficiency and Flow in Work Processes

• Introduces the concept of flow state, questioning how efficiently individuals can navigate through systems.

• Stresses the importance of trust in code generated by AI, especially given the nondeterministic nature of large language models (LLMs).

Trust Issues with AI Code Generation

Trust as a Central Concern

• Highlights trust as a critical issue when working with AI-generated code, referencing previous discussions on this topic.

• Notes that more time will be spent reviewing code rather than writing it, which may reshape work structures.

Rethinking Work Structures

• Suggests opportunities for rethinking workflows and daily structures due to changes in cognitive load from using AI tools.

• References research indicating that individuals can only achieve about four hours of deep work per day.

Deep Work vs. Interruptions

Challenges in Achieving Deep Work

• Acknowledges common experiences where productivity dips lead to less creative tasks like cleaning out inboxes instead of innovative problem-solving.

• Argues that achieving flow often requires longer uninterrupted periods, ideally two hours or more.

The Impact of Interruptions on Productivity

• Compares past work environments with current ones filled with interruptions, questioning whether traditional four-hour blocks are still effective.

• Proposes shorter work blocks might become useful due to machines assisting in maintaining context and flow.

The Role of Engineers as Coordinators

Shifting Roles for Engineers

• Observes a shift where engineers increasingly act as engineering managers coordinating junior AI engineers' tasks.

Enhancing Developer Experience (DevX)

• Discusses how even lengthy work blocks may not allow for deep coding but can facilitate unblocking tasks for AI assistants.

Business Value of Focusing on Developer Experience

Importance of DevX for Business Success

• Argues that enhancing developer experience is crucial for business value, enabling software creation that meets market needs.

Super Rapid Experimentation and Developer Experience

The Importance of Rapid Prototyping

• Modern tools enable super rapid experimentation with customers, allowing for quick prototyping and A/B testing that can be completed in hours rather than weeks.

Listening to Developers

• Product teams should prioritize talking to developers to understand their experiences. Listening is more effective than immediately implementing new tools or automation.

• Engaging developers in discussions about their daily tasks can reveal pain points and areas of friction that need addressing.

Identifying Process Improvements

• Conducting a "listening tour" helps identify processes that are unnecessarily complex or slow, which can often be improved without significant engineering effort.

• Developers are usually willing to share what frustrates them, providing valuable insights into potential improvements.

Simple Changes Can Have Big Impacts

• Sometimes, minor adjustments to existing processes (like changing from physical paperwork to email approvals) can significantly enhance efficiency without major overhauls.

Common Areas for Improvement

• Many companies have outdated processes that could benefit from simplification. Streamlining these processes often requires minimal engineering resources.

• Establishing lightweight team processes can help improve overall productivity, especially in larger organizations where complexity tends to accumulate.

Organizational Change and Communication

Supporting Structural Changes

• Business leaders should provide structure and support for organizational changes by clearly communicating priorities and celebrating wins.

Avoiding Isolation of Projects

• Isolated projects may struggle to gain traction; integrating them into the broader organizational context is crucial for sustained engagement.

Technical Considerations vs. People-Centric Approaches

Balancing Technical Needs with Human Factors

• While technical solutions are important (especially with AI advancements), they should not overshadow the need for process improvement and team morale.

Assessing Team Performance

Signs of Underperformance

Understanding Organizational Friction and Productivity

The Challenges of Switching Tasks and Projects

• Long processes in requesting new systems or provisioning environments can create friction, making it difficult for employees to switch tasks or projects.

• When team members express dissatisfaction with the system, it often indicates underlying friction that hinders productivity; high switching costs discourage movement within the organization.

• In some companies, changing departments incurs a "new hire tax" due to vastly different systems and processes, leading to significant challenges.

The Importance of Strategic Decision-Making

• While speed is essential in execution, it's crucial to consider the quality of decisions being made; faster output without strategic alignment can lead to poor outcomes.

• Product Managers (PMs) play a vital role in ensuring that teams ship valuable products rather than just increasing volume; strategy must guide what features are prioritized.

Identifying Signs of Low Productivity

• Indicators of low productivity include frequent build failures, flaky tests yielding false positives, and difficulties in context-switching between projects.

• AI's potential to enhance engineering speed is acknowledged; however, it’s emphasized that many aspects of engineering performance extend beyond mere coding efficiency.

Leveraging AI for Enhanced Strategy

• AI tools can assist in refining business strategies and experimentation methods but require a solid foundational plan before implementation.

• Rapid prototyping and A/B testing have become feasible within days instead of months due to improved infrastructure; however, careful planning is necessary to ensure relevance.

Collaborating with Data Science Teams

• Engaging with data science experts is critical when planning experiments; understanding what data is needed helps avoid ineffective tests that could disrupt user experience or violate privacy protocols.

Understanding AI's Impact on Productivity

Gains in Productivity with AI

• The speaker emphasizes that while the productivity gains from AI are real, there is a lack of established metrics to quantify these improvements effectively.

• A key measure for productivity is "velocity," which refers to how quickly features or products can move through development and testing phases.

• Rapid prototyping has been observed as a significant benefit of using AI tools, leading to an increase in code generation among regular users of AI coding environments.

• AI tools help unblock developers by speeding up their workflow, particularly during initial stages where starting tasks can be challenging.

• OpenAI Codex has shown remarkable capabilities in debugging complex issues, highlighting its utility in enhancing developer efficiency.

Enhancing Documentation and Unit Testing

• There is a growing reliance on AI for writing and cleaning documentation; however, effective documentation remains crucial for optimal tool performance.

• Better data quality, including thorough documentation and comments, leads to improved outcomes when utilizing AI tools.

Frictionless Development Teams: Insights from the Book "Frictionless"

Overview of the Book

• The book titled "Frictionless" outlines seven steps aimed at removing barriers within development teams to enhance value delivery and competitiveness in the age of AI.

• Co-authored with Abby Nota, who has extensive experience in developer experience (DevX), the book draws insights from numerous engineering leaders and CTOs.

Importance of Developer Experience

• The acquisition of Abby's company DX by Atlassian for a billion dollars underscores the high value placed on improving developer experience across organizations.

• Companies are increasingly focused on measuring productivity within their teams as they adopt tools like Jira; this reflects a broader trend towards optimizing DevX.

Target Audience for "Frictionless"

Framework for Improving Developer Experience

Seven-Step Process Overview

• The framework consists of a seven-step process aimed at enhancing developer experience, starting with initiating the journey.

• Step one involves conducting a "listening tour" to gather insights from team members and visualize current workflows and tools.

• Step two focuses on achieving quick wins by selecting manageable projects that can demonstrate immediate value.

• In step three, teams should leverage data to optimize their work processes, including collecting new data and utilizing surveys for insights.

• Step four emphasizes deciding on strategy and priorities based on the gathered data to identify which issues need addressing next.

Selling Strategy and Driving Change

• Step five is about selling your strategy; this includes gathering feedback and communicating why the chosen approach is beneficial.

• In step six, teams are encouraged to drive change at various scales—whether local (grassroots efforts) or global (top-down initiatives).

• The final step, step seven, involves evaluating progress and demonstrating value while allowing flexibility to jump into any stage depending on current needs.

Additional Practices for Success

• The discussion highlights the importance of resourcing, change management, sustainability in technology, and viewing developer experience as a product with measurable metrics.

Understanding Developer Experience

Importance of Developer Experience vs. Productivity

• The term "developer experience" is intentionally distinct from "developer productivity," focusing on overall satisfaction alongside efficiency.

• Emphasizing value over mere productivity metrics encourages engineers' creativity by allowing them autonomy in problem-solving rather than just measuring output like lines of code.

Starting a Developer Experience Team

• When forming a team focused on developer experience, it often begins with one or two engineers taking initiative alongside project managers or technical program managers for effective communication.

Building Momentum Through Quick Wins

• Identifying small-scale improvements ("paper cuts") can help demonstrate immediate benefits to developers' daily work experiences.

Scaling Up Initiatives

Impact of Engineering Teams on Company Performance

Financial Implications of Engineering Efficiency

• Smaller companies can see impacts in the hundreds of thousands, while larger firms may experience billions in savings due to improved engineering practices.

• The benefits often follow a J-curve pattern: initial quick wins followed by a dip as more complex projects require additional infrastructure and telemetry before compounding benefits are realized.

Measuring Success and Key Metrics

• Identifying key metrics is crucial; understanding who the audience is (developers vs. leadership) helps tailor communication about success.

• Developers value time savings and reduced manual toil, while leadership focuses on cost acceleration, revenue growth, and competitive speed.

Developer Perspective on Productivity Gains

• Developers appreciate automation that reduces manual steps in compliance and security processes, allowing them to focus on higher-value tasks.

• Leadership prioritizes financial metrics such as time to value and customer feedback velocity, which are critical for maintaining competitiveness.

Business Outcomes from Engineering Improvements

• Cleaning up test suites can lead to significant cloud cost savings by reducing unnecessary resource usage from failing tests.

• Correlating productivity gains with business outcomes can demonstrate how faster time-to-value contributes to increased market share.

Understanding AI's Impact on Productivity

• There is uncertainty regarding the impact of AI tools on productivity; organizations need clear strategies for measurement amidst rapid changes.

Understanding Developer Productivity and AI Tools

Framing the Problem

• When addressing developer productivity, it's crucial to align terminology with what stakeholders value, such as "velocity" or "transformation," to ensure resonance and clarity.

• The initial step for companies assessing AI tool impact is identifying key concerns of leadership, which may include market share, profit margins, or velocity.

Measuring Impact

• Companies should establish metrics that reflect their priorities; for instance, tracking time from feature idea to production can indicate efficiency improvements.

• For measuring velocity effectively, consider broad metrics like the duration from idea to customer feedback and how AI tools reduce friction in this process.

Attribution Challenges

• It's important to disclose contributions from both AI tools and developer experience (DevX) improvements when evaluating performance changes.

• Acknowledging the interplay between DevX efforts and AI implementations helps clarify their respective impacts on productivity.

Starting Metrics for Developer Experience

• If beginning measurement of developer experience without existing data, conducting surveys can provide a quick overview of challenges faced by developers.

• Surveys should focus on satisfaction levels and barriers to productivity; asking respondents about their top three challenges can yield actionable insights.

Effective Survey Design

• Crafting clear survey questions is essential; avoid multi-faceted questions that could confuse respondents about specific issues affecting them.

Understanding Developer Satisfaction and Tools

The Role of Happiness Surveys in Engineering

• The speaker mentions that their book includes example surveys for easy implementation, suggesting a structured approach to gathering feedback.

• They express a dislike for happiness surveys, arguing that happiness is influenced by various factors such as work, family, and hobbies.

• Instead of measuring happiness, the speaker advocates for satisfaction surveys which can provide actionable insights about specific tools or aspects of work.

• They humorously relate the concept with "happy devs make happy code," emphasizing that satisfied developers produce better work and collaborate more effectively.

• The speaker concludes that while capturing happiness directly is challenging, measuring satisfaction can yield valuable signals.

Popular Tools Among Developers

• The discussion shifts to commonly used tools in development; notable mentions include Copilot and Claw Code.

• Claw Code is highlighted as an underrated AI tool capable of performing various tasks beyond coding, like cleaning up storage on devices.

• The speaker shares enthusiasm for exploring non-engineering use cases for Claw Code, indicating its versatility.

Adapting to AI in Development

• Emphasizing a product mindset is crucial when improving developer experience (DevX), focusing on identifying user problems and creating MVP experiments.

• Continuous feedback from users is essential; understanding what success looks like helps guide improvements effectively.

• The importance of reassessing metrics over time is discussed—some long-standing metrics may no longer be relevant due to rapid changes brought by AI tools.

Personal Use Cases of AI Tools

• In the segment titled "AI Corner," the speaker shares personal experiences using AI tools like ChatGPT and Gemini for home design projects.

• They describe how these tools assist in visualizing room layouts by modifying images based on user input, showcasing practical applications outside traditional engineering contexts.

What Insights Can We Gain from AI's Understanding of Our Lives?

The Role of AI in Personalization

• The speaker discusses how machines, particularly AI, are capable of listening and learning about personal preferences, exemplified by a mockup that includes a dog bed based on the user's pet ownership.

• A suggestion is made to use AI to generate an image of what the speaker's house might look like based on their interactions with it, highlighting the memory aspect of AI.

Recommended Books for Personal Growth

• "Outlive" by Peter Attia is recommended as a fantastic read. It focuses on health and longevity.

• "Back Mechanic" by Stuart McGill is highlighted for its practical advice on addressing lower back issues, making it accessible for laypersons.

• "How Big Things Get Done" explores large project management failures and successes, relevant in the context of evolving software systems due to AI advancements.

• "The Undoing Project" by Michael Lewis is mentioned as a compelling read that left a strong impression on the speaker.

Entertainment Recommendations

• The speaker enjoys watching "Love is Blind," indicating it's a fun way to unwind at the end of the day.

• They also mention enjoying "Shrinking," which features therapists and offers comedic relief.

Favorite Products Recently Discovered

• The Ninja Creamy is praised for its ability to turn frozen protein shakes into ice cream, appealing to those who enjoy healthy treats.

• A Jura coffee maker is highlighted as an easy solution for making quality coffee drinks like lattes and cappuccinos without much effort.

Life Lessons and Career Updates

• The speaker shares their life motto emphasizing that while hindsight may be 20/20, decisions should be evaluated based on available information at the time they were made.

• They announce their new role as Senior Director of Developer Intelligence at Google, focusing on improving developer experience and productivity through measurement and feedback loops.