The UX Research reckoning is here | Judd Antin (Airbnb, Meta)

User-Centered Performance and Customer Obsession

The discussion delves into the concept of user-centered performance, emphasizing customer obsession and signaling dedication to customer-centric practices.

User-Centered Performance

• User-centered performance signifies a focus on customer obsession rather than genuine learning.

• Common scenario: PMs seek user studies at the end of product development merely to validate assumptions, showcasing user-centered performance that lacks substantial impact.

• Emphasizes the importance of falsifying over validating in research, highlighting a prevalent trend where PMs and designers aim to confirm rather than challenge assumptions.

The Evolution and Challenges in User Research

Explores the evolution of user research, challenges faced within the field, and strategies for effective utilization.

Evolution and Challenges

• Guest speaker Jud Anon's background in building user research practices at Facebook and Airbnb.

• Discussion on the ongoing transformation in user research field necessitating changes both internally and in how companies engage with research.

• Critique on PMs and designers relying on user research for validation instead of seeking answers to pertinent questions.

Insights on Leveraging User Research Effectively

Provides insights into maximizing the value of user researchers, aligning with business goals, hiring strategies, and fostering better collaboration between researchers and PMs.

Leveraging User Research

• Importance of researchers focusing on business objectives alongside user needs.

• Introduction to "user-centered performance," where individuals feign concern for users but prioritize predetermined decisions.

Reflection on Career Growth Opportunities

Discusses career reflection opportunities using sponsor platforms like Teal for job searches.

Career Reflection

• Sponsorship by Teal for career reflection tools aiding job searches effectively.

Acknowledgment of Sponsors - Vanta

Acknowledges Vanta as a sponsor offering security compliance solutions for businesses.

Sponsor Acknowledgment - Vanta

New Section

In this section, the speaker reflects on the unexpected response to their writing and the impact it had on initiating conversations.

Reflections on Writing

• The speaker wrote to initiate a conversation without knowing the audience reach.

• Using the term "Reckoning" sparked drama unintended for a positive discussion.

• Despite some critiques, many found the content resonant and engaging.

Exploring Research Critiques

This part delves into various critiques received by the speaker regarding research practices and responsibilities.

Research Critiques

• Some perceived the speaker as blaming researchers for shortcomings, which was not their intention.

• A notable critique came from those opposing profit-focused research, questioning its alignment with research ethics.

Challenges in Research Impact

Here, the focus shifts towards challenges in research impact within organizations and potential reasons behind these issues.

Research Impact Challenges

• The system may be flawed as research fails to deliver expected value and impact due to various factors.

• Differentiating between macro, middle-range, and micro research types highlights areas where improvements are needed for impactful outcomes.

The Role of Middle Range Research

Discussing middle-range research's significance in understanding user behavior while highlighting its limitations in driving business impact.

Middle Range Research Insights

• Middle-range research focuses on user understanding but often lacks significant business impact despite being intriguing to researchers.

Research Impact and Integration

In this section, the speaker discusses the importance of research in driving impact and the need for integrating research throughout the product development process.

Importance of Research Impact

• Research is crucial for driving impact, but there has been a lack of understanding on how to utilize it effectively.

• Differentiates between middle-range research and macro research, highlighting that middle-range research focuses on specific questions or constituencies.

• Middle-range research provides insights into more specific areas such as understanding how Airbnb hosts feel about payment options.

• Middle-range insights may not be targeted at a business problem directly but offer valuable information like host preferences for flexibility in payment options.

Integrating Research for Maximum Impact

• Delay in decision-making often occurs when research is not integrated from the beginning, leading to issues where decisions rely heavily on additional data points rather than trusting existing insights.

• Product managers should deepen their collaboration with researchers to leverage insights effectively for maximum impact.

• Restructuring product development processes to integrate research fully from start to finish can enhance the role of researchers in driving impactful decisions.

Challenges in Research Integration

This part delves into challenges faced when research is treated as a service function rather than an integral part of the product development process.

Challenges Faced with Service-Oriented Research

• Research being viewed as a service function leads to reactive involvement instead of proactive engagement throughout projects.

• Lack of continuous engagement results in missed opportunities for researchers to shape critical questions that drive business impact and product improvement.

The Vicious Cycle of Reactive Research

The speaker highlights a cycle where companies hire researchers without integrating them effectively, leading to less impactful outcomes and potential layoffs.

Reactive Nature of Research Integration

• Companies hiring researchers with good intentions but failing to integrate them properly result in reactive rather than proactive research practices.

Researcher Skills and Business Orientation

The discussion revolves around the importance of researchers in product processes, highlighting the skills that make a great researcher and how to evaluate these skills during interviews.

Researcher Skills Evaluation

• Great researchers are multi-method, utilizing formative or generative research for innovation, evaluative research for usability testing, rigorous survey design for community responses, applied statistics knowledge, and technical skills like SQL or prompt engineering.

Interviewing Researchers

• When evaluating researchers during interviews, look for those who can handle open-ended questions with a multi-method approach. Assess their ability to propose various methods and timeframes for research projects.

Building Research Teams

• Building a successful research team involves individuals with diverse expertise in the five key tools. Team members should be t-shaped experts with deep knowledge in specific areas to cover all necessary skills.

Business-Oriented Research Practices

The conversation shifts towards the necessity for researchers to be more business-oriented rather than solely user-focused.

Balancing User-Centeredness and Profit Focus

• Researchers need to find an overlap between user needs and business goals. They should explicitly consider how their work contributes to both aspects rather than focusing solely on empathy and user-centered design.

Incorporating Business Understanding

User-Centered Performance in Product Management

In this section, the speaker discusses the importance of user-centered skills in product management and the evolution needed in this aspect.

Evolution of User-Centered Skills

• Historically, user-centered skills have not been a primary focus in product management over the last 15 years.

• Design often falls into this category as well, where convincing researchers and teams about the significance of user-centered approaches is crucial.

Importance of User-Centeredness

• Emphasizes that fixing the broken system requires involvement from everyone - researchers, product managers (PMs), designers, and organizational structures.

• Introduces "user-centered performance" as a term to highlight customer obsession or user-centered practices that are symbolic rather than focused on learning.

Challenges with User-Centered Performance

This part delves into common pitfalls related to user-centered performance and provides insights on how to address them effectively.

User-Centered Performance Pitfalls

• Discusses how some PMs engage in "check-the-box" style research for validation rather than genuine learning.

• Highlights examples like quick user studies requested at the end of a product process as instances of user-centered performance that lack impact.

Implicit Engagement in User Performance

• Mentions implicit ways individuals engage in performative user-centeredness due to cognitive biases like confirmation bias and ego.

• Advocates for a mindset shift towards falsification over validation in research to drive better outcomes.

Balancing Intuition and Evidence

This segment explores the interplay between intuition and evidence-based decision-making in product development.

Role of Intuition

• Acknowledges the importance of intuition but cautions about biases inherent within it.

Gut Feeling vs. Data-Driven Decisions

The discussion revolves around the balance between trusting one's intuition (gut feeling) and relying on data-driven decisions in product management and design.

Gut Feeling and Decision Making

• Product managers and designers should find a balance between trusting their gut feelings and utilizing research-driven insights.

• Great leaders develop intuition through experience, enabling them to make sound judgments based on pattern matching.

• Intuition can lead to biases, so it is crucial to check your gut feeling by engaging in slow, analytical thinking processes.

Leveraging Research for Informed Decisions

• Engage in systematic, methodical thinking (system two) to complement gut feelings with rational analysis.

• Utilize the wisdom of the crowd by gathering diverse perspectives and information sources when making decisions.

Impact of Research Teams in Product Management

The importance of integrating research teams into product management processes for effective decision-making and successful outcomes.

Collaborative Metrics for Success

• All team members, including researchers, should share common metrics for success to foster collaboration and unity in decision-making processes.

• Success is achieved when teams can make decisions without the researcher present, indicating strong relationships and active participation in the process.

Role of Product Managers

• A great product manager enhances team efficiency by facilitating smooth operations within the group.

Challenges in Researcher-Product Manager Relationships

Addressing challenges faced by researchers in establishing effective relationships with product managers.

Relationship Dynamics

• Researchers often feel excluded from the decision-making process led by product managers, leading to strained relationships.

New Section

In this section, the speaker discusses the importance of research at both macro and micro levels, highlighting how detailed insights can drive business value.

Importance of Research in Business

• The speaker emphasizes the significance of macro-level research tied to annual planning processes, citing examples from Airbnb's concept car projects.

• Detailed research can lead to valuable insights for future products, such as creating a concept car for five years ahead.

• An anecdote from Airbnb illustrates how changing just seven characters on a button led to a significant increase in revenue within 48 hours.

• Quick and focused research efforts, like testing different CTAs, can result in substantial business growth within a short timeframe.

New Section

This part delves into the balance between micro and macro research in product development and planning cycles.

Balancing Micro and Macro Research

• Emphasizes the importance of focusing on micro-level research for usability improvements before diving into broader planning cycles.

• Challenges the notion that usability research is only for junior researchers, highlighting its impact on enhancing product functionality and business metrics.

• Advocates for integrating research throughout the product development process to ensure informed decision-making from start to finish.

New Section

This segment addresses common misconceptions about user research and highlights the role of researchers in generating valuable insights.

Misconceptions About User Research

• Counters the belief that anyone can conduct effective user research by emphasizing the expertise needed to derive meaningful insights from interactions with users.

• Discusses how researchers help transform raw data or feedback into actionable insights while mitigating biases that may arise from informal user engagements.

Understanding the Value of Data and Research in Product Management

The discussion delves into the importance of data and research in product management, addressing common critiques and highlighting the significance of understanding both causal relationships and user behavior.

Importance of AB Testing and Research Collaboration

• AB testing is crucial for making causal claims about products, while research focuses on understanding how and why things happen.

• Collaboration between data scientists, researchers, and insights professionals is essential for developing better products by combining statistical evidence with in-depth understanding.

Challenging Tropes in Research

• Disputing the validity of a popular Henry Ford quote emphasizes the role of researchers in not relying solely on customer input but seeking deeper insights.

• Discussing post-talk bias highlights the tendency to oversimplify past events based on hindsight, emphasizing the need to avoid constructing narratives that align with preconceived notions.

Cognitive Biases and Narrative Fallacy

• Duncan Watts' book "Everything is Obvious if You Already Know the Answer" underscores how intuitionistic thinking can lead to self-deception through selective memory and narrative construction.

• Reflecting on personal experiences challenges the narrative fallacy by acknowledging uncertainty in tracing one's path, cautioning against overly simplistic storytelling.

User Research: Enhancing Mental Models for Product Development

The conversation explores the dual role of user research in shaping mental models for product development, emphasizing its value in refining user understanding and guiding product decisions.

User Research as Model Enhancement

• User research serves to improve mental models by providing insights into user needs, yet it should not dictate product decisions entirely but rather inform them.

Developing a Framework Based on Evidence

In this section, the speaker emphasizes the importance of building a framework based on evidence and collaboration to achieve success in product development.

Advice for Teams and Researchers

• The speaker advises teams and researchers to move beyond traditional approaches and embrace experimentation to progress in the field.

• Research is evolving rather than dying, prompting the need for researchers and companies to adapt to this new version effectively.

• Encouragement for researchers to develop diverse research skills, deepen business knowledge, speak the language of product and business, and build strong relationships within their teams.

Creating Integrated Insights Functions

This part focuses on breaking down silos between different insights disciplines within companies to create an integrated approach towards research.

Breaking Down Silos

• Siloing different insights disciplines is deemed counterproductive; integrating various insights streams can enhance overall understanding.

• Companies should aim to integrate all insights sources into one cohesive machine for optimal efficiency.

Enhancing Collaboration Between Product Managers and User Researchers

The discussion centers around improving collaboration between product managers and user researchers for more effective outcomes.

Collaborative Process

• Product managers should integrate user researchers throughout the product development process from start to finish.

Interview Insights

In this section, the speaker discusses their experience with user research and shares a story related to Facebook's post quality evaluation.

User Research Experience

• The speaker recalls a time at Facebook when they led a team working on post quality evaluation in Newsfeed during the rise of ads in 2012-2013.

• They describe how some engineers proposed using hidden posts as a signal for post quality, prompting the speaker to investigate the distribution of hiding behavior on Facebook.

• Discovering a power law distribution of hiding behavior, they identified "super hiders" and conducted user interviews to understand their actions better.

Insights from User Study

• During a user study, a participant consistently hid stories on Facebook due to aiming for inbox zero, surprising researchers and challenging assumptions about user behavior.

• This unique behavior highlighted the importance of individual insights over statistical significance in research, leading to reevaluation of signals like post hides.

Research Challenges and Solutions

The conversation shifts towards the value of individual insights in research and how they can challenge established beliefs.

Value of Individual Insights

• The story exemplifies how one unique user's behavior can shatter common assumptions, emphasizing the power of individual insights over statistical generalizations.

• Individual insights can lead to significant discoveries without relying solely on statistical significance, encouraging validation based on compelling examples rather than absolute certainty.

Application in Research Practices

• Drawing parallels with another study involving hosts' comfort levels with instant bookings based on Facebook connections highlights the impact of individual behaviors on research outcomes.

• The reluctance observed in Parisian hosts regarding sharing data with Facebook underscores broader privacy concerns that transcend cultural boundaries.

Optimizing Research Teams

The discussion delves into optimizing research teams amidst organizational changes and evolving needs for user research expertise.

Team Optimization Strategies

• Reflecting on layoffs within user research teams, there is an exploration into determining optimal researcher-to-team ratios amid shifting priorities and resource allocations.

How to Advocate for Research in a Company

In this section, the speaker discusses the importance of advocating for research within a company and how to approach getting more headcount for researchers.

Approaching Headcount for Researchers

• The speaker's approach to getting more headcount is by asking partners who see the need for research to advocate for it.

• The number of researchers in early-stage companies is often minimal, but having even one can drive significant value.

• A good researcher can accelerate decision-making and provide insights that are otherwise unattainable.

The Value of Researchers in Startups

This part emphasizes the impact researchers can have on startups and late-stage companies, highlighting their role in driving progress and decision-making.

Impact of Researchers

• Researchers help companies move faster and make informed decisions that lead to impactful outcomes.

• Matching researchers with strong partners across different business areas optimizes their effectiveness within a company.

Researcher vs. Engineer Hiring Decision

This segment delves into the trade-offs between hiring a researcher versus an engineer in startup scenarios, emphasizing the unique value researchers bring to the table.

Hiring Considerations

• Hiring a researcher can offer more leverage and speed compared to hiring an engineer.

• Having a versatile researcher on board can be invaluable during pivotal moments like product iterations or pivots.

Empowering Researchers and Driving Impact

Here, the focus shifts towards empowering researchers within organizations, encouraging them to drive impactful research initiatives and shape their careers proactively.

Empowerment Strategies

• Researchers are urged to push back against ineffective research practices and strive for meaningful contributions.

Detailed Analysis of NPS (Net Promoter Score)

In this section, the speaker provides a critical analysis of the Net Promoter Score (NPS) and its limitations, contrasting it with customer satisfaction as a more effective metric.

Critique of NPS

• The Net Promoter Score (NPS) is criticized as an inaccurate metric by the survey science community.

• NPS is flawed due to using a 0 to 1 scale and being unlabeled, leading to imprecise data collection.

• The likelihood to recommend question in NPS is deemed flawed as it may not apply universally across different products or services.

• Customer satisfaction is proposed as a superior alternative to NPS, offering better data quality and correlation with business outcomes.

Research Findings on Customer Satisfaction

• Research conducted with Airbnb's survey science team supports the superiority of customer satisfaction over NPS in predicting business success.

• The recommended customer satisfaction question focuses on overall experience satisfaction rather than likelihood to recommend, providing more reliable insights.

Limitations of Benchmarking with NPS

• Comparing NPS scores across industries for benchmarking purposes is cautioned against due to the idiosyncratic nature of NPS results and variations in survey administration.

Product Walkthroughs and User Perspective

This segment delves into the importance of product walkthroughs from a user perspective and challenges the notion that relying solely on personal usage experiences can lead to biased product evaluations.

Importance of Product Walkthroughs

• Emphasizes the significance of conducting product walkthroughs for identifying potential issues within products effectively.

• Advocates for prioritizing identified issues based on user feedback rather than personal opinions or intuition during product development processes.

User Perspective in Product Development

• Highlights the bias that can arise when product managers rely solely on their own usage experiences, emphasizing the need for diverse perspectives in evaluating product features.

Book Recommendations

The speaker shares book recommendations spanning business literature and fiction genres, offering valuable insights into leadership, social issues, and entertainment.

Recommended Books

• "Bad Leadership" by Barbara Kellerman explores characteristics of ineffective leaders, providing valuable lessons beyond traditional leadership discussions.

Murderbot Diaries and Favorite Recent Movie/TV Show

The conversation delves into entertainment preferences, including the Murderbot Diaries series and a recent favorite movie or TV show.

Murderbot Diaries

• The Murderbot Diaries feature a sarcastic killer robot, offering an engaging read for fiction enthusiasts.

Favorite Recent Movie/TV Show

• "The Last of Us" left a lasting impact after watching it post-playing the video game.

• Emphasizes the importance of experiencing the video game before the show.