Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor)

Marketplace Management Insights

The Whac-a-Mole Analogy in Marketplaces

• Marketplaces can be likened to a game of whac-a-mole, where addressing one issue often leads to another emerging problem.

• An example is provided where custom features improved the experience for new suppliers but negatively impacted existing ones, illustrating the constant balancing act in marketplace management.

• Changes made to enhance one side of the market can create winners and losers, necessitating careful consideration of which outcomes are more beneficial for the business.

Guest Introduction: Ramesh Johari

• Ramesh Johari is introduced as a Stanford professor specializing in data science methods related to online marketplaces.

• He has extensive experience advising major companies like Airbnb, Uber, and Stripe on building successful marketplaces.

Key Topics of Discussion

• The conversation will delve into strategies for fostering marketplace growth and effective resource allocation.

• Importance of data science in developing successful marketplaces will be emphasized, alongside discussions on review systems and founder perspectives.

• AI's impact on data science and experimentation within marketplaces will also be explored.

Sponsor Messages

Sanity CMS Promotion

• Sanity is highlighted as a powerful headless CMS that enables rapid content creation and management without developer constraints.

• Companies like Figma and Riot Games utilize Sanity for scalable content growth engines.

Hex Data Tool Promotion

• Hex is presented as an all-in-one tool for data analysis that integrates SQL, Python, and no-code solutions with collaborative features.

• It offers AI tools that assist users in generating queries and visualizations from natural language prompts.

Opening Remarks with Ramesh Johari

• Lenny thanks Riley Newman for connecting him with Ramesh; Riley's background at Airbnb sets the stage for discussing data-driven marketplace strategies.

What is a Marketplace Business and the Role of Data?

Understanding Marketplace Businesses

• A marketplace business connects buyers and sellers, but it’s crucial to understand that they don’t sell products directly. For example, Airbnb and Uber facilitate transactions between hosts/drivers and customers.

• The essence of these platforms lies in reducing friction for users. They eliminate transaction costs associated with finding accommodations or rides, which is often overlooked.

• Market failures occur when potential transactions cannot happen due to frictions, such as lack of information about available services or providers.

• Friction examples include not knowing who can provide a ride or accommodation at a specific time. Marketplaces address these issues by connecting users efficiently.

• Both sides of the marketplace (hosts/drivers and guests/riders) are considered customers. Each side relies on the platform to reduce friction in their respective transactions.

The Importance of Data in Marketplaces

• Entrepreneurs must recognize that their value proposition revolves around minimizing transaction costs; this understanding is vital for building successful marketplace models.

• Historical marketplaces like Agoras illustrate how physical constraints limited flexibility; modern technology allows for dynamic adjustments in marketplace structures based on data insights.

• Data science plays a critical role in three main areas: finding matches between users, making those matches effectively, and learning from past interactions to improve future matches.

• Finding matches involves identifying available listings or drivers based on user needs. This requires robust algorithms to connect supply with demand efficiently.

• After making matches, feedback systems collect data on user experiences (e.g., ratings), which informs future matchmaking processes and enhances overall service quality.

Continuous Improvement through Feedback Loops

• The cycle of finding potential matches, making them, learning from outcomes, and refining processes illustrates how data science underpins effective marketplace operations.

Marketplace Challenges and Insights

Common Flaws in Marketplace Startups

• The speaker notes that many marketplace ventures, such as cleaning services and car washes, often fail to succeed. This raises questions about the viability of on-demand marketplaces.

• Acknowledging their experience with various marketplace companies, the speaker emphasizes that they may not disclose specific company names due to sensitivity but aims to provide valuable insights.

• The primary flaw identified is that aspiring marketplace founders tend to overthink the concept of a marketplace before it has even been established.

Case Study: UrbanSitter

• UrbanSitter is introduced as a successful babysitting marketplace. Initially, it addressed a significant friction point: the need for cash payments for babysitters who typically do not accept checks or credit cards.

• By allowing credit card payments, UrbanSitter reduced this friction and leveraged Facebook networks for trusted introductions between parents and sitters.

• Once liquidity was achieved on their platform, UrbanSitter shifted its focus towards solving additional frictions related to finding and matching sitters.

Evolution of Marketplace Value Proposition

• The speaker explains that early-stage marketplaces should prioritize identifying unique value propositions rather than trying to operate at scale from the outset.

• They illustrate this with oDesk's initial approach, which focused on providing tools for remote workers to verify their work hours—addressing trust issues rather than liquidity concerns.

Key Takeaways for Aspiring Founders

• The moral drawn is that a true marketplace business does not start as one; instead, founders must identify specific problems they can solve without having scaled operations.

• Founders are encouraged to recognize that 90% of challenges faced will be similar to those encountered by any startup—not just unique to marketplaces.

Rethinking Marketplace Foundership

Marketplace Founders: A New Perspective

The Concept of Marketplace Founders

• Every entrepreneur can be viewed as a marketplace founder due to the pervasive influence of online transactions on business models.

• OpenAI, despite not identifying as a marketplace, operates as one through its plugins, creating a two-sided market between plugin creators and users.

Challenges in Building Marketplaces

• Founders must be cautious about overcommitting to future business models that may limit their flexibility later on.

• Early decisions regarding trust-building and monetization strategies can restrict options for growth and adaptation in the long run.

Disintermediation Risks

• As relationships mature on platforms like oDesk (now Upwork), the platform's value diminishes, leading to potential disintermediation where users no longer need the platform.

• Real-life examples illustrate how users may bypass platforms after initial connections are made, highlighting the importance of sustainable monetization strategies.

Case Studies: Successes and Failures

• Substack exemplifies a successful marketplace by enhancing value for writers through subscriber acquisition efforts, demonstrating effective demand generation.

• In contrast, eBay faced backlash from sellers when it altered fee structures, breaking established social contracts with long-term users who relied on the platform for their livelihoods.

Advice for Aspiring Marketplace Founders

Understanding Scaled Liquidity in Marketplace Businesses

The Concept of Scaled Liquidity

• The speaker introduces the idea of "scaled liquidity" as a crucial factor for marketplace businesses, emphasizing the need for both buyers and sellers to be present on the platform.

• Scaled liquidity is defined as having a significant number of buyers and sellers; lacking either side means the business cannot truly be classified as a marketplace.

• If one side (buyers or sellers) is strong, businesses can focus on leveraging that strength to attract the other side, highlighting strategic growth options.

Strategies for Attracting Users

• An example from Uber illustrates how they incentivized drivers with free ride coupons to build a user base, demonstrating effective strategies to create market liquidity.

• The speaker stresses that if neither side has scaled liquidity, entrepreneurs should prioritize scaling one side rather than forcing a marketplace model prematurely.

Business Models and Unit Economics

• Discussion shifts towards alternative business models like DoorDash, questioning whether it’s feasible to operate without being a traditional marketplace.

• The conversation touches on economic theories regarding firms versus markets, referencing Ronald Coase's insights about transaction costs and efficiency in labor matching.

Curation of Labor Pools

• The importance of curation in labor relationships is highlighted; not all marketplaces require freelance arrangements.

• Examples like Stitch Fix illustrate how curated experiences can enhance customer satisfaction by fostering personal connections rather than random assignments.

Leveraging Data for Marketplace Efficiency

• A transition into data utilization emphasizes its role in optimizing marketplace operations.

Understanding Pricing and Decision-Making in Marketplaces

The Role of Pricing in Marketplaces

• Discussion on the pricing structure for services like DoorDash, emphasizing that the fee paid is distinct from restaurant prices and may include surcharges.

• In platforms like Airbnb, hosts set their own prices, contrasting with marketplaces where the platform dictates fees.

Importance of Data Science in Marketplaces

• Highlights the necessity for data scientists to analyze pricing based on real-time supply and demand dynamics within a marketplace.

• Introduces machine learning models as a primary task for data scientists, focusing on predictive analytics rather than just data collection.

Predictive Analytics in Hiring Processes

• Shares an anecdote about predicting which workers are likely to be hired on oDesk using historical job and applicant data.

• Explains how algorithms utilize past hiring patterns to improve decision-making processes regarding applicant selection.

The Value of Human Insight in Data Interpretation

• Discusses the importance of human judgment alongside algorithmic predictions, stressing that algorithms identify patterns but do not inherently add value.

• Compares marketing strategies by illustrating how lifetime value (LTV) predictions should focus on incremental spending rather than absolute values.

Differentiating Between Prediction and Decision-Making

• Emphasizes that understanding differences in LTV due to promotions is more critical than merely identifying high-value customers.

Understanding the Distinction Between Correlation and Causation

The Importance of Causation in Decision Making

• Prediction relies on correlation, but decision-making requires understanding causation. Businesses must evaluate whether actions will genuinely enhance value.

• Data scientists should always keep in mind that their role is to assist businesses in making informed decisions, emphasizing the critical difference between prediction and decision-making.

Transitioning from Machine Learning to Causal Inference

• The concept of causal inference is essential for data scientists as they shift focus from merely predicting outcomes to understanding the impact of decisions.

• A practical example includes search and recommendation systems, where ranking algorithms play a crucial role in determining user preferences.

Evaluating Ranking Algorithms

• When comparing ranking algorithms, it’s vital to assess their effectiveness based on future matches rather than past predictions.

• Core metrics for businesses like Airbnb include bookings and revenue; thus, evaluating which algorithm leads to more bookings is paramount.

Quality of Matches Over Predictive Accuracy

• In hiring scenarios, the focus should be on assessing match quality rather than just predictive accuracy. Feedback from hires can provide insights into algorithm effectiveness.

• Future evaluations should center around how well algorithms facilitate valuable business outcomes rather than simply recreating historical data.

The Role of Experimentation in Business Decisions

Introduction to Eppo's Experimentation Platform

• Eppo offers advanced A/B testing capabilities designed for modern growth teams, enhancing experimentation velocity while providing deep analytical insights.

Balancing Experimentation with Strategic Opportunities

Understanding the Role of Experiments in Business Decisions

The Importance of Experimentation

• The speaker emphasizes the significance of experimentation in business, stating that it is essential for making informed decisions. They express a strong belief in working with businesses that prioritize experiments.

Limitations of Experimentation

• While advocating for experimentation, the speaker cautions against the notion that one can experiment their way out of every problem. They highlight that there are various degrees of freedom in what constitutes "testing everything."

Decision-Making Before Experiments

• Prior to running experiments, organizations must make critical choices about what to test and how long to run these tests. These decisions are influenced by organizational structure and team dynamics.

Risk Aversion in Testing

• The speaker notes a tendency towards risk aversion in experimentation, leading companies to focus on incremental changes rather than bold innovations. This often results in prolonged testing periods.

Incentives Affecting Experimentation Culture

• Companies deeply invested in experimentation may create incentive structures that encourage data scientists to pursue safer, more incremental wins rather than exploring riskier opportunities. This can stifle innovation.

Learning vs Winning: A Cultural Shift Needed

Redefining Success in Experiments

• The speaker critiques the common practice of labeling experimental outcomes as "wins" or "losses," arguing this mindset detracts from the true purpose of experimentation—learning from hypotheses tested.

Insights from Failed Experiments

• An example is provided regarding marketplace features like badging; even if an idea fails (e.g., badges negatively impacting user attention), valuable insights about user behavior can still be gained.

Emphasizing Learning Over Metrics

• The discussion stresses that learning should be viewed as a success metric instead of merely counting wins or losses. This shift requires cultural acceptance within organizations.

Changing Employee Contracts Around Experimentation

• To foster a culture where learning is prioritized over winning, companies need to adjust their employee contracts and expectations for data scientists regarding experimental outcomes.

Case Study: Badging at Airbnb

Launching Superhost Badge at Airbnb

Ranking Algorithms and Experimentation in Marketplaces

The Impact of Badges on Listings

• A data scientist expressed concerns about adding badges to random listings, fearing it would undermine the effectiveness of a well-crafted ranking algorithm designed to predict successful bookings.

• An experiment was conducted showing the badge to some users, revealing no significant impact on business metrics, although hosts reported feeling more satisfied with their status.

Understanding Data Science in Business Context

• Emphasis is placed on not disregarding business understanding when interpreting experimental results; data science should accumulate evidence rather than rely on isolated findings.

• The long-term effects of features like Superhost are challenging to measure due to short-term inventory rebalancing that creates winners and losers among hosts.

Competing Beliefs and Decision Making

• Encouragement for teams to adopt a "quantified" approach rather than strictly "data-driven," allowing for consideration of varying beliefs about retention value within leadership discussions.

• Suggestion to integrate experimental results with existing beliefs about the business, akin to a prediction market, facilitating informed decision-making despite flat short-term test outcomes.

Marketplace Dynamics and User Experience

• Reflection on the Superhost feature as a valuable addition to Airbnb's marketplace, even without initial evidence of impact; it contributes positively to user experience.

• Discussion on how marketplaces resemble a game of whac-a-mole where shifting attention can create unintended consequences for different user groups.

Managing Marketplace Changes Effectively

• Noted that focusing attention on certain hosts may detract from others' experiences; this dynamic can lead to fluctuating booking metrics across the platform.

Understanding Marketplace Dynamics and Experimentation

The Trade-offs in Marketplace Features

• Recognizing the balance between winners and losers in marketplace features is crucial for business success, despite the discomfort it may cause to acknowledge negative impacts on certain users.

Short-term vs Long-term Gains

• Decisions made today may not yield immediate positive outcomes; often, they involve strategic bets aimed at future growth through resource reallocation.

Cultural Shifts in Experimentation

• Emphasizing a culture of rapid experimentation over waiting for statistically significant results can be challenging due to existing performance metrics tied to impact.

Incentives for Data Scientists

• There is ongoing research into how reward mechanisms can shift company culture towards valuing learning alongside measurable impact, particularly within data science teams.

Broadening the Scope of Measurement

• A cultural shift is needed where data scientists are encouraged to engage with broader business strategies rather than focusing solely on narrowly defined statistical results.

Hypothesis-driven Testing

• Experiments should not only identify winners and losers but also explore insights about customer preferences and demand elasticity, enhancing understanding of business dynamics.

Incorporating Past Learnings into Future Experiments

• Establishing norms that prioritize learning from experiments can lead to more informed decision-making processes within organizations.

Bayesian Approaches in A/B Testing

• Utilizing Bayesian methods allows companies to integrate past experiment learnings into current analyses, fostering a richer understanding of user behavior and improving future testing frameworks.

Positive Externalities from Learning Integration

• By encoding insights from failed experiments into future analyses, businesses can create a network effect that enhances overall strategy development and execution.

Learning Comes at a Cost

The Value of Learning and Experimentation

• Professors often deal with incomplete knowledge, emphasizing that learning is not free; there are costs associated with it.

• A real estate platform's marketing manager used a holdout group in experiments without authorization, leading to significant financial loss but also revealing the team's value.

• The concept of a "holdout group" in experimentation highlights the importance of allocating samples to both treatment and control groups for valid results.

• Reflecting on past decisions can lead to questioning why resources were wasted on control when treatment was later proven better, akin to regretting over-ordering food after a meal.

• The cultural perception of winners and losers in business creates a disconnect regarding the value of learning from failures, which should be seen as an investment rather than waste.

Cultural Implications of Learning Costs

• Language around success and failure influences how businesses perceive experimentation; testing non-successful ideas is often viewed negatively.

• Many individuals in business lack backgrounds in data science or experimentation, making it difficult for them to understand that learning incurs costs.

• The anecdote about the real estate platform serves as a clear example of lost opportunities due to inadequate experimental practices.

Designing Effective Rating Systems

• When designing rating systems for marketplaces, it's crucial to consider existing challenges like rating inflation that have persisted since platforms like eBay and Amazon emerged.

Understanding Rating Systems in Marketplaces

The Importance of Ratings and Norming

• The speaker discusses the social dynamics of rating systems, noting that most people prefer to leave positive ratings to avoid being perceived as mean.

• As marketplace ratings increase, the perception of what constitutes a good rating shifts; for example, a four-star rating may feel inadequate compared to earlier standards.

• Renorming is introduced as a concept where ratings could reflect experiences relative to past high-rated stays, enhancing the context of feedback.

• This comparative approach allows users to express nuanced opinions without feeling pressured to give lower scores based on expectations.

Distributional Consequences of Averaging Ratings

• The speaker warns about the implications of averaging ratings in marketplaces, which can disproportionately affect new entrants versus established players.

• Established entities (e.g., restaurants with thousands of reviews) are less impacted by new ratings, while newcomers face significant risks from negative initial reviews.

• Research indicates that a single negative review can lead to an 8% drop in expected revenue for new sellers on platforms like eBay.

• Some platforms mitigate this risk by withholding visibility of ratings until users have accumulated several reviews.

Addressing Fairness in Rating Systems

• A proposed solution involves using prior beliefs when calculating averages; this method helps cushion new entrants against immediate negative impacts from their first reviews.

• The speaker emphasizes the need for distributional fairness in designing rating systems and notes that this area remains underexplored despite its significance.

Insights from Airbnb's Review System

• The speaker shares experiences from leading review system flows at Airbnb, highlighting the implementation of double-blind reviews aimed at increasing honesty and accuracy in feedback.

• This approach led not only to more honest reviews but also significantly increased overall review rates due to user engagement incentives.

Uncovering Hidden Information in Ratings

• The concept known as "the sound of silence" reveals that unleft ratings carry valuable information about seller performance and user satisfaction levels.

• Effective percent positive metrics incorporate both left and unleft ratings, providing deeper insights into seller reliability than traditional methods.

The Role of AI in Data Science

The Misconception of Automation in Data Science

• The speaker discusses the common belief that AI and language models (LMs) will automate significant portions of data science, suggesting this perspective may be misguided.

• While tools have made coding and visualization easier, the speaker emphasizes that AI has expanded the range of hypotheses and ideas available for exploration in data science.

• The increased capabilities provided by AI necessitate greater human involvement to filter through numerous hypotheses and focus on what truly matters in analysis.

• As experimentation scales up (e.g., testing 1,000 creatives instead of 10), new challenges arise regarding evaluation criteria and identifying successful outcomes.

• Contrary to expectations, humans are becoming more critical in the data science process rather than less, highlighting a need for ongoing human interaction with AI tools.

Recommended Reading for Data Enthusiasts

• A highly recommended book is "How to Lie with Statistics" by Darrell Huff, which offers an engaging introduction to understanding data manipulation.

• David Freedman's writings are praised for emphasizing practical engagement with data ("shoe leather statistics") and understanding processes behind data generation.

• Freedman’s insights encourage data scientists to examine real-world examples before applying analytical techniques, enhancing their understanding of context.

• "Four Thousand Weeks" by Oliver Burkeman is suggested as a thought-provoking read about time management and prioritization amidst life's endless tasks.

Insights on Media Consumption

• The speaker shares a favorite movie recommendation: "The Alpinist," which explores psychological aspects of climbing while reflecting on risk-taking behavior.

• On television, they mention enjoying "Only Murders in the Building," although they refrain from spoilers due to being behind on episodes.

Interviewing Techniques

Exploring Impact and Vision in Conversations

The Importance of Envisioning Outcomes

• The speaker emphasizes the value of asking individuals about the potential impact of their plans, encouraging them to think beyond immediate challenges.

• This question often reveals expanded visions and additional spheres affected by their actions, highlighting a deeper understanding of their goals.

• Startup founders are noted to be particularly adept at this reflective thinking compared to others.

Perspectives on Technology and Personal Interests

• The speaker shares a personal affinity for e-bikes, describing them as transformative for busy cyclists who still want an enjoyable experience.

• A humorous anecdote about a portable outdoor pizza oven illustrates how personal interests can lead to unexpected lifestyle changes, especially when influenced by family.

Advice on Slowing Down for Deeper Understanding

• The speaker advises students and professionals alike to "slow down" in order to develop meaningful mental models that inform decision-making processes.

• Emphasizing the importance of understanding marketplace dynamics, he argues that rapid-paced environments often sacrifice depth in thinking.

• This advice resonates with similar sentiments shared by other guests regarding the balance between speed and thoroughness.

Insights into Academic Culture at Stanford

• The speaker reflects on his experiences at Stanford University, noting a lack of emphasis on credentialing during conversations, which fosters open dialogue.

• He highlights how this cultural dynamic encourages collaboration across disciplines without preconceptions based on credentials or backgrounds.

Exploring Stanford's Unique Culture

The Environment at Stanford

• The culture at Stanford encourages a non-credentialed environment, fostering creativity and collaboration among individuals.

• This unique aspect of Stanford is often overlooked but contributes significantly to the enjoyment and vibrancy of campus life.

• The campus itself is described as "dreamy" and "joyful," enhancing the overall experience for students and visitors.

Engaging with Ramesh: Marketplace Founders Discussion

Insights on Marketplace Founders

• Ramesh reflects on whether the discussions have inspired new marketplace founders or clarified that some may not fit that role.

• He emphasizes the importance of understanding one's position in the entrepreneurial landscape.

Connecting Online

• Ramesh suggests LinkedIn as an easy way for interested individuals to connect with him, especially those from an industrial background.

• He also mentions his academic webpage at Stanford as another point of contact.

The Importance of Data Literacy

Key Takeaways on Data Literacy

• Ramesh stresses that promoting data literacy is crucial for effective interaction with AI tools and each other.

• He warns against the potential pitfalls of excessive prose generated by AI, which can lead to confusion rather than clarity.

Personal Commitment

• His work—both teaching and research—is deeply connected to enhancing data literacy among individuals interacting with technology.

Closing Thoughts and Future Engagement

Final Reflections

• Ramesh expresses gratitude for being part of the discussion, indicating a positive impact on listeners' understanding of data literacy.

Call to Action