Email A/B Testing Game: Guess the Winner

Welcome to Email Einstein Podcast

Introduction and Episode Overview

• Hosts Vera Sadlock and Andre Bchuk introduce the episode, emphasizing a light-hearted discussion during a busy time of year.

• The focus will be on A/B testing, specifically testing Andre's marketing intuition to predict winning variations for clients.

Marketing Intuition vs. Data

• Andre expresses initial confidence but admits to feeling apprehensive about relying solely on intuition.

• He emphasizes that data should guide decisions rather than personal preferences, highlighting the importance of understanding audience behavior in marketing strategies.

Importance of A/B Testing

• Vera shares her humbling experience with A/B tests, noting that personal preferences can mislead results.

• They discuss how A/B testing is crucial for aligning marketing efforts with audience preferences.

A/B Test Case Study: Apparel Brand

First Test: Discount Code in Hero Image

• The first test involves comparing two hero images: one with discount code information and one without.

• The primary goal is to improve click-through rates (CTR), with revenue as a secondary benefit.

Andre's Prediction

• Andre predicts that the image without revealing the discount will have a higher CTR due to curiosity factors.

Results Discussion

• The results confirm Andre's prediction; the version without the discount outperformed in both CTR and placed order rates.

Key Takeaway from First Test

• Understanding your audience is essential; results may vary significantly across different accounts, reinforcing that there’s no universal solution in marketing strategies.

Second Test: Opt-in Forms Comparison

Second Test Setup

• This test compares two opt-in forms: one with "Get 50% off" CTA and another with "Claim 50% off" plus an additional "Decline offer" CTA.

Andre's Thoughts on CTAs

• He favors the second option ("Decline offer") because it plays into psychological triggers by creating a sense of counterintuitive choice.

Results Analysis

• Although the difference was not statistically significant, there was noticeable evidence favoring option B regarding revenue and form submissions.

Psychological Insights

Understanding the Impact of Form Design on User Engagement

The Power of Choice in Forms

• Discusses how offering options (yes or no) can create a sense of control for users, enhancing engagement through reverse psychology.

• Highlights that forms with clear choices tend to perform better than those lacking them, as they give the illusion of user agency.

• Suggests experimenting with different form designs to see which generates more interaction and engagement.

Testing Call-to-Action (CTA) Effectiveness

• Introduces two similar forms: one where a person points to a CTA and another without a clear CTA.

• Reveals that the form pointing to the survey link outperformed the other in terms of clickthrough rates, despite not being statistically significant.

• Notes that while clickthrough rates were similar, revenue generated by the second form was significantly higher.

Analyzing Revenue vs. Clickthrough Rates

• Engages in a debate about whether revenue is a more important metric than clickthrough rates when evaluating form performance.

• Mentions that small design tweaks can lead to improved overall performance metrics for forms.

Long vs. Short Campaign Strategies

• Initiates discussion on testing long versus short email campaigns, particularly focusing on urgency and discount offers.

• Explores preferences between long and short formats based on product variety; longer formats may provide more options but could also distract from CTAs.

Results of Campaign Length Testing

• Shares insights that shorter campaigns often outperform longer ones due to fewer distractions and clearer calls-to-action.

Email Marketing Campaign Insights and Strategies

Testing Long vs. Short Email Campaigns

• The discussion begins with the idea that showcasing collections can reduce curiosity about products, leading to a more straightforward marketing approach.

• A wellness brand campaign tested shorter versus longer email formats, focusing on educating customers through relevant information.

• Variation A was the longer email containing detailed ingredient benefits, while Variation B was shorter with minimal product descriptions.

• Despite initial expectations for the shorter email to outperform, it was revealed that the longer version actually won in terms of engagement.

• Over multiple campaigns, longer emails consistently resulted in higher placed order rates due to their educational content.

Key Findings from Campaign Results

• The clickthrough rate difference between long and short emails was minimal (0.7% vs. 0.72%), but placed order rates were significantly better for longer emails.

• The brand's commitment to community connection played a crucial role in customer engagement and revenue generation through educational content.

• For promotional campaigns like Black Friday or Cyber Monday, simpler emails focused on discount codes and product blocks were effective.

Recommendations for E-commerce Owners

• It is advised for e-commerce businesses to document key findings from split tests to avoid redundant testing and build upon successful strategies.

• Keeping track of insights helps streamline future campaigns, especially during high-stakes periods like holiday sales.

Internal Innovations in Marketing Strategy

• An internal tool resembling ChatGPT is being developed to help teams recall past test results and theories efficiently.

• This AI tool aims to provide insights based on historical data, preventing repetitive mistakes in marketing strategies.

Conclusion: Importance of Learning from Past Campaigns

Discussion on Email Design Preferences

Light vs. Dark Mode in Email Design

• The conversation begins with a debate over the effectiveness of light versus dark background designs for emails, focusing on brand colors and click-through rates.

• One participant initially believed that dark mode would outperform light mode due to its popularity among users, particularly iPhone users.

• Contrary to expectations, white background emails performed significantly better in two out of three tasks tested, raising questions about user preferences.

• A hypothesis suggests that the white background may stand out more against a sea of black emails, making it visually appealing and easier to read.

• The discussion highlights an interesting finding regarding user engagement with different email designs.

Testing Imagery Types for Automotive Brand

Lifestyle vs. Product Imagery

• The focus shifts to testing lifestyle imagery versus product imagery for an automotive brand, with initial beliefs leaning towards lifestyle images being more effective.

• The product is described as somewhat dry when presented alone; however, lifestyle images could enhance the perception of adventure associated with the product.

• Despite expectations for lifestyle images to perform better, recent tests showed that product images consistently outperformed lifestyle ones in terms of placed order rates.

• Data indicates only slight differences in click-through rates between image types but significant advantages for product images regarding actual orders placed.

• Personal preferences lean towards lifestyle imagery; however, insights suggest that target customers are likely experts who prefer detailed views of products.

Understanding Customer Engagement Through Education

Target Audience Insights

• There’s surprise at the results favoring product images since they appeal less aesthetically but resonate more with knowledgeable customers seeking education about specific products.

• The brand excels at educating its audience about various shocks and their applications across different vehicles, fostering customer loyalty through informative content.

Testing Strategies for Email Marketing

Enhancing Product Presentation in Emails

• Discussion on incorporating product images directly onto lifestyle images to enhance visual appeal and engagement.

• Successful tests led to updates in the majority of email images and flows, indicating a strong correlation between testing and improved performance.

Understanding User Interaction

• Recommendation to utilize heat maps to analyze user clicks on call-to-action buttons within emails, providing insights into user behavior.

• Noted that revenue tends to be higher for product-focused emails compared to lifestyle ones, suggesting that showcasing expertise can resonate better with customers.

Practical Tools for Analysis

• Suggested searching for "email heat maps" on YouTube for tutorials on how to implement tracking codes that reveal where users click in emails.

Testing Opt-in Forms: A Case Study

Comparing Visual Approaches

• Introduction of two opt-in forms: one featuring a high-fashion model and another with the brand's founder, aiming to improve click-through rates.

• The contrast between a polished high-fashion image versus a warmer, more personal image raises questions about audience connection.

Audience Connection Insights

• Personal opinion shared that audiences may prefer familiar faces over stock images, as they feel more connected.

• Discussion about the relatability of images; the second version featured Megan Trainor, which contributed significantly to its success.

Results from Testing Variations

Performance Metrics

• The form submission rate showed significant differences: 1.37% for the first option versus 2.2% for the second option featuring Megan Trainor.

Broader Advertising Trends

• Debate over whether polished or relatable imagery works better in advertising; examples from brands like Nike highlight a shift towards more authentic representations.

Key Takeaways from Testing Experiences

Importance of Comprehensive Testing

Testing Strategies and Insights

Importance of Testing in Performance

• The speaker emphasizes that assumptions about performance can be misleading; what seems like a better option may not always yield better results. This highlights the necessity of testing to validate hypotheses.

Engaging with the Audience

• A call to action is made for listeners to share their split test experiences, suggesting an interactive approach where the hosts could analyze audience submissions.

Methodology of Split Testing

• The speaker mentions a recorded video detailing their internal process for conducting split tests, which includes forming a hypothesis, collecting data, drawing conclusions, and planning next steps.

Resource Sharing

• There is an intention to include links to resources mentioned in the podcast description, ensuring listeners have access to all relevant materials discussed during the episode.

Timing for A/B Testing