Martech: Martech is Marketing Logo
  • Topics
    • Customer & Digital Experience
    • Digital Transformation
    • Data
    • Marketing Management
    • Marketing Operations
    • Performance Marketing
    • Special Reports
    • All Topics
  • Conference
  • Webinars
  • Intelligence Reports
  • White Papers
  • What is MarTech

Processing...Please wait.

MarTech » Customer & Digital Experience » Achieve big testing wins by analyzing results by device type

Achieve big testing wins by analyzing results by device type

Responsive design sites utilize the same HTML for users, regardless of device -- but that doesn't mean that all users are getting the same experience. Columnist Abraham Nord shares a case study in which segmenting results by device type provided insights that led to a significant conversion boost.

Abraham Nord on September 8, 2017 at 1:13 pm | Reading time: 3 minutes

For some time now, responsive design has been all the rage — and this trend shows no sign of letting up as the dominant configuration for mobile websites.

Responsive design allows us to present a consistent site experience to users regardless of screen size. Because the HTML is the same for all devices, it has the added benefit of also reducing development and maintenance time. However, different experiences based on screen size or device type can pose challenges when running tests and analyzing results.

Testing challenges

For our discussion, we will consider the three most common device types that visitors use when accessing a website:

  1. Mobile
  2. Tablet
  3. Desktop

Even with responsive design, the user experience for a website can be radically different for each of these devices. Running a test without considering and analyzing results specific to each device type can lead to erroneous conclusions, resulting in poor decisions that could have a potentially disastrous impact on conversion rates.

Understand test results by device type

Many testing tools allow us to segment results by device type in order to get a more granular picture of performance on our website. However, depending on how your responsive site is coded, this may not be the best indicator of the visitor’s experience.

For example, our landing pages are coded responsively based on the width of the visitor’s browser. So, if a visitor has their browser open on only half of their desktop screen, they would actually be experiencing the “tablet” version of our page, even though their device type is still “desktop.”

Understanding how pages behave and how devices are labeled is critical for accurate analysis of your testing data.

Test example

Let’s finish up by looking at a real-world example to show the dramatic difference this can make on the performance of a website or landing page.

We recently ran an A/B/C test of a new look and feel for a responsive landing page. After about six weeks, both test variations were outperforming the control (by 51.79 percent and 51.23 percent, respectively) for all traffic. We could say with statistical confidence that both test variations were an improvement over the control; however, there was no statistically significant difference between test variations to help us determine which one to use.

We could have simply run the test longer until statistical confidence was achieved between test variations, but before doing so, we decided to look at the results by browser screen width (described as mobile, tablet and desktop below). What we found was startling.

  • Test Variation 1 actually had a 41 percent decrease in conversions for mobile visitors, but a 148.03 percent increase in conversions for desktop visitors.
  • Test Variation 2 had a 42 percent increase in conversions for mobile visitors and a 62.64 percent increase in conversions for desktop visitors.
  • Tablet data for both variations was too small for statistically relevant results.

Our conclusion

Based on this data, we concluded the test with a hybrid winning page, combining the desktop experience of Variation 1 and the mobile experience of Variation 2.

This hybrid approach resulted in an overall 86.40 percent increase in conversions instead of the ~51 percent increase that we were looking at previously. The hybrid solution also allowed us to conclude the test sooner than would have otherwise been possible.

I recommend that testers analyze results by device type or screen width whenever possible and implement hybrid solutions utilizing the best results for each experience. The results may surprise you!


Opinions expressed in this article are those of the guest author and not necessarily MarTech. Staff authors are listed here.


New on MarTech

    TransUnion partners with Canvas Worldwide to boost omnichannel CX
    A new way to find the tech talent you need 
    The only two things that matter in marketing
    How marketers are preparing for the future of in-game ads
    How CDPs transform donor experience for a nonprofit organization

About The Author

Abraham Nord
As head of the Conversion Improvement department at SmartSearch Marketing, Abraham leads landing page testing strategy, implementation and analysis along with client communication and delivery. Abraham brings 8+ years of experience in conversion improvement, landing page testing and usability with a specific focus towards B2B and business with a complex sale. Prior to joining SmartSearch Marketing, Abraham successfully ran his own ecommerce site and holds a bachelor of fine arts in painting along with a degree in web design and interactive media. Abraham currently resides in Denver, CO and enjoys spending time with his family, camping, hiking, playing tennis and online gaming.

Related Topics

Customer & Digital ExperienceDataPerformance Marketing

Get the daily newsletter digital marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS The MarTech Conference logo.

September 28-29, 2022: Fall

Start Training Now: Master Classes

Start Discovering Now: Spring



The SMX Conference logo.

Start Training Now:: SMX Advanced

November 14-15, 2022: SMX Next

March 8-9, 2022: Master Classes

Webinars

Tracking Growth From Organic Search

Beyond the Buzzword: Transform Digitally to Drive Organic & SEO Growth

Leap or Linger: Determining Which Ad Platforms to Test for Your B2B Brand

See More Webinars
Intelligence Reports

Enterprise Marketing Performance Management Platforms: A Marketer’s Guide

Enterprise Customer Journey Orchestration Platforms: A Marketer’s Guide

Enterprise Account-Based Marketing Platforms: A Marketer’s Guide

See More Intelligence Reports
Featured White Paper

The CMO’s Formula To 3x Your Digital Marketing Campaign Results

See More Whitepapers
Search Our Site

Receive daily marketing news & analysis.

Processing...Please wait.

Topics

  • Transformation
  • Operations
  • Data
  • Experience
  • Performance
  • Management
  • All Topics
  • Home

Our Events

  • MarTech
  • Search Marketing Expo - SMX

About

  • What is MarTech
  • Contact
  • Privacy
  • Terms Of Use
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS

© 2022 Third Door Media, Inc. All rights reserved.