Martech: Martech is Marketing Logo
  • Topics
    Transformation
    Operations
    Data
    Experience
    Performance
    Management
    Special Reports
    All Topics
  • Conference
  • Webinars
  • Intelligence Reports
  • White Papers
  • What is MarTech
    Mission
    Team
    Newsletter
    Search Engine Land
    Third Door Media

Processing...Please wait.

MarTech » Marketing Operations » Is A/B testing dead?

Is A/B testing dead?

A/B testing is still valuable, but marketers should take advantage of its next evolution.

Corey Patterson on March 9, 2022 at 11:13 am

“Today, A/B testing is thriving — it’s been a huge improvement from non-A/B testing,” said George Khachatryan, CEO of AI company OfferFit, in a recent webinar. “At the same time, the people performing these tests every day recognize that’s it’s a lot more difficult than it may seem.”

Designing A/B tests, determining samples sizes, and deploying them takes up a lot of time and resources, and analyzing the findings requires high levels of precision. All in all, the manual tasks required by A/B testing can place a heavy burden on marketers.

“When you’re running a full experimentation program, it’s never enough to run one A/B test,” Khachatryan said. “When you run one, you gain valuable insights, and inevitably want to gain more. So you end up running more.”

He added, “Those who are doing this hands-on realize it just becomes a rapid explosion in the number of tests they need – it becomes infeasible very early on this exponential curve.”

graph showing how A/B testing fails to scale
Source: George Khachatryan

Marketers need a solution that allows them to test a growing number of campaign variables while simultaneously giving them enough time to analyze the data. Fortunately, A/B testing is evolving.

Expanding the power of A/B testing

In the webinar, Khachatryan highlighted the “multi-armed bandit problem” that’s affecting modern A/B testing. In the traditional version of this scenario, a person at a casino must determine which slot machines (the “one-armed bandits” that steal your money) are going to have the best payouts, then figure out which order will be optimal. With A/B testing, the variables are the multi-armed bandits, and the marketer must discover which are most effective so they can allocate more resources to the areas performing well.

“You can think of [a multi-armed bandit] like a smart A/B test,” he said. “It’ll navigate the exploration-exploitation tradeoff — it’ll start randomly pulling those ten handles, but as it goes, it’ll dynamically reallocate resources so that if something looks bad it’ll stop pulling.”

mult-armed bandits vs contextual bandits in A/B testing
Source: George Khachatryan

He added, “These multi-arm bandits are designed to experiment just the right amount so you’re learning but also taking advantage of what you’ve already learned.”

While these multi-armed, or A/B, models have served marketers well over the years, there’s a new iteration of the framework that is more accurate and effective. According to Khachatryan, these are “contextual bandits.”

“It does what a multi-armed bandit does, but it takes into account different contexts,” he said. “So, if you have two different customers, with different characteristics, it’ll know to pull different levers.”

Contextual bandit frameworks are essentially automated experimentation and personalization at scale. It’s a model that can completely automate the process, and it’s what every marketer should be moving toward to improve campaign effectiveness at scale.


Get the daily newsletter digital marketers rely on.

Processing...Please wait.

See terms.


Automated experimentation is the future

Many of the tech giants have already adopted contextual bandit frameworks, but marketers should note that this technology is still incredibly new. Brands should allocate enough time and resources to make the transition process easier, because, according to Khachatryan, it’s the “future of experimentation.”

“In the past, manual A/B testing works with one mention at a time,” he said. “With these contextual bandits, you can set it up to simlutantous test multiple dimensions.”

Whether it’s testing email subject line efficacy, call-to-action click-through rates, or optimal article posting times, marketers have a lot of experimental data to keep up with. Automated testing solutions can make these processes more manageable by decreasing the time spent on manual tasks, replacing them with continuous automated experimentation.

automated campaign testing
Source: George Khachatryan

“You can think of this as the next iteration of experimentation, or test-and-learn programs,” Khachatryan said. “When a marketer sets up this system of continuous automated experimentation, it creates this interplay where you can see what happens, gain insights, and then use those insights to get new ideas.”

He added, “So you still have the agile test-and-learn cycle, but it’s accelerated.”

Time will tell how quickly marketers adopt these automated experimentation technologies. But, with the high level of marketing technology replacements that took place over the past year, there’s a good chance more brands will sign on sooner rather than later.

Watch this webinar presentation at Digital Marketing Depot.


Marketing automation: A snapshot

What they are. For today’s marketers, automation platforms are often the center of the marketing stack. They aren’t shiny new technologies, but rather dependable stalwarts that marketers can rely upon to help them stand out in a crowded inbox and on the web amidst a deluge of content.

How they’ve changed. To help marketers win the attention battle, marketing automation vendors have expanded from dependence on static email campaigns to offering dynamic content deployment for email, landing pages, mobile and social. They’ve also incorporated features that rely on machine learning and artificial intelligence for functions such as lead scoring, in addition to investing in the user interface and scalability.

Why we care. The growing popularity of account-based marketing has also been a force influencing vendors’ roadmaps, as marketers seek to serve the buying group in a holistic manner — speaking to all of its members and their different priorities. And, ideally, these tools let marketers send buyer information through their tight integrations with CRMs, giving the sales team a leg up when it comes to closing the deal.

Read next: What is marketing automation?


New on MarTech

    Unveiling our first MarTech Intelligence Report on email marketing platforms

    How product analytics can unite marketing and product teams to boost customer lifetime value

    Create a B2B GTM strategy that buyers, execs and revenue teams love

    Martech failure? 50% say loyalty programs don’t offer much value

    5 tips for building customer trust during the supply chain crisis

About The Author

Corey Patterson
Corey Patterson is an Editor for MarTech and Search Engine Land. With a background in SEO, content marketing, and journalism, he covers SEO and PPC to help marketers improve their campaigns.

Related Topics

Marketing Operations

Get the daily newsletter digital marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

June 7, 2022: Master Classes

September 28-29, 2022: Fall

Start Discovering Now: Spring

Learn More About Our MarTech Events

June 14-15, 2022: SMX Advanced (virtual)

November 14-15, 2022: SMX Next (virtual)

March 8-9, 2022: Master Classes (virtual)

Learn More About Our SMX Events

Webinars

Take a Crawl, Walk, Run Approach to Multi-Channel ABM

Content Comes First: Transform Your Operations With DAM

Dominate Your Competition with Google Auction Insights and Search Intelligence

See More Webinars

Intelligence Reports

Enterprise SEO Platforms: A Marketer’s Guide

Enterprise Identity Resolution Platforms

Email Marketing Platforms: A Marketer’s Guide

Enterprise Sales Enablement Platforms: A Marketer’s Guide

Enterprise Digital Experience Platforms: A Marketer’s Guide

Enterprise Call Analytics Platforms: A Marketer’s Guide

See More Intelligence Reports

White Papers

Reputation Management For Healthcare Organizations

Unlock the App Marketing Potential of QR Codes

Realising the power of virtual events for demand generation

The Progressive Marketer’s Ultimate Events Strategy 2022 Worksheet

CMO Guide: How to Plan Smart and Pivot Fast

See More Whitepapers

Receive daily marketing news & analysis.

Processing...Please wait.

Topics

  • Transformation
  • Operations
  • Data
  • Experience
  • Performance
  • Management
  • All Topics
  • Home

Our Events

  • MarTech
  • Search Marketing Expo - SMX

About

  • What is MarTech
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS

© 2022 Third Door Media, Inc. All rights reserved.