Most marketers say AI works. Few can prove it.

AI can make work faster while leaving teams unable to prove the value. The problem starts before launch, with weak baselines and success metrics.

Table of Contents

    Spy on Any Website

    Get traffic data and keyword intel on competitors instantly.

    Most marketers say AI is improving things, but few of them can prove it.

    According to a new Metric Sherpa study produced with Sprinklr, 92% of organizations with customer-facing AI in regular use said it met or exceeded expectations. Only 27% said they could show proof of that.

    Similarly, 67% of the people who said AI made work faster also said it reduced costs. However, most of those had no baseline to base it on: Only 32% of that group said they tracked costs before launch.

    Measurement needs to start before the AI does

    This is likely because most teams know what they want AI to improve, but few define the metric, record the starting point, or set a clear threshold for success. So when the results arrive, even if there’s a success story, there’s no way to prove it.

    The study found that while 99% could name an intended outcome for their AI use, only 26% defined success metrics, and just 10% also recorded a baseline.

    Without the last two, it’s impossible to prove AI’s ROI.

    If the goal is lower acquisition cost, the team needs a cost baseline before the AI starts making media or campaign decisions. If the goal is better customer service, containment alone won’t tell you much. The team needs data on resolution, repeat contacts, escalations, and handoffs.

    These measurements must be built into the workflow before launch.

    Don’t let the platform define success

    If they aren’t, teams are forced to rely on whatever numbers the platform gives them.

    A paid media platform can optimize conversion. A service platform can optimize containment. A content system can optimize engagement. All three can show improvement inside their own dashboards while the business outcome stays flat.

    Eighty-five percent of those surveyed reported improvement in at least one metric category they hadn’t tracked before launch. The researchers couldn’t tell whether those teams added measurement later or relied on another source, but the disconnect is hard to ignore.

    Marketing ops needs to connect platform activity to CRM, finance, customer, or experiment data. Otherwise, the tool doing the work also gets to decide whether the work succeeded.

    Someone still has to own the result

    Even with the right metrics in place, somebody has to be responsible for the outcome.

    Only 17% said their organization names an owner for AI business outcomes. It was the least common of the eight practices measured in the study.

    That role is different from owning the platform or the workflow. The platform owner makes sure the tool works. The workflow owner makes sure the process runs. The outcome owner answers for whether the AI produced the desired result.

    Organizations using a named outcome owner were able to quantify results 46% of the time, versus 23% of those without one. They also used controlled tests and financial reporting more often. The study doesn’t establish that ownership caused better measurement, but the association is substantial.

    Better measurement can make AI look worse

    Teams that measure more carefully may also find more things going wrong.

    Organizations using three or more of the study’s measurement practices reported more AI-to-employee handoff problems, adoption issues, and data-quality friction than those using zero or one. It isn’t clear if the increased measurement exposed more problems or if more complicated deployments had more problems to find. Either way, the operational lesson is useful.

    10X your SEO with Semrush for Enterprise.

    The world’s most powerful SEO platform, purpose-built for Enterprise.

    Request demo

    A mature AI program can look messier because somebody is finally checking what happens after the automation fires. That means reviews need to cover more than output and speed. Look at where handoffs break, where employees ignore recommendations, where customers repeat themselves, and where a metric moves in the wrong direction.

    Decide what would make you stop

    Only 28%  said their organization uses agreed criteria to decide whether to scale, change, or stop an AI application. Just 37% review AI results on a regular schedule.

    That leaves plenty of AI running without a clear rule for what happens if the direct metric improves but the business result doesn’t.

    For one application, define the outcome, baseline, success threshold, and comparison before the next review. Then write down what result would justify scaling it, what would require a change, and what would cause the team to stop.

    AI can make a martech stack much more capable. It can also make weak measurement much harder to unwind once the system is running.

    The full report can be downloaded here. (No registration required)


    MarTech is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.

    Constantine von Hoffman
    Senior Editor, MarTech

    Constantine von Hoffman is senior editor of MarTech. A veteran journalist, Con has covered business, finance, marketing and tech for CBSNews.com, Brandweek, CMO, and Inc. He has been city editor of the Boston Herald, news producer at NPR, and has written for Harvard Business Review, Boston Magazine, Sierra, and many other publications. He has also been a professional stand-up comedian, given talks at anime and gaming conventions on everything from My Neighbor Totoro to the history of dice and boardgames, and is author of the magical realist novel John Henry the Revelator. He lives in Boston with his wife, Jennifer, and either too many or too few dogs.

    View Author Profile