Super Bowl ad rankings: Trust your gut or employ rational thinking?

The Super Bowl is the single most viewed televised event in the US. Every year, more than 100 million people tune in to see who will be crowned NFL champion. There is no bigger or brighter stage for impactful ad content, so it is no wonder that a 30-second commercial this year sold for an average of $5 million. With the stakes that high, brands fight tooth and nail for audience attention and, like most consumers, they want to make sure they get their money’s worth.

Though the need for strong evaluative metrics is clear, the means of selection is less evident. Ad Meter, Real Eyes, YouTube, Ipsos (my employer) and many others all created rankings of winners — but each used a different system and got different results. This year, Amazon’s Alexa ad crowned USA Today’s Ad Meter rankings, while Tide, which did not even make the Ad Meter top 10, won out in the inaugural neuro study conducted by Ipsos.

When you’re right, you’re right

While your first instinct may be to assume that one of these studies got it wrong, the divergence in these ratings represents differences in testing approaches, each reflecting a distinct goal and research philosophy. Though methodologies vary, the most dramatic distinctions arise between studies that evaluate audiences’ conscious responses and those that measure nonconscious reactions.

Behavioral science research has revealed that people employ two kinds of information processing: System 1 and System 2. The former is automatic, rapid, efficient, and often operates below our conscious awareness. It can be thought of as our intuitive “gut” reactions and feelings. By contrast, System 2 is controlled, analytical, and deliberate. It is only active when we have the ability and the motivation to consciously process information.

System 1 and system 2 in the spin cycle

These different systems come into play at different points in consumers’ exposure to a brand’s narrative. In the early stage of a product’s lifecycle, the best communication strategy may be to inform and educate prospects as to the new product’s superiority to other existing solutions. Emotional appeals may not reach a skeptical consumer, but providing viewers with the information they need to make an informed judgment can help win them over.

After the early stages of a product’s lifecycle, when the shine of the new has fallen away and other products have caught up in terms of rational benefits, it’s the brand’s image, relative appeal, and closeness that drive consumer choice. Lists of product benefits and demonstrations of their relative strengths become burdensome to viewers, while emotionally-stirring or visually-exciting ads can captivate their attention and maintain their interest.

Picking the right tool for measuring effectiveness

Just as the stages of a product’s lifecycle should inform the choice between educational and emotive advertising, a brand’s needs dictate whether an understanding of an ad’s System 1 or System 2 impact is most relevant. When making this determination, it is important to consider the strengths and limitations of viewer self-reporting. While System 2 studies may yield more thoughtful responses, they can overlook nonconscious viewer engagement.

Like most research techniques, the Ad Meter evaluates participants’ articulated response to individual ads. It relies entirely on viewers’ conscious assessment of the ads with self-recruited panelists scoring each ad on a 1 to 10 scale. As marketers we are very familiar with these types of survey questions: “Which ad did you like best”, “Which ads did you dislike”, or “Which ads were most memorable?” These are all examples of System 2 processing.

Measure reactions with the blink of an eye

Methodologies that measure System 1 processing of ad content may be less familiar, but they can be powerful. Neuroscience has armed researchers with an arsenal of techniques, including EEG (electroencephalogram), Facial Coding, Eye Tracking, and Biometrics. The merit of each of these approaches is determined by the importance of recreating a natural viewing environment and which level of analysis most directly answers a brand’s questions.

• EEGs measure fluctuations in the brain’s electrical activity using a headset. They provide great temporal resolution by matching voltage spikes and waves to the introduction of stimuli and can be read at fast speeds, giving almost immediate results. However, EEGs struggle to simulate a natural viewing environment, as respondents are required to wear a headset and must be mindful of moving while participating in the study.
• Facial coding reads emotions from facial expressions — happiness, surprise, sadness, disgust, fear, and confusion. It can track minute changes in mood, revealing if a scene is having the desired effect. Capable of distinguishing between a sincere and an insincere smile, it can provide great insight about the true emotional state of viewers when watching ads, but participants need to sit in front of a camera and remain focused on the content.
• Eye tracking measures the movement of a viewer’s gaze across the screen and allows practitioners to see through the audience’s eyes. It is one of the best ways to determine what attracts and captures a viewer’s attention, though it does little to explain what (if anything) the viewer is experiencing. Eye tracking also requires participants to sit in front of a specialized camera, which, while less intrusive than an EEG, is still short of a natural viewing environment.
• Biometrics can be measured in many ways, including Galvanic Skin Response (GSR), also referred to as skin conductance or electro-dermal activity. When the viewer becomes excited, sweat production increases, making it possible to directly monitor the impact of stimuli on the viewer. Measuring GSR is far less disruptive than other System 1 tests, as it does not require cumbersome equipment or severely limits participants’ movement.

Tides of sweat make for winning ads

GSR was chosen for Ipsos’ study because it was deemed the best way to collect System 1 data while preserving the natural viewing environment so important for the Super Bowl. The human body has the highest number of sweat glands in the hands and feet, making them ideal sites for testing. In the interest of both maximizing accuracy and minimizing disruption, a wearable Shimmer GSR device — designed to be attached to participants’ fingers — was used.

Image via Shimmer

Participants enjoyed the Super Bowl in the comfort of a small theater, while the devices tracked their levels of excitement. When Alshon Jeffery reached for a one-handed catch, Eagles fans tensed in anticipation. Their heart rates spiked, and sweat production increased, as part of their bodies’ natural response. Every time David Harbour delivered his soon-familiar line–” It’s a Tide ad” –their amusement was nonconsciously expressed the same way.

It all comes back to the brand

The ads in this year’s Super Bowl were tested and ranked by no fewer than six companies using various System 1 and System 2 methodologies. Each was “right” for a specific context, delivering a unique perspective on the ads’ impact on viewers. Smart marketers and their agencies know to choose a methodology based on their specific objectives and are paying increasing attention to the needs of their communication to engage System 1 and System processing.