Every year, around February 3rd, newspapers and websites line up to play the post-Super Bowl favorite reporters pastime: Analyzing Super Bowl commercials. Some scrutinize their humor, others focus on the sexy factor, while many others come up with more or less passable definition for the “best” and “worst” ads of the year. Many of them, however subjective and unscientific their scoring system, often rely on and then quote either one of the three major measurement metrics: Ace Metrix’s Ace Score of Super Bowl ad effectiveness, USA Today’s Ad Meter Survey, and Nielsen’s survey of the most memorable and likable ads. The three measurements differ in their measurement strategy, but more often than not, reach pretty much the same conclusions.

How Do They Do It?

Ace Metrix conducts a viewer reaction survey of randomly selected people across the US, representative of the US TV viewing audience, and asks them to review the ads on a scale of 1-950 while keeping in mind the following attributes: Persuasion, relevance, information, attention, change, desire and watchability.

USA Today Ad Meter relies on a live poll during the telecast. The survey, which started in 1989, uses a 0-10 scale of focus groups based in Virginia, USA Today’s HQ and one or more sites around the country. It measures only the national ads shown during the game, excluding those shown at halftime. In 2013, Facebook users and people logging onto the USA Today’s website could also participate in the survey, and their results were added to the regular meter.

Nielsen, one of the largest information and measurement companies in the world, gauges two parameters in its scientifically generated survey: general recall score (percent of viewers who remember seeing a specific ad versus the mean general score of all ads) and likability (percent of viewers who like an ad versus the mean general likeability score of all ads). Nielsen then published two “Top 10 lists”, one for the most memorable ads and the other for the most likable ads. In contrast to USA Today’s Ad Meter, Nielsen also measures ads during halftime.

Does Likability Help Sales?

It does not often happen that an ad that appears on Ace Metrix most effective list is absent from the other two, and vice versa. But the question remains: whether an ad is likable, memorable, relevant or attention-grabbing – do Super Bowl ads really help brands sell products?

According to the most recent and comprehensive study on Super Bowl related brand recall (which qualitatively measures how well a brand name is connected with a product type or a class of products) the answer to that question is: Not that much.

Communicus, a US research based advertising consultancy, conducted a study prior to Super Bowl 2014 that found that about 60% of ads do not increase purchase or purchase intent.

The firm interviewed more than 1,000 consumers, before and after they were exposed to the ads in the 2012 and 2013 Super Bowls. According to CEO Jeri Smith, who spoke to Ad Age, it takes its first read on a typical ad four weeks after it starts running to allow for multiple exposures. And herein lies the Super Bowl problem: many Super Bowl ads don’t air regularly after the game, so one ad exposure simply does start the ball rolling.

Communicus also found that brand association with Super Bowl commercials is much lower than with a typical buy, oftentimes because of the way the creative is structured. In other words, brands focus so much on being creative instead of the brand itself, that people who remember seeing a Super Bowl ad recall the brand only 35% of the time, versus 50% of the time for other ads of the same brand.

Take the following two ads for example: Budweiser hit pay dirt in 2013 with its Brotherhood ad. Ace Metrix respondents gave it a 665 score out of 900. USA Today Ad Meter’s respondents agreed, and the Budweiser ad got 1st place there as well. According to Nielsen, the ad was fourth in likability that year. Communicus found that the ad fared also well in terms of increasing purchase intent. The reason: the iconic horse used in the ad was well linked to the brand.

In contrast, Tide’s Miracle Stain ad, which scored pretty high with Nielsen’s (coming 7th in terms of likability, 5th in terms of memorability) did not increase purchases or intent to purchase, as “it didn’t tell people anything they didn’t already know”, according to Smith, or carried any emotional value. The 60-second ad, Smith says, made people so caught up in the story that they forgot the brand.

According to a series of studies conducted by Harvard Business School professor Thales Teixeira, Super Bowl ads often run the risk of being too entertaining. The research team worked with 275 randomly selected people, showing them a sample of 82 Super Bowl ads for 30 brands. Researchers relied on cameras that track visible changes on a person’s face as they watch an ad, thereby determining interest in viewing the ad and their intent to purchase the brand. Results determined that the level of entertainment increases a viewer’s interest in watching an ad; however, after a certain amount, that persuasiveness starts to decrease and advertisers lose the link to the brand and the power to influence buying. In other words, too much entertainment is a big no-no, as it can actually reduce peoples’ desire to buy the product, Teixeira told Forbes in an interview.

Another survey conducted prior to Super Bowl 2014, this time by NRF (National Retail Foundation) also found that entertainment value seldom coincides with intent to purchase. According to the survey, almost 25% tuned in to the game just to watch the commercials, but a vast majority of those people viewed commercials only as entertainment. Although 16.9% admitted that the ads made them aware of brands, only 8.9% of overall respondents said Super Bowl ads influenced them to buy the product. A striking majority of respondents (over 80%) said they view Super Bowls as nothing but good fun.