I was a psych major. For those of you who wondered what us “soft” science Psych majors studied in college– I was basically trained on how to call BS on any claim, statement, or inference of causality. Why? Because measurement in Psychology is not like measuring mass or temperature. It’s an art. It’s complex. And 9 times out of 10, your causal conclusions are flawed by some confounding factor you haven’t thought of. This is the basis of my obsession with Digital Media, and my long-time frustration with the oversimplification of Performance Analysis of Display media campaigns.
Because I have nothing better to do than read 30 page papers about Display and Search Media Attribution, I wanted to share with you all a detailed criticism of the Harvard Business School “Do Display Ads Influence Search? Attribution and Dynamics in Online Advertising” working paper that was published a couple of months ago. My hope is that some day, my fellow Digital Media professionals will one day accept the fact that campaign results and analysis for display campaigns will never be appropriately understood or accurately summarized in a one sentence executive summary. As long as we continue to condense the complexity of campaign analysis into quick sound bites and talking points, none of us will ever actually understand the full picture of the media mix.
First, the overall message I take away from the paper is clear and valid, “managers should carefully consider the interaction and dynamic effects of search and display advertising”. You should know this already. And if this is the first time you’ve heard this, the paper is a great place to get a deep dive into the justifications for this statement. It’s an important message that is often forgotten in the madness of the corporate game and I’m glad they’ve called this out.
Now, on to my pain points…
A huge problem with the generalization of their conclusions lies in their data. It’s all in the dirty details at the bottom of page 7. Their whole data set relied on “Using Internet Cookies”. I feel that a deeper understanding of the ad serving ecosystem begs the question if this is a valid data collection methodology for the kind of advanced statistical analysis they are performing. The paper makes several assumptions by making this statement:
- Cookies are reliable – This is not nor has it ever been true. Reporting what the researchers would call “standard metrics” using cookie methodology is fairly reliable and great for directional insights, general campaign auto-optimizations, and a top level view of success. But they are not a reliable data source for the type of quantitative statistical analysis that is being performed in this paper. Doubleclick For Publishers (DFP) clearly states “it is common to see campaign variances of up to 20%”. I think it’s conceivable that people less likely to be cookie’d could correlate with a variable that dramatically effects the campaign results. For example, people who access desktop sites through non-home computers are less likely to have a cookie track activity past an impression, and might be more likely to be served an impression that is not tracked. Could this group have a higher conversion rate for some confounding reason?
- The entire population only used one desktop computer during the campaign - This data does not account for users who, for example, were exposed to a display ad on one device, and completed an application through an organic or direct site visit through a second device or location. This is one of the many reasons understanding effects of display ads on brand awareness or purchase intent is crucial for measuring display ROI.
- All conversions took place on a desktop – Does this bank have a mobile app or site? Is that part of the measurement methodology? Users could have similarly navigated to the banking website on an iPhone after having been exposed to a Display Ad on a desktop. Also, the study never outlines whether mobile search or display placements were a part of the data set. These ad impressions are much less likely to track through a conversion from an impression. Since mobile smart phones now account for half all mobile phones, a large group of people have the capability of being exposed to a banner ad impression without marketers being able to attribute it to a conversion. The authors themselves mention that their study does “not consider spillover effects of search and display into [mobile and offline]“.
Aside from the reliability of cookies, the agency attributed any post impression conversion within a 30 day look-back window fully to Display (a common but misguided practice) except for Paid Search. But only because there was no technical solution in place to de-dupe attribution from Paid Search (a la Doubleclick Search). This is another questionable oversight in their data which, in my opinion, calls to question whether the authors’ directional insights on SEM vs. Display media investment in the empirical environment are valid in the real world. I believe the answer is no. There are simply too many user cases where a consumer can be influenced by a display ad impression before a paid search click without an ability to measure the overlap or the value of each channel’s influence.
The paper goes on to describe a series of statistical tests that I had to pull out my college stat text book to get through. I don’t find anything unsound in these methodologies, nor is evaluating the statistical methodologies something I’m attempting to do in this criticism. Rather, I’m bringing hands on knowledge of the ad delivery and reporting methodologies into the context of this study. In display and search measurement, there are many holes in the data and inaccuracies caused by technical malfunctions in the transmission of ads or cookies as well as human error in technology implementation. Without a robust exploration of the possible effects of these important variables, I don’t feel the statistical analysis provides results that should be used to make business decisions blindly from other sources of analysis like brand surveys, focus groups, or cross-channel analysis. However, I believe it provides extremely valuable insights in the way of questions that marketers should ask themselves when trying to understand the attribution model for their specific product or brand.