Commentary
Memo From CFO: Ad Metrics Not Good Enough
- by Pat LaPointe , Op-Ed Contributor, March 30, 2010
Following is a sanitized version of an actual email from a CFO to a CMO at a global 1000 company:
TO: Susan James - Chief Marketing Officer
FROM: Amy Ivers - Chief Financial Officer
RE: Congratulations on your recent recognition for marketing efficiency
Susan -
Congratulations on being ranked in Mediaweek's recent list of top 10 "most efficient media buyers." It is a reflection of your ongoing commitment to getting the most out of every expense dollar, and well-earned recognition.
But I can't help but wonder, what are we getting for all that efficiency?
Sure, we seem to be purchasing GRPs and click-throughs at a lower cost than most other companies, but what value is a GRP to us? How do we know that GRPs have any value at all for us, separate from what others are willing to pay for them? How much more/less would we sell if we purchased several hundred more/less GRPs?
And why can't we connect GRPs to click-thrus? Don't get me wrong, I love the direct relationship we can see of how click-throughs translate into sales dollars. And I understand that when we advertise broadly, our click-throughs increase. But what exactly is the relationship between these? Would our click-through rate double if we purchased twice as much advertising offline?
Also, I'm confused about online advertising and all the money we spend on both "online display" and "paid search." I realize that we are generally able to get exposure for less by spending online versus offline, but I really don't understand how much more and what value we get for that piece either.
In short, I think we need to look beyond these efficiency metrics and find a way to compare all these options on the basis of effectiveness. We need a way to reasonably relate our expenses to the actual impact they have on the business, not just on the reach and frequency we create among prospective customers. Until we can do this, I'm not comfortable supporting further purchases of advertising exposure either online or offline.
It seems to me that, if we put some of our best minds on the challenge, we could create a series of test markets using different levels of advertising exposure (including none) in different markets that might actually give us some better sense of the payback on our marketing expenditures. Certainly I understand that just measuring the short-term impact may be a bit short-sighted, but it seems to me that we should be able (at the very least) to determine where we get NO lift in sales in the short term, and safely conclude that we are unlikely to get the payback we seek in the longer term either.
Clearly I'm not an expert on this topic. But my experience tells me that we are not approaching our marketing programs with enough emphasis on learning how to increase the payback, and are at best just getting better at spending less to achieve the same results. While this benefit is helpful, it isn't enough to propel us to our growth goals and, I believe, presents an increasing risk to our continued profitability over time as markets get more competitive.
I'd be delighted to spend some time discussing this with you further, but we need a new way to look at this problem to find solutions. It's time we stop spending money without a clear idea of what result we're getting. We owe it to each other as shareholders to make the best use of our available resources.
I'll look forward to your reply.
Thank you.
So how would you respond? Comment below. I'll discuss the most creative/effective responses in two weeks.
Efficiency is terrific, once you determine what it is you're trying to accomplish with the process you're improving. Improving an INEFFECTIVE process will ultimately just make you better at running in circles. Even though I'm more of a marketing geek than a finance geek, I applaud this post for reminding us all that marketing efforts have to originate from business strategy and goals, not just getting more ads/placements/whatever for less than the other guy.
Terrific article. Reminds me of the saying how one of the quickest ways to go out of business is to become incredibly efficient at doing the wrong thing.
Ms. Ivers is right on target here.
Unless a product or service is successful apart from its advertising, the ROI curve will eventually level out.
The solution is not in the improvement of an advertising approach, but in the improvement of a product/service.
This has been a debate that has gone on in every company that has separate Finance/Accounting and Marketing departments since time immemorial. Accounting wants to count dollars and cents going out and coming in, Marketing is dealing with the perceptions, emotions, wants and needs of their potential customers. Casey is correct that you need to begin with a business strategy and specific goals. In addition, recent advances are now better at determining conversions. However, trial and error based on educated guesses, knowledge of your product or service and of your target market are still the rule. The fact is, you never really know where or even what medium your client first discovered you or what media influenced their decision and how it was influenced. Just to add to the confusion, customers change media as well as perceptions over time, which means that no marketing plan will be effective over time.
I think Amy Ivers knows that increased sales for the corporation and improved annual returns on investments are the key barometers of a company's marketing success in 2010. Amy would make a great CEO at any company with her focus on how to grow a business.
Over the years, company such as Oracle and Citicorp have inherently asked the same question - literally going dark on all awareness building media to determine if there was a materially negative impact on channel yields. In fact, what Ms. Ivers recommends is what econometric-based 'marketing mix' companies do - determine the effect one stimulus has on another and where diminishing returns occur. I, personally, have led such engagements, and the results can have Board-level implications, such as (a) the brand has such strength that below-market levels is now required, (b) good ol' TV is 80% responsible for our paid search yield (not Google), or even (c) display is driving registrations, but extrapolating the increase in display to reach competitive yields would bankrupt the company. What's more important here is not optimizing mix, it's establishing and separately effectiveness vs. efficiency yields. The emerging real-time display bidding platforms (RTB) like Turn, DataXu, and AppNexus are fundamentally media efficiency experts. The effectiveness component is the engine that most effectively engages the target audience to improve acquisition volume, velocity, and value; as well as cross-sells and retains the most valuable customers.
Yeah -- Amy's a bit of a buzz-kill, but that meme about "efficiency at doing the wrong thing" is pure bean-counting techno-wonkery at its resonant best.
Imagining Susan's reply: "Yes, it would be great to know this stuff. Of course, the only way TO know it is to turn our global advertising operation -- now both a highly-leveraged and highly-agile operation -- into a stable statistical test-bed, and then run it exhaustively against what we hope is a valid model of customer behavior.
"That's doable under the following conditions:
"At this point, in order to achieve the GRP, click-through and other short-term goals we've been given under budget authorized, we're treating a very complicated set of choices as black-boxes, via aggregators. So first, we'll need to unbundle essentially all of the cost-efficiencies in our operation, and bring all this decision-making back in-house, in order to have point-control of all relevant variables through 100% a-la-carte purchase of custom media plans."
- "Then we'll have to set up real experiment and control groups -- probably a LOT of them, which will occasion a great deal of planning, management and administrative cost -- that will let us run the same creative out over infinitesimally-varied media plans, and do this enough times, and long enough, to build a reliable map of dependent and independent variables in the system as they pertain to us.
- "To do this right, you guys will have to agree to run an entirely-static messaging and creative program through all media for the duration of the experiment. I'm sure -- sophisticate that you are -- you understand that if you run the ads with the cute kitten, they're going to pull differently from the ads with the scary clown for reasons not easily associated with the media buy and related operations. So what would really be best is to run for a year or so -- everywhere -- with really, really flat creative, so we're sure our stats aren't being deformed because some agency art-director is making us look cooler than we really are."
- "As for actually retrieving information: these new media plans will require special instrumentation. We're talking about instrumenting, surveying and nailing-down an influence-chain that might start in a totally-forgettable IMU display, turn into a complex sales/engineering-engagement 18 months long, ballasted by branding ads in leading magazines, white papers, and 50 other kinds of collateral and programs, and ending (hopefully) in the purchase of $40million worth of terabyte switches, right? And I know you understand both the mind-boggling complexity of that undertaking, and the need to repeat the process with tiny variations again and again in order to understand and measure the relevant phenomena -- as well as the need to explore failure conditions around the edges of the system, which will certainly (and more than once) involve doing things that deliberately prevent the consummation of the desired sale in order to refine our model."
"Looking forward to discussing this further for the entire remainder of our careers!"
From the wonky, bean-counter side...she asks some pretty good Mktg questions. Interesting, too, that while she places value on "spending less", she also places value on "audience", which to me means Brand...and therefore harder in most cases to quantify, all at the same time. Good stuff.
Dear CMO, Just call it BRANDING! If that line still works...
Regards,
Chris
What a great reminder about effectiveness to Susan James. However, I would love to hear her response to this email.
I echo Casey Quinlan's comment about efficiency - "Efficiency is terrific, once you determine what it is you're trying to accomplish with the process you're improving. Improving an INEFFECTIVE process will ultimately just make you better at running in circles." Well said!
Sounds like my conversations with our CFO ... So I just show him the RumbaFish analytics.
Since when is it the media metric's job to perform? I thought that was the message's job. And why doesn't Amy already know this?
In terms of the reply, I think it is based on who the CMO is...
... if it is a traditional marketer then it would probably be in the realm of "oh sh*t, I think it may be time for me to take my leave if we are actually going to look at the bottom line impact of the area I manage". Or they could be really honest and say "well this year I thought I would make even more efficiency moves by screwing our already commoditised media agency into complete submission by reducing their margin even more and ensuring that they can afford to add no value to the company moving forward. I hadn't thought about all this testing malarky."
... Personally it sounds like data nerd porn. Love the test and experimentation approach to different media in different markets and while we're at it lets use multivariate testing to decide on offer, creative and web-site structure too.
Then overlap the econometric data and media attribution analysis with the MVT to provide the most efficient and effective mix of offer and media in each market (geographical, vertical or otherwise) and see the results fly from there.
While we're at it, lets build a neural network that 'learns' the effect of these various levers and allows us to utilise the most effective system in 'new' markets from the very start.
However the most important thing that Susan could do, the most immediate and possibly the most valuable would the most basic. It would be to reply and say "Please, stop talking about marketing spend or expense dollars and start talking about marketing investment."
...but then again, I guess that's the point :o)
Love the post Pat - once again you are right on the money. Mind you, my Aussie sense of humour for a reply goes along the following lines ...
Dear Amy,
You provide very sound advice that I intend to apply rigorously. Starting now.
It has come to my attention that since you joined our business as CFO, that the profitability of our business has PLUMMETED! Yes, the bottom has fallen out of both our effectiveness and efficiency. I've done the analysis, and the only common variable that keeps re-appearing is you tenure as CFO. Therefore, I can only conclude that you are at the root of all our financial problems.
I'd be delighted to spend some time discussing this with you - as long as you don't trot out that "Global Financial Crisis" excuse I keep appearing.
Kind regards,
Susan.
OK, John Jainschigg's response made me laugh out loud. It bears noting that none of this is 'just numbers,' there is a certain amount of experience, intuition and artistry that goes into media selection and placement. And as John pointed out, to the creative. Or we can count angels on the head of a pin until we all keel over.
Amy,
Agree 100%.
And yes I would like to meet with you to walk through the challenges associated with aligning expenditures to value and return.
In fact, maybe we can review the proposal that was rejected at the last planning meeting in which I requested another headcount (Director of Marketing Analytics) to manage the complexity inherent in today's multichannel, real-time and disparate data points media world.
Then, if you have a bit more time, we could review the recommendation (that was also tabled) to invest in the relevant systems to monitor and measure the key performance indicators (value) that are required to for us to achieve and accomplish this goal.
How is next Wednesday at 2pm?
Best,
Randy
Ok, Pat. I'll bite. Below is my response to Amy. Enjoy.
-----------
Amy,
Thanks for the wake-up call. Your e-mail reminded me why I took this job two years ago. I knew you'd challenge me to push myself and always try to think bigger and step back to think about the business.
You are absolutely right about GRPs. GRPs can be effective or ineffective. And click throughs? Forget it! That's so 2000. I'd love to completely give up that metric.
If you are truly committed to understanding how far marketing can take our business, I'd love to have you meet with the digital experts I met with today from Dynamic Logic. They made me completely rethink how we approach digital marketing.
Here is what I learned.
1) The main driver of ad effectiveness is creative quality. This acccounts for 50-75% of how influential those GRPs can be. If we spent as much time and effort on creative quality as we do "optimizing campaigns", we'd see much greater impact.
- Recommendation: Dynamic Logic will present their creative best practices to our creative folks at no charge. They also recommend we test creative prior to putting them in market. Cool blog posting on GRP debate: http://tinyurl.com/yh5jkx9
2) They've confirmed my belief that CTRs are useless for us. They have data based on hundreds of campaigns showing a lack of correlation between CTR and anything we care about (trying to change perceptions of our brand and trying to drive product sales).
Recommendation. Stop reporting CTRs and start reporting metrics that are relevant to our business.
3) Related to (2), Dynamic Logic has given me a simple equation for how to measure the impact of our campaigns and pointed me to an interested blog debate on metrics (http://tinyurl.com/kl4ykc):
The simple concept is this.
Total impact = Reach times percent impacted.
Simple, eh? And, here's the cool part. They have a method for estimating the percent of people impacted. It's called AdIndex and it has nothing to do with CTR!
As we were talking, they made me realize that, for us, only two things matter. Our ads are effective if they either change consumer perceptions about us or drive product sales. They have a way to measure both.
And, I love the fact that the formula worked for all media. We want our TV ads to change people's opinions about our brand. So, why not hold the same standard for our digital advertising?
4) Finally, you know how we are always worrying that we are not staying on top of the next big thing? They made me relax about that too. They advised me to MASTER the PRESENT and TEST the FUTURE. They have this cool website where they recently talked about how people spend too much time chasing the next big thing (http://www.dynamiclogic.com//AskKen/).
Recommendation. Mine Dynamic Logic's MarketNorms (http://www.marketnorms.com) database of thousands of campaigns to learn from other people's successes and failures so that we can step back and make good, solid decisions about many of the digital media choices we have (ad sizes, formats, website choices, use of video, etc.).
I know what you are thinking --- "Dynamic who?" But, these guys are for real. They've been doing this for 10+ years. What were you doing ten years ago? The founded their company on the believe that CTRs were of little value and they devoted themselves to giving marketers relevant metrics.
They've measured the impact of 6000+ digital campaign and have about 200,000 creatives in their database.
So, Amy, thanks again for diverting my eyes away from spreadsheets. Your e-mail was so timely given the conversations I had today with DL.
I'm going to arrange for them to come in to talk to you. I'll find 90 minutes on your calendar.
Thanks,
Susan
-------------
Pretty creative stuff folks.
Thanks for contributing.
Putting aside for the moment any fervent beliefs in the power of a proprietary solution to answer all the questions, what would you do if you were CMO??
What approach would 1. diffuse the issue; 2. address it effectively; and 3. leave the company better off as a result?
Keep those cards and letters coming :)
Thanks for sharing this, and do love the comments as well.
Just to add some levity to the discussion:
Amy, we live in a world that has clicks, and those clicks have to be guarded by media buyers. Whose gonna do it? You? You, CFO Ivers? I have a greater responsibility than you could possibly fathom. You weep for ROI, and you curse the brand. You have that luxury. You have the luxury of not knowing what I know. That the lack of ROI metrics, while tragic, probably pushed units. And my existence, while grotesque and incomprehensible to you, pushes those units. You don't want the truth because deep down in places you don't talk about at parties, you want me on that site, you need me on that site. We use words like display, search, code. We use these words as the backbone of a life spent defending something. You use them as a punchline. I have neither the time nor the inclination to explain myself to a CFO who rises and sleeps under the spreadsheet of the very service that I provide, and then questions the manner in which I provide it. I would rather you just said thank you, and went on your way, Otherwise, I suggest you pick up a mouse, and start clicking.
Of course, paraphrased from A Few Good Men. :-)
I haven't had this much fun reading a comments stream in years! (And I've got to figure out how to tap into my "inner Aussie" next time I've got to respond to a CFO -- best {snarky} response of the bunch!)
Pat,
This letter comes as no surprise; every day, we see more and more conversations like this between the Susans and the Amys of the world. And while replying to these kinds of emails is never easy for CMOs, it has become an increasingly necessary wake-up call.
Everyone still uses the same out-of-date metrics. Susan and other CMOs need to respond by no longer just looking at media efficiency metrics that are easy to obtain, but to demand metrics that are actually valuable and can demonstrate marketing effectiveness to the CFOs of the world. It’s not to say the “old” metrics are no longer useful, because that’s the currency most people still use, but there are few very important points the marketers are missing.
In an ideal world, here’s what Susan would be saying:
--------
Thanks, Amy, and I agree. Our current media metrics, which include CTR, GRP and CPA within a channel only give me the half the information I really need. Currently our campaigns are optimizing their creatives, keywords and offers in silos. In short, our teams are busy in optimizing ineffective metrics. Like you, what I want to know is how our overall advertising helps our business metrics such as Return On Media Investments (ROMI), Life Time Value (LTV) of our customers and Net Present Value (NPV) of our company.
My first step to resolve this is to look at all the channels together and see how it affects our retail and online sales. When I say all channels, I mean online display, online search, social media, TV, radio, OOH, events and print. There is so much lift given and taken between them, but as of now, I can’t see it because I’m looking at the channels in silos. I also need to be able to analyze the cross-channel paths that our customers have taken before they have converted. Not all the conversions are equal and not all the customers equal, so our metrics must discriminate the media that brings high ROMI and high LTV customers to the table.
One way I’m going to do all this is by using advanced attribution and modeling techniques. With carefully designed tests, I can find this lifts between channels, business units, publishers and campaigns. It will tell me the effect of TV on our Search campaigns and how the social media campaigns affecting the retail sales—both give us better insight into what it means from an actual dollars and cents perspective. We’ll be able to predict how much more revenue we can recognize from all channels together, or even how my increasing, say, the Print Channel, will impact our bottom line.
Susan
------------
Only a holistic campaign strategy that is strongly based on customer analytics and business metrics will give CMOs the results their CFOs now require—ultimately, how each program is impacting the bottom line.
Note: Visual IQ provides holistic cross-channel attribution management systems and marketing mix models based on client specific business rules/metrics. Our clients enjoy an average of 22% incremental ROMI by using our solutions.
Wish you all the best
These comments and the article itself raise two issues: first, both the CFO and CMO are not looking at the business as a whole. Someone above commented that the CFO was being a "beancounter". Unfortunately, that seems to be true. But there must be a corollary term describing marketers who follow metrics toward the penalty of the greater good. Given the information provided, Amy is being rewarded for things that obviously make no sense to the one providing the budget for the activities. Does that make sense to anybody?
Second, a strategic partnership between financial and marketing departments is a fundamental component of overall business success. Either the CMO or the CFO (ideally both) would initiate a collaborative partnership, agreeing on business objectives and parameters, and engaging in a shared information exchange around their particular technical skill sets, identifying which translate and which don’t.
Or, they could air their dirty laundry on a major website like this for all to see.
Dear Amy,
The solution is combination of data from various entities providing metrics with a single source data stream that measures viewing ads/commercials from the TV/Internet/mobile to actual point of sale.
The method to attain this data is to qualify a 100,000 member panel by having each member opt-in and complete the Xyting Insight assessment once. In return panel participants receive a 15-page assessment of their strengths and various insights that they can use in their daily life. Once the panel members have been profie qualified, an entity such as Experian can filter out the Personally Identifiabale Information (PII) and merges the data with other data on members' purchasing of CGP products, shopping habits, the set-top box, and loyalty card data. The resulting set of consumers that can be tracked from their viewing habits to actual purchases to determine ad effectiveness and explain "who does what and why". What is vastly important about tying this data together with the behavioral profiles is that the result provides the means for advertisers to match their commercials with the right media mix, right program, right tailored marketing message, and adjust in real time.
The Xyting Insight instrument utilizes a system-science model based on the way a person processes information and is statistically both reliable and valid. It goes beyond demographics to identify the innate, predictable behaviors of consumers including their intrinsic motivations, input preferences, how they process and retain information, they buying habits, and more. This information is used to tailor the marketing message for better engagement of consumers in the advertisement.
The Xyting Insight qualified panel, syndicated data, custom market research, loyalty card data make measurement of ad effectiveness a reality--the holy grail of marketing.