The Dilemma of Social Media Measurement

Editor’s note: I am thrilled to welcome our newest guest author on the GreenBook Blog: Michael Wolfe of BBDO. Michael is a renowned expert on marketing mix modeling, social media metrics, and advanced analytics and is a sought after speaker and contributor on those topics. He is doing some ground breaking work on developing new, ROI-driven models for social media analysis and it is a great honor to have him share some of his insights with us.  Michael is getting to the meat of what drives investment in research in this post and poses a provocative challenge to market research at the end. He will also be sharing more details on his work with all attendees of the upcoming Market Research in the Mobile World conference next week in Atlanta. I think all of our readers are going to really enjoy this one!

Everyone with any sense of what is going on in the field of marketing right now knows that there is almost a mad frenzy about social media, the likes of which we have seldom seen.    In fact, this frenzy has become so strong that it has spawned almost a gazillion self-proclaimed experts and gurus.   It almost feels like the American Wild West, with the itinerant snake-oil salesman coming to town to sell us a bottle of his miracle cures.

As I have reviewed the literature on this subject, there appears to be three camps or approaches towards social media measurement.   These are as follows:

1. The word counters.   These simply count the frequency of mentions of any given subject or brand name across social media such as Twitter, Facebook and various Blogs.    The theory behind this approach is that word frequency is sort of like market penetration, the more the better.   Some who follow this approach might go one step further and parse their word counts into positive and negative sentiment groups.  This sometimes evolves into more sophisticated sentiment analysis via sophisticated text mining tools.
2. The influence enumerators.   This camp believes that it’s not the words that matter, but the influence of the person saying them.  Via some algorithms for scoring individuals based on number of friends, followers, re-tweets or klout scores, it is possible to differentially score the word counts based on the level of influence of the person posting the conversation.
3. The linguists or engagement scorers.   This approach believes that language and semantics is everything.   Through the science and rules of linguistics, it is possible to “quantitatively score” social media conversations and determine how “engaged” these conversations are.   Influence and impact in social media is very much a function of language, word nuance and context.

Presently, there is a lot of confusion regarding which approach is most valid.   The only research I can cite is from Bernardo A. Huberman of Hewlett Packard.   Bernardo found that social media can predict such things as movie success.  He also found that the quality of a message can pierce through the noise and influence an audience.  He likewise found that the number of friends and followers on social media does not add much to the influence of the message.

The impetus for developing social media metrics is obviously targeted at developing a capability to measure the impact of social media conversations on people’s attitudes and behaviors.   Significant here is the conversations on brands and how these conversations affect brand performance and ROI.

My own journey here tends to favor the linguistic approach.   As a player in BBDO’s marketing-mix modeling consultancy, clients challenged me to come up with an approach to measure the impact of social media on their brands’ performance.   My approach to this is fairly simple.   That was to find a metric which correlates best with brand sales of any of our 8 blue-chip clients.    In this journey, I tested and evaluated many of the “word counting” and “sentiment” metrics from major social media data vendors.  Sorry to say, none of them were found to have greater net correlations to brand sales of about 0.15.   Some of the sentiment indicators, in addition, even had counter-intuitive relationships with client brand sales.

Our journey finally led us to a metric called the Semantic Engagement Index or SEI as developed by a company called Linguistics Insights.  This metric was developed by Dr. Peyton Mason and Dr. Boyd Davis.  Davis is Professor of Applied Linguistics at the University of North Carolina.  Together, Mason and Davis developed an approach which first parses large quantities of social media conversations into positive and negative sentiment groups.   From there, each group is quantitatively scored using Linguistics science and rules.   The scoring algorithm used defines consumer engagement by scoring conversations on two dimensions.  Those dimensions are the degree of personalization and emotional effect contained in the conversational language.’

With our first client, we generated a time-based SEI metric.   We plotted this metric against brand sales for one of our clients.   As shown here, we plotted the ratio of positive to negative SEI.  It’s correlation to brand sales was an amazingly high +87%.   We have never found any single social media metric which tracks so closely and has such a strong connection to brand sales and performance!

We have since found similar and strong correlations of the SEI metric with brand sales of other clients.  We obviously think we have made a very significant discovery here.   Certainly, our efforts have enabled us to include the SEI metric in our marketing-mix models and directly measure the impact and ROI of social media on our respective client brands.   Overall, the evidence so far seems to support the linguistics approach to social media measurement; but obviously we need more supportive evidence across many more brands in order to draw definitive conclusions.

In sum, our research has found at least three different approaches to developing social media metrics, all with the purpose of measuring its impact on brands and consumer attitudes.   While I believe there is some evidence supporting the linguistics approach, I must say that the evidence to-date of which approach is superior is significantly lacking.   The bottom-line here is that this really represents an opportunity for Marketing Research to step to the center and demonstrate leadership.   Certainly, many researchers have access both to the metrics and brand sales data to do so.  Whether they have sufficient moxie to do so remains to be seen.