• by Dave Morgan , Featured Contributor, October 25, 2012

Today I would like to introduce you to Jack Smith, our chief product officer at Simulmedia. He's a digital advertising visionary, and the founder of Solariat, a venture-backed company that uses artificial intelligence for social media marketing and analysis. Before that, Jack founded and ran The MIG (now Xaxis) for WPP Group, where he created ZAP & B3 to help agencies and advertisers improve the process of planning, buying, trafficking and optimizing digital media advertising campaigns.

Dave: Jack, tell me about this conference you think is so important.

Jack: It’s likely the least sexy conference in the world to the marketers and agency employees who get to regularly attend The Cannes Lions or Festival of Media.  But as it kicks off this week in New York as a part of Data Week, the Strata Conference + Hadoop World show how the role of technology, science and big data will forever alter the makeup of marketing organizations.  As customer data grows at an astronomical rate and new types of customer interaction media like social make this data less structured, marketers are turning to new tools to handle the volume and variety of big data.

Dave: Tell me more about Apache Hadoop, the open source framework for the management of clusters of large scale data sets. 

Jack: If traditional customer relationship management data is the gasoline of marketing organizations, big data is the new crude oil.  The Apache Hadoop project contains tools for the management of this data -- and increasingly, the tools like Hive and PIG to refine this crude oil into actionable marketing information.

In March of this year, IDC estimated that the Hadoop market will grow at a 60.2% compound annual growth rate to more than $812 million in 2016.  But that’s just for Hadoop itself.  Its impact will be much wider and felt across content creators, agencies and marketers’ technology infrastructure, culture and talent, and new business models.

Dave: And I should care about this because...

Jack: Relational databases full of summarized, structured customer data are good at answering questions about what happened in the past.  But these data structures are expensive and many require proprietary hardware. Hadoop typically holds data in a more raw form.  Little gets thrown away.  So, new mathematical models may be applied in the future to historical data to better predict what consumers are more likely to do.

The rise of demand-side platforms, data-management platforms and real-time bidding systems in digital marketing represented the first attempt at media agencies to embrace big data as a way to create new kinds of scalable, technology-based businesses based around Hadoop and other big data tools.  Marketer who resist these new agency business models are following suit by building their own data and trading platforms.

Dave: But there’s a big downside in having all this data online: You need the right skills to make sense of it.

Jack: Indeed. Data scientists have been rare in the marketing suite or on agency account teams. in October’s Harvard Business Review article “Data Scientist: The Sexiest Job of the 21^st Century,” Thomas Davenport and D.J. Patil point out “the challenge for managers is to learn how to identify that talent, attract it to an enterprise, and make it productive.”  At our company, we’ve seen that the most effective data scientists come from non-media backgrounds. Scientific methods and technology savvy are increasingly present throughout marketing organizations. To see this kind of culture shift inside an agency, you have to look to the digital groups.

Dave: You get the final word, Jack.

Jack: Whether it’s for mining unfiltered, “firehose” data from social networks to create a more optimal media mix, employing real-time bidding for digital banner campaigns, or segmenting television audiences through return path set-top-box data, big data infrastructures and Hadoop are here to stay in marketing.  While it may not be the sexiest advertising conference, Hadoop World may well be the most important one this year. I’ll see you there.