• by Max Kalehoff , April 3, 2012

It seems as if every new business-tech venture positions itself around “big data.”

Not surprisingly, so do VCs. So do business and tech publications and blogs, which are dedicating increasing resources to covering big data. Even the PR spin-sters have caught on, as demonstrated by the pitches I regularly receive.

How big is the buzz? Enough to drive Google searches for the term “big data” upwards of 300% since the beginning of 2011.

The problem is that “big data” is jargon. While it may mean something specific in data and software engineering circles, it is too easy to find myriad definitions and interpretations, even from people who work in the technology industry.

Now, I’m not a software engineer or data scientist. But I can assure you there is substance and weight to big data -- particularly in marketing.

To shed some light on big data for those less technical, I turned to my colleague Sanjay Gupta, who leads our engineering team at my company. He previously spent 17 years at some of the world’s top ad-tech venues, including paid search at Microsoft and Yahoo's R&D Center in Bangalore.

His latest challenge is migrating our business to our new technology stack, which includes deployment of Hadoop, a leading database framework for bid-data analysis.

Kalehoff: What is "big data"?

Gupta: “Big data” refers to the huge volume of data that cannot be stored and processed using conventional databases in a reasonable time. While the storage problem has been solved by large offline storage systems, analysis tasks need to combine data in many ways; data read from one source may need to be combined with the data from multiple sources. This is where conventional data processing systems fail. Big-data technology tackles this problem by running logic on large data chunks in parallel and then combining them together in a manner that gives global results. We’re now mining data sets that are so large in volume that they become truly representational of the entire problem space. This leads to more accurate predictions and more valuable insights.

Kalehoff: What industries and applications are most suited to big data?

Gupta:  Any industry that can tap past data, or data from other organizations, to improve its business, process or predictive insights is well suited for big data analysis. For example, the New York Stock Exchange generates many terabytes of data per day. Any trading company would like to mine these data along with past data to understand stock trends.

In the Internet media industry, for example, companies can use similarly large volumes of search, visitor and click data to better understand audiences. A broadcast network can use viewership data to unlock prime times over different demographic segments. A bank can use its historical fraud data to raise alerts for suspected new fraud. An e-commerce company can use its huge customer and sales data to better predict demand and supply. In short, any company that can use large volumes of data -- internal or external -- to learn and derive value for better decision-making is a strong candidate for big data.   

Kalehoff: What is the promise of big data in marketing?

Gupta: The ability to leverage big data has opened opportunities for startups to build tools and platforms that can surface insights around demand, supply, consumer behavior, segmentation, positioning and targeting. Big data is a natural fit for marketing because of the huge volumes of sample data that can drive analysis and more meaningful predictions.

Kalehoff: Do you feel the marketing industry is properly equipped to manage the voluminous amount of data needed to make smart marketing decisions?

Gupta: Big-data analytics require data to be collected from every step of marketing flow, continuously and accurately, so as to mine the “true function” of the output with a high degree of confidence. The biggest challenge the marketing industry faces is that it is not equipped with technology for real-time collection, storage and analysis. Most marketing organizations also lack process, and face significant restrictions that prohibit the exposure of data for analysis.  

Marketing departments need better coordination and capabilities to share and analyze data in a more systematic manner. Interpretation is the biggest challenge, and understanding patterns and surfacing insights demands high intimacy with the individual business and larger market. Analysis of big data trends can be outsourced to external agencies, but the closer analysts are to the business, the better. That’s why organically developed capabilities are often best.

Kalehoff: How have Facebook's recent moves changed the big data landscape?

Gupta: Facebook’s marketing platform continues to emphasize the need for real-time data and insights. Launching, tracking and optimizing brand presence on Facebook, across multiple APIs certainly requires big-data analysis in real time.

Importantly, marketers want not only insights into Facebook, but insights into Facebook that are integrated with data and insights from multiple other channels, including internal customer databases. There is a huge opportunity to integrate and mine these large volumes of data in order to surface hidden relationships and business insights, and present them beautifully and persuasively to key stakeholders.  

Kalehoff: Will developments in big data prompt the Federal Trade Commission or other government regulatory bodies to ramp up efforts to protect consumers' online data?

The promise of big-data analysis relies on accessing trends and patterns on a large volume of past data to predict the future. As long as consumers’ online data are used to mine insights of entire segments and marketplaces without isolating, storing and tracking personally identifiable information, there should be no issue. A clear mandate is that data from different sources will need to be processed (transformed or encrypted) at the source in real time, and not stored, transported or aggregated offline in original form. This is going to be the biggest challenge for big-data systems.

Kalehoff: What comes after big data?

Gupta: Today’s big-data analysis is used to optimize existing business. The semantics of big data is shifting from “data of action” to “data of intention.” The future of big data will be to use it as a tool to discover new segments & audiences, and invent new products.