My Date With Big Data

I recently closed the doors on my company, a social media analytics start-up. As soon as I dispose of my house, cars, and other major worldly possessions, I’m heading out for some extensive travel to far-away places (a mid-life gap year, if you will).

Image via shutterstock

And on the threshold of these major changes, I can’t help but wonder: what will the analytics world — especially “big data” — look like in a year from now?

Big Data (a phrase used to describe datasets that are so large they become awkward to work with) promises insights that will cut costs, improve efficiency, and, according to IBM and others, make all of us that much smarter. 

Marketers stand to gain big from big data: customer segmentation, one-to-one and real-time marketing, multi-channel attribution and micro-targeting of messages across channels are just a few of the areas that big data will likely improve. (Want more examples? Have a look at Smart Marketing Using Big Data).

But, can big data deliver on such big promises? And if so, how soon can we expect to see some of the benefits that are so widely and confidently promised?

Well, that depends on who you ask. Vendors in the space are saying “yes,” and “soon.” A study commissioned by SAP (through the market research firm IDC) predicts $220 billion in revenue for SAP globally from big data analytics over the next five years.

Others are saying, “Not so fast.” For example, a recent New York Times article questioned big data’s economic value and suggested that companies are perhaps over-selling their ability to manipulate large datasets. (Gartner has predicted for over a year now that we will begin to see more negative coverage like the Times article, simply because big data as a concept will soon be entering the “trough of disillusionment” phase of a normal technology Hype Cycle).

No one knows exactly what the future holds for Big Data and its many potential beneficiaries; but even without  a crystal ball, it’s fairly easy to see some of the obstacles looming in the path ahead for the tool vendors. Below are five challenges facing Big Data.

1. Where Data Resides

“Big data” includes many types of data from many different sources (e.g., social data, public records, sensor networks, RFID — to name a few). To reap real, palpable  benefits, the easy-to-get data (mostly from public sources) must be combined with sensitive, proprietary data that has security fences around it (such as customer data or any data that cannot go outside of the walls of the organization).

The problem of where and how to combine multiple sources of data owned by multiple stakeholders will plague even vendors at the small data level. (Think, for example, of Google Analytics’ Universal Analytics. It’s great that UA allows companies to import their own data for cross-analysis with social data and Web analytics data, but how many companies will eagerly import sensitive data — such as sales data — into a Google tool for analysis?)

For those in the Big Data space, the issue of determining what data can reside where will be especially difficult in cases where datasets are stored across international borders with varying compliance requirements.

2. Access To Social Data

It is easy to forget that much of the social data we think of as free, or public, is not equally accessible by all vendors. A simple example: Twitter’s quota levels for their public API. Requests by some vendors for elevated API call levels get addressed by Twitter; others go ignored (many for no stated reason). By doing this, Twitter seemingly picks winners and losers in the vendor space (even if that is not their intent).

The bottom line is that no owner of publicly available social data is beholden to provide it equally to everyone who wants it. And remember, Twitter is only one social source. Each seems to play by different rules that change almost as frequently as their leadership changes underwear. Any data vendor (and again, this applies to small data vendors, too) will be challenged by maintaining uninterrupted access to hundreds (ultimately, many thousands) of social data sources, the crucial linchpin of Big Data analytics for the most basic of marketing applications.

3. Ownership & Privacy

Privacy issues related to collecting consumer information is nothing new. But expect the stories (such as Orbitz up-charging Apple users after data-crunching revealed they are willing to pay a higher price) to command more and more press, especially as end-users get better at analyzing those treasure troves of data for new, interesting (or creepy, depending on your viewpoint) kinds of predictive formulas.

More press will lead to more inquiry, such as the new movie documentary Terms and Conditions May Apply (video), which attempts to educate consumers and encourage more vigilance about what data is being collected and by whom. The U.S. government is several steps ahead of consumers when it comes to concern about data collectors (read: anyone not in a three letter intelligence agency), with several members of Congress expressing interest in investigating data brokers.

Ultimately, some unlucky tool vendors may wake up one morning to find that what they have spent millions to develop is no longer legal.

4. Digestibility

Volume is an issue here (hence the “Big” in Big Data), but the real problem is a lack of people who know how to do something with all of this data. We currently have a desperate (and well-documented) need for qualified data scientists — according to Gartner, 4.4 million IT jobs are likely to be created in support of Big Data, but only one-third of those jobs will be filled due to a lack of trained data professionals.

The gravity of this situation cannot be over-stated. The quality, complexity and potential of tools developed by vendors mean nothing if buyers don’t have the expertise to understand and use the data generated by those tools.

5. Cost/Benefit

Big data is a big-ticket item. It’s expensive. The question for buyers is whether that data is worth it. Given the lack of skilled data scientists (see #4 above), will end-users pinpoint enough game-changing insights in the short-term to withstand ROI scrutiny?

Overcoming Big Data Challenges

With sufficient time and patience (read: funding), each of these challenges can probably be met and overcome. But the business world – and certainly the market for analytics technology — is notorious for demanding instant gratification.

So here’s my gambit: I’ll go on my gypsy way for the next year or so, visiting various sunny and intriguing corners of this planet and leaving behind a well-marked trail of blog postings (oh, if Hansel and Gretel had only been carrying iPads and a GPS, instead of those bread crumbs!), and I’ll leave it for Big Data to work out where I’ll turn up on New Year’s Day, 2015. If Big Data shows up, I’ll buy the drinks and we’ll trade success stories and tales of derring-do in unlikely settings.

C’mon, Big Data. Let’s see what you’re really made of…