Big Data – Problems and Progress

Big Data – Problems and ProgressIn recent posts we have looked at Big Data, and we have identified some big problems. Firms are struggling to come to grips with Big Data. One study found that most of them don't feel they are handling it with much success. It poses security and regulatory compliance challenges. And truly adequate tools for handling it are only just starting to appear.

But that is not the whole story when it comes to Big Data. Another recent study by Wakefield Research, commissioned by vendor Avanade, offers another side to the picture. The half-empty glass, it seems, is also half full.

The Wakefield study of 500 executives and IT leaders finds that nearly three quarters of their firms – 73 percent – report that they have achieved revenue increases by harnessing Big Data. Of those firms that are making gains, 57 percent are doing so by boosting their existing revenue streams.

But in an intriguing angle, 43 percent of the firms that have added revenues thanks to Big Data have done so by creating entirely new revenue streams. The study did not offer a detailed breakdown of these new revenue streams, but did provide examples, such as a shoe store using customer data to gain a foothold in other, related clothing segments.

Some other interesting patterns emerge from the study. Gathering data may remain an IT function. But analyzing that data is now largely separated from IT. Nearly 95 percent of the firms covered in the study now treat data analysis as a distinct function from IT, a function spread across business units.

Challenges certainly remain, though some – such as employee mobility, cited as a problem by 73 percent of firms – are not peculiar to Big Data, but instead reflective of contemporary business culture in general. Other, more IT-specific complications include the cloud, social networking, and the "consumerization of IT."

In the big picture, the Big Data revolution brings with it a mix of good and bad news. Something that, really, ought to surprise no one. But success is most likely to come to those firms that are proactive in meeting the Big Data challenge.

The security principles set forth in industry standard ISO/IEC 27002 provide a framework for effective security, built around the cycle of Plan, Do, Check, and Act (PDCA). Many good security products are on the market, but all are designed to meet specific threats – and will not block other threats. At GRT Corp. our security philosophy is built around these words by noted security expert Dr. Bruce Schneier: "Security is not a product, but a process."