Big Data is generating big buzz in the business IT world. And it has big potential for allowing firms to assemble much more information, from a wide variety of sources, and extract actionable insights from it.
But Big Data also poses a big challenge for regulatory compliance. The Big Data compliance stakes could be critical for industries such as finance and health care that are subject to tight regulatory standards.
The compliance challenge posed by Big Data is not just its sheer volume, but its complexity and lack of consistent structure.
Until the Big Data wave started to hit, most compliance-critical enterprise data was stored in relational databases with a highly structured format. The notion of data warehousing – which now sounds so quaint – was built around this assumption, which was valid for the most part. If you wanted to look for compliance-critical data, such as customer Social Security numbers, you knew where and how to look for it.
Yes, there were exceptions, such as sensitive emails and other documents, but the volume of such unstructured sensitive data was relatively limited. With a bit of careful compliance policy it could be managed. Now, the emerging Big Data era means that compliance policy must come to grips with enormous volumes of mostly unstructured data.
The key to ensuring compliance in dealing with Big Data is to track down and isolate the compliance-sensitive portions of that data. As one compliance and security expert, Jon Heimerl, says, "big data stores are leading organizations to not worry enough about very specific pieces of information."