Big Data – Buzz Word or Way of Life?

The most definitive thing that can be said is that everyone has a different definition of Big Data.

Historically, industries such pharmaceuticals and financial services have been dealing with Big Data for a long time, but it was called by different names, such as data marts. Further, the use of Big Data was focused on solving complex business problems, and not necessarily viewed as a replacement or augmentation for SIEM. These industries are now beginning to analyze Big Data with an eye to solving information security issues.

Challenges around Big Data include:

Securing the enormous amount of data
Determining who has access to the data
Determining what data feeds are going into the Big Data store and how much data must be brought into the organization to make it useful
Identifying and engaging people with the skill set to work with Big Data. It is found that the analysts aren’t security people; they are usually people with Ph.D. degrees in mathematics or statistics.

Best practices include:

Having the right business buy-in and focus and support for Big Data projects
Anonymizing the data to support privacy needs
Determining the realistic amount of data that must be put into a Big Data framework
Standardizing the data so that databases can talk to each other and connect the dots

Two high-powered tools that are being used to perform Big Data analysis are Palantir and the open source Hadoop tool. One organization that is providing useful information on threat data is FS-ISAC.