Earlier this year I posited that, due to the exponential rate of growth, the amount of data collected for analysis is becoming beyond the scope of the current analytical staffs to examine it all.  And that the answer to growing the cumulative brain power necessary for this exponential growth in analysis is going to have to be machine learning and artifici

The amount of data that is now available for analysis is growing at an exponential rate.  As the hardware to capture the data becomes cheaper, faster and omnipresent, the amount of data collected for analysis is becoming beyond the scope of the current analytical staffs to examine it all. 

The theme of this blog is an examination of forces that would disrupt existing data warehouse implementations.  I categorize these as either long tail or black swan events.

During my first coding class, after the obligatory “Hello, World,” my assignment was to find the least cost path in a matrix of values (linear programming).  The computer took some time to crunch the data to come to a solution.  Now, this is an app on your phone.  

This blog concentrates on the expected unexpected external factors that can have a (negative) impact on your organizations’ Integrated Data Warehouse (IDW).  The current discussions around what NSA can and cannot capture and store for data analysis got me thinking about the biggest elephant in the room:  the government.

A couple of recent articles in Wired got me thinking about just how social media services, and thus the value of the big data that they create, could be under threat from their own customers.

When this blog started, it was based on my 2008 Teradata Partners User Conference presentation on the future data explosion and its impact on the Enterprise Data Warehouse:  “’Long Tails,’ ‘Black Swans,’ and Their Impact on EDW and AEI.”  One could foresee a definite future negative impact on the existing EDW platforms trying to support all this new volume and types of data (social, sensor, etc.).

Most of the “Big Data” examples deal with enterprise analytics being run on terabytes or petabytes of data.  However, this example shows how widespread social data analytics have become.  And, as I live in the wine country, this example is quite close to home.

Darryl McDonald, President of Teradata Applications, tweeted this link today:  a practical example of when many of the subjects discussed on this blog come together.