Earlier this year I posited that, due to the exponential rate of growth, the amount of data collected for analysis is becoming beyond the scope of the current analytical staffs to examine it all.  And that the answer to growing the cumulative brain power necessary for this exponential growth in analysis is going to have to be machine learning and artifici

The amount of data that is now available for analysis is growing at an exponential rate.  As the hardware to capture the data becomes cheaper, faster and omnipresent, the amount of data collected for analysis is becoming beyond the scope of the current analytical staffs to examine it all. 

The theme of this blog is an examination of forces that would disrupt existing data warehouse implementations.  I categorize these as either long tail or black swan events.

During my first coding class, after the obligatory “Hello, World,” my assignment was to find the least cost path in a matrix of values (linear programming).  The computer took some time to crunch the data to come to a solution.  Now, this is an app on your phone.  

A couple of recent articles in Wired got me thinking about just how social media services, and thus the value of the big data that they create, could be under threat from their own customers.

When this blog started, it was based on my 2008 Teradata Partners User Conference presentation on the future data explosion and its impact on the Enterprise Data Warehouse:  “’Long Tails,’ ‘Black Swans,’ and Their Impact on EDW and AEI.”  One could foresee a definite future negative impact on the existing EDW platforms trying to support all this new volume and types of data (social, sensor, etc.).

Most of the “Big Data” examples deal with enterprise analytics being run on terabytes or petabytes of data.  However, this example shows how widespread social data analytics have become.  And, as I live in the wine country, this example is quite close to home.

Darryl McDonald, President of Teradata Applications, tweeted this link today:  a practical example of when many of the subjects discussed on this blog come together. 

Continuing on the “42” theme, Chris Anderson drills done on the impact of big data on health and agriculture. 

“42” is the Answer to the Ultimate Question of Life, the Universe, and Everything.  But for Big Data, the answer is simply “everything.” 

Ever wish they would fill-up the pothole quicker?  Municipalities are creating apps to allow the crowd to give them up-to-date notification of problems in the street.  This is in addition to the automated systems that tap into the stream of GPS and cell phone sign

One of the more infamous data usage stories over the last week concerned the removal of the iPhone app, “Girls Around Me.”  This app aggregated Foursquare and Facebook data via their respective APIs to generate a map, with pictures, of wo

I will admit it:  I did not “get” Twitter.  When it first came out, I could understand news organizations and media tweeting information, and well-known personalities tweeting the minutiae of their daily lives.  But I couldn’t understand the draw of the remaining 99% to also publish their lives on a real-time basis.  But my curmudgeonly world filter needs

Over the last couple of years, I have discussed on this blog a number of data explosions coming down the line that will have a huge impact on data warehouse implementations now and into the future.  Many of these topics now fall under the umbrella of “big data.”

Schumpeter wrote a column in last week’s Economist that did an excellent job in summarizing many of the big data sources that I have addressed on this blog.  It is based on a McKinsey Global Institute write-up