Is Big Data in the Trough of Disillusionment?

Gartner hype cycle explainedBig Data is showing the first signs of heading for Gartner’s Trough of Disillusionment as a spate of voices, some very well known, like Nassim Taleb, author of Antifragile and other prescient books, begin to question the hype. I’m going to show you why Big Data is in the Trough in some ways and beyond that and on the Slope of Enlightenment in others.

The Hype Cycle

In case you’re not aware, Gartner publishes a hype cycle for emerging technologies meant to give us an idea of when we should expect new technology to be commercially viable. With all of the noise in the marketplace fed by both vendors and media, it would be easy to think anything less than the latest product offering is already antiquated and won’t work. Gartner’s Hype Cycle helps to bust that myth.

Because the truth is much different. Most companies can’t afford to chase their technology tail and most IT departments can’t suffer the lost credibility of deploying technology that isn’t ready for prime time.  All technology goes through a hype cycle that follows the pattern in this image, even if the peak may not be as high or the trough as deep…or it could be worse.

Big Data’s tracking along the Hype Cycle

Getting back to Big Data, we’re seeing a technology pattern that is certainly evolving. Gartner has five ‘key phases’ that determine where technology falls, and it would be worthwhile to break down Big Data’s circumstances using Gartner’s familiar language:

Hadoop-Elephant-LogoTechnology Trigger: The technology breakthrough that brought Big Data to the technology stage was the development of Hadoop. While it is a usable technology, it wasn’t developed for public consumption and is challenging for organizations that don’t have strong IT support. Hadoop came about to satisfy the demands of web scale problems like search and social media.

Peak of Inflated Expectations: There were certainly early success stories from the companies that developed and first used Hadoop like Google, Facebook and Yahoo. Their success launched a series of Hadoop-based solutions that attracted VC investment, including companies like Cloudera, Datameer, Platfora and SiSense. Likewise, there has been fast growth for powerful analytics solutions that allow business users to  manipulate and visualize enormous data sets. The marketplace doesn’t lack for solutions.

Trough of Disillusionment: In some industries and use cases, we’re at the point where the rubber meets the road…the point where Big Data delivers or doesn’t. What happens now determines if we’re extending the peak of expectations or starting the slide into the Trough. Patience with experimentation will wear thin over the next year or so and there need to be more ‘everyday’ companies taking advantage of Big Data and talking about their successes. Big Data is headed for the Trough as long as there are more people trying than succeeding, and that’s where we are right now.

Companies can only sell the idea of “uncovering hidden insights” for so long. At some point, Big Data needs to be operational and automated.

Analyzing samplesSlope of Enlightenment: Curiously, in some ways Big Data is on the Slope already in some areas. There are use cases around pharmaceutical clinical trials, customer preference, transport and logistics, healthcare and others. In some cases, second and third generation products are here, especially in the analytics space. There are many home-built solutions. Where Hadoop-y vendors need to be careful is simple: Big Data was here before the hype. People were solving the problem without Hadoop, believe it or not.

You might not realize from the hype that many of the organizations with demonstrated success on the books were managing Big Data before Hadoop was ever developed. For them, Hadoop isn’t a breakthrough and is just a choice among many. This is where you’ll see Big Data hype geting the most eye rolls and groans.

The irony of Big Data is that the places where success is quite clear are often the places where the term simply isn’t used. In those places, the term may actually be disliked and kept at arm’s length.

Plateau of Productivity: This is a far-off land at the moment all but those just mentioned. The mainstream technology buyer is struggling with the plethora of messages and products, suspicious of the arguments that make Big Data sound too good to be true.

Nearly on year ago, Gartner had Big Data nearing the Peak of Inflated Expectations (see below). It will be interesting to see where they place Big Data for the report due out this Summer.


July 2012 Gartner Hype Cycle


Tags: , ,

Categories: Data Analytics / Big Data

Author:Chris Taylor

Reimagining the way work is done through big data, analytics, and event processing. There's no end to what we can change and improve. I wear myself out...

Subscribe to the blog

Subscribe and receive an email when new articles are published

4 Comments on “Is Big Data in the Trough of Disillusionment?”

    • May 29, 2013 at 1:45 pm #

      Thanks, Tom. We both cited Nassim Taleb’s criticism in Wired but used it to make differing points. I am about to read the other dissenters.


  1. Geek Reading May 29, 2013 | Regular Geek - May 29, 2013

    […] Is Big Data in the Trough of Disillusionment? (Successful Workplace) […]

  2. Big Data: the shift from ocean to stream will hit you like a tsunami | Successful Workplace - May 30, 2013

    […] fits the model perfectly. But with people already crying that Big Data has hit the infamous Trough of Disillusionment the industry needs to re-examine what it means to harness data, analyze it and act on […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: