Forget the ‘big’ part for a moment…think about data – InterOp Las Vegas

Sometimes, just sometimes, what happens in Las Vegas shouldn’t stay in Las Vegas. That was clearly the case this morning when TIBCO CTO Matt Quinn took the stage to talk about the myths and realities of Big Data. In Why Big Data Won’t Make You Smart Rich or Pretty, Quinn provided his perspective on Big Data based on years of experience in some of the biggest data environments around like FedEx and Nielsen and others.

Forget the ‘big’ part for a moment, think about variety

Big Data PuzzleQuinn pointed out that most customers are not struggling with ‘big’ data but are instead still struggling with data. In Quinn’s view, it is the complex interactions between customer data sets that cause the majority of the issues. Success depends more on piecing together different data sets across wildly different applications and systems with variety of data being the key.

In Quinn’s opinion, solving this data ‘jigsaw puzzle’ is often overlooked and tools like Hadoop, while clearly in focus, is just one tool in the toolbelt and can be a clumsy tool when dealing with real-life complexity.

New ‘new’ application architecture

Quinn went on to say that he’s seeming more and more focus given to aspects of Big Data and Cloud as people build new business applications. According to Quinn, these applications have common elements that stand out:

  • While operational data is still being stored in a regular relational DB;
  • MongoDB, Cassandra, Giga, ActiveSpaces used more frequently to handle transient and fast moving data for real-time analysis
  • With direct integration to some flavor of Hadoop for offline or batch oriented analysis

The resulting analysis is still being stored in an operational data store as of today, but that’s mostly due to the various web framework support available more than anything else. This will change as new frameworks gain support, leading to an increase in use of the new architecture. 

The big momentum challenges

Early iterations of Big Data projects were driven as science experiments, often managed by consultants long on theories but short on practical experience. This drove vendor overload and left the marketplace without many specific success stories for a particular industry or a particular problem. For some, it became and excuse for IT to ‘play around’ with data. Unfortunately, getting a handle on the real challenge means taking a step backward to think about the data itself more than its size. It’s hard to take that step backward for most organizations as it involves decisions, investments and personalities.

Obstacles and opportunities

Quinn raised data security and organizational change as two of the biggest obstacles and opportunities. Societies and our various cultures are being affected by the privacy and other questions that Big Data raises, leading to data sovereignty and protection legislation entering the discussion. Getting specific, Quinn talked about PCI data and the need to anonymize or keep ‘chinese walls’ between business units that could abuse the public’s or customer’s trust with too much access.

On the issue of organizational change, Quinn points to the science experiments that are not helping to inform companies of what potential change looks like. Before the Big Data cycle passes, there will need to be serious thought about data governance and data stewardship and how that gets set up and managed. Quinn painted the following scenario to emphasize his point:

  • Imagine you have 1000s of applications and databases
  • Each has an owner and (probably) a specific business domain and is often from a specific/unique perspective
  • Each is guarded by various IT and business gatekeepers
  • Trying to piece together this data set – in a consistent manner – often takes a centralized data governance group
  • And in most cases these groups have failed (re: company wide data warehouse initiatives)

These are very real issues that others beyond Quinn are starting to raise.

What does it matter if you can’t act on it?

Operational Big DataToward the end, Quinn put up a graph that highlights the business value of data as it decays over time as a way to point out that insights are most valuable when the capture, analysis and decision latency can be kept to a minimum. His customers that are getting the highest value from data aren’t necessarily looking for needles in haystacks, but instead are engaged in shortening the cycles that cause data to lose value.

This point stresses the aspect of big data that is most often overlooked… that finding, understanding and deciding how to act on data may very well mean more than amassing more data and taking longer to get to the place where the organization can move.

From a debunking of the hype point of view, Matt Quinn’s talk was a great start to a day focused on myths and realities of Big Data. I’ll be following up this piece with more on the InterOp Big Data Workshop happening today in Las Vegas.

Advertisements

Tags:

Categories: Data Analytics / Big Data

Author:Chris Taylor

Reimagining the way work is done through big data, analytics, and event processing. There's no end to what we can change and improve. I wear myself out...

Subscribe to the blog

Subscribe and receive an email when new articles are published

4 Comments on “Forget the ‘big’ part for a moment…think about data – InterOp Las Vegas”

  1. May 7, 2013 at 10:07 am #

    Except there are some foundational flaws here;

    1) You are still operating in outdated paradigm that says you have to wait to the data is done being generated or is stored AFTER the business event before you can put it analytical use. Not true, making use of it in flight is readily done today.

    2) The notion that business value always tails-off over time isn’t true, in come cases it is the aggregate or accumulation of history that makes it more valuable. Ditto when new sources are commingled – that can spike data value that has otherwise been flat-lined.

    3) The latency of the action-taken here, per #1, is legacy thought. This a function of choice, not technology limitations, at this point

    4) This is represented as a left-to-right flow as opposed to a loop, loops work better in the real-world if you are impact and quality/precision driven.

    Chris, I understand there is room to “de-hype” the big data noise but doing that without being accurate or understanding how these technologies are actually deployed isn’t as helpful as I’d expect (assuming you aren’t simply expressing TIBCO feeling threatened by the changing technology landscape) from you.

    • May 7, 2013 at 2:10 pm #

      Tom, thanks for your comment. The reality is that several speakers from across the tech spectrum spoke to the same challenge of the volatility of data for operational purposes. You are correct that aggregated data has value…that’s not in question in what Matt provided…it is simply a different use case.

      Surprised an IBM guy has so much room to criticize :-). No single vendor has the lock on innovation or forward thinking by any stretch. In fact, the larger vendors tend to have much more silo’d products that introduce latency, however small, that is a real challenge in the fast-moving-data world.

  2. May 8, 2013 at 10:14 am #

    Chris, respectfully you need to better understand what you are criticizing.

    First, the errors I in your post above I pointed out above are vendor neutral. Second, if you want to talk about commercial firms. you clearly don’t know the portfolio options we offer including our broad array of real-time technologies including Streams that can do analysts while the data hasn’t been stored yet.

    • May 8, 2013 at 1:09 pm #

      Thanks for the ‘broad array’ of IBM advertising, Tom :-). As several expert panelists said yesterday, edge use cases can’t be ‘sold’ to the market as the norm, regardless of how much a company wants to sell their newest software. Streaming data holds remarkable promise for companies that first have the ability to effectively manage the data they already have…and that isn’t everyone.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: