Forget Big Data — let’s talk about all data

The following is a guest post by Matt Quinn.

All DataWe constantly hear people talking about big data as a monstrous mess. They usually describe it as a difficult problem to tackle, as though big data itself has a beginning and an end, or a neat solution. I’ve been to many conferences in the last few weeks, and over and over again, I’ve heard people talking about what they think of big data. Speaker after speaker, it became apparent that the way we talk about big data is our attempt to compartmentalize it.

Compartmentalizing big data is a natural thing to do. Since we first began to track things using numbers, we had to know where the information started and ended because that’s where work began and finished. When we first computerized data, we had to know where a batch was beginning and ending because the digitized world was a transactional one, with data tracking sales, costs, inventory and other finite information.

Keeping it comfortable

That’s the reason we talk about data sets and batch jobs. It fits a paradigm we’re comfortable with. But at the same time, aren’t we missing something? Aren’t we failing to take into account all of the data being created everywhere, every day? The perceived value of different types of data will grow exponentially in the future. Today certain types of data may not appear useful or valuable or may be too difficult to process, but I can guarantee that your future self will see things differently.

What if, instead, we think about dealing with all of the data that already exists and focus our efforts on linking data together, analyzing it more broadly, and letting it provide value to a wide range of people in our organizations? Does that sound much harder than using the patterns of the past? Maybe, but isn’t the difficulty level of something merely a measure of the value of outcome against effort? With the term ‘big data,’ we’re actually thinking small, solving point problems without dealing with the more pressing, more valuable outcomes that are available when we think about all data.

Discovering all data

Some of the most successful companies in the world won’t use the term big data because they’ve discovered all data. One of my best customers is on a constant quest for ever-greater connectivity to any and all consumer preference data. They know that each and every day they learn more about what matters most to their customers. They aren’t putting boundaries around what they’ll monitor, and they have what is considered to be one of the most advanced data architectures in the world.

So let’s get out of the big-data (which is really a small-data) mindset. All data is the real mess to clean up, whether we see it right now or not. All data is waiting to be discovered and put to use. All data is where the real value is found.

This article was first posted on Venture Beat and has been lightly edited.

Matt Quinn is CTO at TIBCO in Palo Alto, California. His role includes creating a common, corporate-wide vision for all of TIBCO’s products and technologies. His past experience includes the role of global architect, responsible for delivery of some of the company’s largest implementations in transportation and logistics, energy and finance.


Tags: , ,

Categories: Data Analytics / Big Data

Author:Chris Taylor

Reimagining the way work is done through big data, analytics, and event processing. There's no end to what we can change and improve. I wear myself out...

Subscribe to the blog

Subscribe and receive an email when new articles are published

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: