Data in motion divides the haves and have nots

Data in motion is not a new term, but it has more meaning today than ever before. There is a significant shift in the way technology is being used thanks to the increasingly cheap price of RAM (Random Access Memory) and an increasingly interconnected world.

RAM was always the ‘pricey’ memory that was also volatile…it didn’t hold data when electricity is cut off. But engineers just can’t resist the idea that RAM is 10,000 times faster and the perfect way to manage our high-speed, complex world.

Prices plummeted, engineers did their clever work.

We’re now able to use RAM, AKA in-memory computing, the same way we used disk storage not long ago. The implications are enormous as we can now put data in cache memory in the amounts that allow for looking at enormous amounts of information in the moment, not through the ‘crunching’ (processing chunks at a time) that we’ve adapted due to database and processing limitations.

We can use this speed to make retail, manufacturing, healthcare…you name it, faster.

We’ve come so far

It’s hard to believe how far we’ve come. My first computer experience was my school’s 16 kB Commodore Pet. By our standards back then it was incredibly fast. It held onto data waiting for the next command. It was perfect for managing data at rest. This pattern is still what occurs in many systems today. Despite the clear benefits, In-memory capability hasn’t been evenly adopted.

For many businesses, it just takes time to get the organization aligned and a willingness to go through the changeover. The ‘chasm’ analogy works very well here as the early adopters and early mainstream have this figured out. Using in-memory processing they know that they can:

  • Use in-memory data as a way to link applications without ‘writing’ to yet another database
  • Scale data capacity 1:1 by adding servers without increasing management overhead
  • Persist (save) in-memory data at the level required by the organization, no more, no less
  • Use unstructured data not just structured records of columns and rows

Leaning forward

These are the technical benefits, but what does this mean for business? It means organizations are able to perform analysis on enormous amounts of data, making the output real-time and actionable. It allows them to bring in any information from anywhere and quickly make it part of their decisions and actions.

Ultimately, they are able to manage a model of their world, defined and redefined by hyper-fast analysis, enriched with information from anywhere that allows them to lean forward, anticipating opportunities and threats. They are the information have’s and will quickly out-compete the information have-not’s.

This doesn’t mean that relational databases are dead. Far from it. It had and continues to have its role as the safe, peaceful place to keep data at rest.


Tags: , , , ,

Categories: Data Analytics / Big Data, Disruption, Information Technology, Real-time

Author:Chris Taylor

Reimagining the way work is done through big data, analytics, and event processing. There's no end to what we can change and improve. I wear myself out...

Subscribe to the blog

Subscribe and receive an email when new articles are published

One Comment on “Data in motion divides the haves and have nots”

  1. July 24, 2012 at 6:54 am #

    Chris, I see this very issue hampering many of the organizaitons I work with daily. They are trying to move to a point to where they are making mistakes moving forward vs. sitting still (which I’ve tound to be the worst mistake you can make). The management and staff fully support the approach (the culture is there), but their data can’t keep up with them.

    While they want to make the right decision, they can’t get the information they need. They can only move forward as fast as they can get the basic information needed to move forward.

    And, I’m constantly amazed at how little data is actually required to make solid decisions. The “minimum” or “basic” level needed isn’t that great, but organizations can’t turn the massive amount of data into the basic information they need to lean forward.

    As you and I have discussed, this is usually due to the fact they don’t have an enterprise way to look at their data. They get data flying at them from various functional repositories, and ERP, and three different customer/marketing channels. There is no framework to tie it all together to give it context.

    I’ve found that once an organization crosses this threshold to understand how to view their enterprise data, they can get very agile very quickly on a minimal amount of data. With the right information available, brigth people will act quickly (and correctly most of the time), I’ve found.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: