To realize how absolutely crucial data integration has become, you need to first think back to the mid-2000’s. The hot conversation was around service-oriented architecture (SOA) and the benefits of loosely coupled business systems that that allowed information to flow between applications and databases through reusable interfaces built to be agnostic to vendor, product or technology. January 2004, a Microsoft paper said, “It seems probable that eventually most software capabilities will be delivered and consumed as services.”
SOA’s time had come thanks to the slow but steady maturing of the Internet and Web services, which starkly exposed data silos as costly, risky, unnecessary and unacceptable in a modern world. SOA was the answer and no one was arguing its value. Lots of work was done and some of today’s biggest enterprises run on a SOA infrastructure that is core to their business.
The Great Recession
But SOA never reached its potential within many other organizations. The Great Recession slashed IT budgets and companies hunkered down in survival mode. Costly and risky integration limitations became acceptable in a climate of bigger and worse fears and the concept fell from favor. After all, there was no budget for strategic investment like SOA when the goal was to just keep the company alive for the next twelve months.
The Great Integration
That brings us to today. The value of SOA never went away…we just stopped thinking of integration in terms of services because we were too busy babbling about big data, cloud, mobile, social and API management. While vendors are busy touting solutions for all of those hyped terms via applications you can buy, we stopped talking about the underlying architecture that makes those applications actually work.
Just as the Great Recession slashed budgets and turned back the strategy clock, the Great Integration is just starting to gain momentum as companies need to not only play catch up with infrastructure technology, but deal with the rise of new and more dispersed sources of data than ever before. It’s a new dawn for the concepts of SOA and integration led in large part by the following:
- Cloud applications create quick functionality gains, sure, but at the cost of more silo’d data than ever before (and harder to get to).
- Big Data stretches existing systems’ capabilities to manage (find, move, store, refresh) rapidly growing and increasingly complex data.
- API management is everyone’s best friend but that data still needs to move to and from systems of record somewhere.
- Mobile technology means threading the needle of a small device that has to serve data from anywhere to anywhere around the clock.
These are complicated problems that won’t get solved without a new approach that sounds a lot like an old approach that never was completed…SOA and integration. Without it, a real-time world is impossible and more applications, SaaS or on-premise, create accelerating data challenges.
We’re currently struggling with things like:
- Hospitals that aren’t integrated, especially in an era of Accountable Care
- Marketing that isn’t integrated, even though mobility and analytics changed the game
- Factories that aren’t integrated, even though they have CRM, ERP, MRP and more and are acquiring internationally
- Process that isn’t integrated, even though we’re quickly automating what used to be human work
Without integration, the future, faster, smarter, data-enriched analytics-driven world is a long way off. The timing is perfect for the Great Integration.
- A great read on The New Focus on Data Integration by David Linthicum
- Interconnectedness and The Enterprise Brain by Jim Harris
- Replacing the ETL model for Big Data at Edmunds in Big Data Applications Require New Thinking on Data Integration by Craig Stedman
- Enterprise Service Bus and the Cloud in Cloud Application Integration — Why a New Approach is Needed by Ashwin Viswanath
- A great overview in New Data Challenges Drive Need for Updated Integration Strategy by Loraine Lawson