A wise man once said, “What has been, will be again. What has been done, will be done again. There is nothing new under the sun.”
We’ve seen this before
The year was 1996 and the birth of the term Web 1.0 was ten years in the future. A new computer technology was just becoming mainstream, and business people were trying to make sense of it and how to put all the pieces together. By our standards, the technology was clunky and hard to use – single threaded CGI scripts, poor session management, limited integration, and development tools on the level of Notepad and FrontPage.
Developers with any depth of experience were rare, those with both depth and breadth of experience were rarer still. Few in large enterprises knew how to put all the pieces – web servers, application servers, email, databases, search – together in a way that delivered value. However, in just 4 short years virtually all this changed and web apps became ubiquitous because they were extremely easy to architect, develop and deploy.
Back to the future
The same rapid capability evolution is happening today with Big Data and the Hadoop ecosystem and platform. Today’s Big Data world looks very much like the web landscape of 1996: Somewhat immature tools (Hadoop only just released a 1.x version), few people broadly trained in the needed skills, fewer still with ability to put parts together or who have actually implemented successful projects, and few development tools to support rapid development, testing and deployment. But a very strong sense that the future is arriving.
The very scarcity of talent and tools confers real opportunity and a competitive advantage for those few who can deploy big data solutions right now. The wine recommendation engine that knows not just what the experts say, but what you liked in the past, the clever algorithm to promote customer loyalty and other solutions stand to pull ahead of their less ‘intelligent’ competitors.
This window won’t be open forever as big players emerge, small players are acquired or blow up, and big data becomes another piece of the technology stack. Until that happens, however, those who figure out how to deploy quickly and successfully and those who incorporate it into their model effectively will leap ahead of the competition.
What is your plan for Big Data?