Big Data is the greenest data of all

The following is a guest post. As Co-founder of Fabless Labs, Piyush is in his element while developing the next generation of truly open, open source analytics solutions.Piyush is likewise no stranger to the challenges of real-time and complex information management. For him, this is the perfect intersection of his passion for real-time decision-making and the rapid transformation taking place in the marketplace.

Green is the new black…or so you’d think from the incredible amount of focus paid on efficient energy production and consumption. With so much emphasis on building a green planet and increased government and utility spending on energy efficiency, one would expect a reduction in total energy spend in commercial and residential markets. It hasn’t happened.

Despite the recent push to implement energy efficiency programs, total energy consumption and average energy prices in both markets continues to rise.  Consumers continue to face higher energy bills because the average energy price is rising faster than the reduction in customer demand.  These indicators might make you think we’re going in the wrong direction, but it is actually too early to say all of this effort isn’t working.

Give it time

The reason for optimism? Smart meters and a variety of sensors across the energy supply chain are creating the ability to collect and analyze massive amounts of energy production, transmission and consumption data. The arrival of Hadoop and other Big Data tools makes it possible for analysis to keep up with rapidly increasing data volumes. All of that means nothing if it isn’t actionable. Let’s take a look at just a couple of ways that Big Data can be Green Data.

  • Forecasting demand – We’re slowly moving toward homes and businesses having smart meters that report actual consumption back to utilities and allow decisions on how to supply energy more efficiently. Right now, those meters provide data in 15-minute increments but down the road, we can increase the frequency of reporting as the ability to take and crunch data increases. When we know more, we make better energy supply decisions.
  • Conservation ‘signals’ – To make markets behave efficiently, there needs to be a way for energy use to change with availability. The most common way this is done is through price. Once energy providers can accurately forecast demand and in-the-moment use, pricing signals will cause consumers, individual or commercial, to lower usage. This proactive approach is mostly missing from today’s energy markets. When we know more, we make more efficient usage decisions.
  • Measuring efficiency – The US Department of Energy is currently developing the SEED database, meant to be a way to allow buildings to benchmark against other facilities. While that may not seem so hard, there are big factors that need to be part of any efficiency algorithm, like weather, the number of people, what types of machines are being operated and when. Once buildings have an ‘energy value’, decisions can be made on how much real estate is worth, where to retrofit and how to design new facilities. When we know more, we can make better design decisions.

Change needs to happen

This is all great and in theory, will make our planet better for everyone. There are still a few things, however, that stand in the way. There needs to be better standards for how information is collected, stored and shared so that energy supply chains can be better analyzed and operated optimally. We also need to make sure energy providers aren’t reaping excessive benefit from more efficient consumption without passing that benefit to energy consumers, which would ‘mute’ the signals that drive positive change. Balancing the right amount of regulation with allowing market forces to operate is an age-old challenge.

Assuming we’ll overcome these challenges, where things go is wide open. There are gamification possibilities (who’s the most green in the neighborhood/city/region?) and countless other strategies on the horizon. There’s no doubt Big Data will be driving us toward a greener planet.


Tags: , , ,

Categories: Data Analytics / Big Data, Energy, Gamification

Author:Chris Taylor

Reimagining the way work is done through big data, analytics, and event processing. There's no end to what we can change and improve. I wear myself out...

Subscribe to the blog

Subscribe and receive an email when new articles are published

3 Comments on “Big Data is the greenest data of all”

  1. June 9, 2012 at 1:42 pm #

    Piyush, Great post. You hit the nail on the head about demand management through smart meters that signal real time demand. We used to do marginal cost based rate design for many large North American utilities. The economics of this was very well understood – that you could reach the best economic outcomes by pricing at the marginal cost. We did a lot of modeling, expert testimony and rate design around this. However, there were two elements that were missing at the time to make this a truly robust approach:

    1) The ability to do this analysis in real time. Basically, what’s been called the 2 Second Advantage – knowing the right information in the right context at the right time is often much more valuable than all the analysis and data in the world after it’s too late to act.

    2) The ability to signal those marginal costs as prices (again in real time). For marginal cost pricing to work, those prices have to be able to act as signals. In the past there was no way to communicate these real time signals. With mobile technology and the growth of connectivity, this technological constraint no longer exists.

    I look to energy space, especially electricity, to be one of those industries likely to be fundamentally changed by Big Data.


  1. Buggy whips and newspapers | Successful Workplace - June 17, 2012

    […] Energy – Forecasting, conservation and efficiency will be increasingly driven by data […]

  2. Shopping in the Big Data marketplace | Successful Workplace - December 8, 2012

    […] With so much attention on making our lives more ‘green’, Big Data offers the chance to solve energy problems in all new […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: