By Hendrik Hamann
Five years ago, a few of my IBM Research colleagues and I played a hunch. Large-scale solar power was taking off, but we realized that for solar to fulfill its potential for helping to produce a more sustainable energy future, it would have to be integrated into electrical grids. For that to work, you would have to know ahead to time how much solar power would be generated when and where. That realization spawned our solar forecasting research project.
Today, we have shown that we can generate accurate forecasts of solar energy (from minutes ahead to many days ahead), which in turn can have a significant impact on the energy business – and on the future of sustainable energy. Our preliminary findings, including a test conducted at ISO-New England, the grid operator serving Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island and Vermont, show that our system can be 30 percent more accurate than other state-of-the-art approaches.
Through the US Department of Energy’s SunShot initiative, we’re now making our forecasts available to government agencies, utilities and grid operators in 48 states so they can evaluate how solar forecasting could affect supply and demand planning, as well as grid operations.
The forecast technology uses a combination of machine learning, big data analytics and mathematical modeling of complex weather systems. We developed it in collaboration with government, academic and industry partners. The system continuously monitors weather conditions including satellite observations, and analyzes that data to forecast how much solar energy will be available at different locations and times. One of the differentiating features of the IBM approach is that we incorporate a large number of weather and solar energy prediction models, which are then blended using historical data as a function of weather situation, forecast horizon and location to create a so called supermodel. Other solar forecasting systems instead take narrower location and timeframe views.
Our advances are important for the future of alternative energy. (We also use the same basic approach to produce wind and hydro forecasts.) But the implications are far broader. Our approach of combining information from domain models (in this case, weather models) with machine learning and big data analytics can be applied to many other domains – everything from structural health management of large infrastructure such as bridges to predict when maintenance will be needed, to understanding the workings of the human body to better administer medications.
Think of it as big data meets science. By bringing these elements together, people in a wide range of industries and professions will be able to understand better how the world works so they can make better decisions.
But let me take you back to the beginning…
Five years ago, my colleagues and I were eager to test some ideas about how we could provide better solar forecasts, but we needed some data. So, we developed a specialized sky camera system which projects the image of the sky in real-time on a video camera. This sky camera, which is good for short-term forecasting (5-10 minutes into the future), allows tracking clouds with respect to the position of the sun, as well as observing the formation and dissipation of clouds. The system had to be special because it requires unique optics (the original system used a projecting mirror, but we now use a fish eye lens) that can’t be saturated by sunlight – it’s like staring at the sun without squinting. In the photo below, the first generation camera is to the right, with the second generation to the far left, and the third generation in the middle.
The camera designs have come a long way since we built the first one. The camera is almost 10x smaller now, and to avoid saturation it uses electronics rather than a mechanical shutter. Because of their compact size, the new cameras can be readily deployed at a solar power plant, where is can measure the height of the clouds using triangulation techniques. This in turn helps to improve the forecasts further.
With these cameras, we capture thousands of high resolution images per day, which we feed to the system so it learns to identify different kinds of clouds and cloud behavior. We also provide the system with a stream of satellite images of the earth and countless weather forecast models. Our “secret sauce” is the way we categorize different weather and climate situations so we can learn why various weather models fail under certain circumstances. That helps us fine-tune our weather models for particular locations.
Because solar and wind energy production is intermittent and it has been largely impractical to store large amounts of electricity, it’s vital for utilities and grid operators to be able to accurately predict the amount of solar and wind energy that will be available. Armed with that information, they can efficiently manage their entire portfolio of energy sources, including coal, gas, wind, and solar.
Until now, most experts agree that because of intermittency, solar energy won’t supply more than 20 to 30 percent of the United States’ energy. However, there is good reason to believe that with better forecasts, it might be possible to push solar’s energy contribution up to 50 percent. As we continue to refine our system in collaboration with the DOE, we hope to double the accuracy of the system in the next year. That could have a huge impact on the energy industry – and on local businesses, the economy and the natural environment.
Physical science has been a big part of my life. So it’s gratifying to bring together my curiosity for understanding the physical world with two of the most exciting fields in computer science – data analytics and machine learning. I’m convinced that it’s at the intersection of scientific disciplines that the future will be invented.