Between the Poles

Applying machine learning to identifying housing starts from satellite imagery

Deep learning is increasingly finding practical applications using geospatial data, specifically, satellite imagery. At the GeoIgnite conference in Ottawa, Nicolas Martinez of Statistics Canada described a project scheduled to kick-off in July to use machine learning to identify housing starts across Canada.  The goal of the project is to improve the accuracy and coverage of housing starts survey, specifically to fill data gaps for smaller remote and indiginous communities which are generally excluded from the current survey program. 

Canada Mortgage and Housing Corporation (CMHC) currently employs a significant number of people at their head office in addition to field agents across the country who physically confirm residential starts and completions.  The objective of the project is to apply convolutional neural network (CNN) machine learning to satellite imagery to extract information about housing starts and completions. The imagery will be preprocessed to prepare it to be used in developing a machine learning processing model.  The largest challenge will be developing training and testing imagery data sets.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded.  More recent applications of satellite imagery targeted landuse and agricultural identification. For example, remote sensed imagery from a satellite can be used to create a model that will differentiate between corn and potatoes, using factors that can be calculated from satellite imagery such as normalized difference vegetation index (NDVI) (min, max, mean), texture (min, max, mean), vegetation height, and geometric factors such as orientation.  I blogged about how open source code and publicly available training data has been applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil. A study in the Netherlands used CNN to identify blocked waterways using overflight imagery with a success rate of 97 %.

An important application of satellite imagery is identifying built structures. I have blogged about how satellite imagery has been used to identify buildings and transportation networks by applying a neural network model together with OpenStreetMap layers and high resolution Worldview multispectral imagery.  NVIDIA has demonstrated the ability to automate detection of many road networks using deep learning algorithms and multi-spectral high resolution imagery.  Another application of CNN and imagery in the Netherlands was able to differentiate residential roofs with dormer windows from those without.

The essential key to effective application of deep learning is good training and test data - ground-truthed data that involves, for example, someone on the ground identifying whether the fields seen in the imagery are corn or potatoes or something else. There are  publicly available training data sets that can be used to train satellite imagery CNN models. For example, I blogged about the System for Terrrestrial Ecosystem Parameterization (STEP) dataset which has 2000 manually labeled sites covering 17 different land cover types scattered across all continents. A large publicly available training dataset derived from Sentinel 2 imagery contains 30,000 polygons of land use training data for ten classes of land use: annual crop, forest, herbaceous vegetation, highway, industrial, pastures, permanent crop, residential, river and sea or lake.

The Statistics Canada project is scheduled to begin in July with initial work on a proof of concept and CNN testing to begin in August and September.  The communities chosen for the initial phase are  Kitchener-Cambridge-Waterloo, Red Deer, and Iqaliut.  The first phase is scheduled to be completed by winter 2019.  The project plans to use Tensorflow and Keras, open source libraries and a python framework for applying machine learning.  Artificial intelligence (AI) has often suffered from inflated expectations, complex code, heavy processing requirements and not very practical applications. There is a growing body of practical applications in the geospatial domain that shows that as a result of the immense processing power available in the cloud machine learning is able to generate fairly easily practical and useful results.  As Chris Holmes pointed out in his talk at FOSS4GNA, the challenge for Statistics Canda will be to develop reliable and publicly available training and test datasets to enable deep learning models to be created for a broader range of applications using the huge volumes of satellite imagery and other geospatial data that are now available.

Deep learning is increasingly finding practical applications using geospatial data, specifically, satellite imagery. At the GeoIgnite conference in Ottawa, Nicolas Martinez of Statistics Canada described a project scheduled to kick-off in July to use machine learning to identify housing starts across Canada.  The goal of the project is to improve the accuracy and coverage of housing starts survey, specifically to fill data gaps for smaller remote and indiginous communities which are generally excluded from the current survey program. 

Canada Mortgage and Housing Corporation (CMHC) currently employs a significant number of people at their head office in addition to field agents across the country who physically confirm residential starts and completions.  The objective of the project is to apply convolutional neural network (CNN) machine learning to satellite imagery to extract information about housing starts and completions. The imagery will be preprocessed to prepare it to be used in developing a machine learning processing model.  The largest challenge will be developing training and testing imagery data sets.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded.  More recent applications of satellite imagery targeted landuse and agricultural identification. For example, remote sensed imagery from a satellite can be used to create a model that will differentiate between corn and potatoes, using factors that can be calculated from satellite imagery such as normalized difference vegetation index (NDVI) (min, max, mean), texture (min, max, mean), vegetation height, and geometric factors such as orientation.  I blogged about how open source code and publicly available training data has been applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil. A study in the Netherlands used CNN to identify blocked waterways using overflight imagery with a success rate of 97 %.

An important application of satellite imagery is identifying built structures. I have blogged about how satellite imagery has been used to identify buildings and transportation networks by applying a neural network model together with OpenStreetMap layers and high resolution Worldview multispectral imagery.  NVIDIA has demonstrated the ability to automate detection of many road networks using deep learning algorithms and multi-spectral high resolution imagery.  Another application of CNN and imagery in the Netherlands was able to differentiate residential roofs with dormer windows from those without.

The essential key to effective application of deep learning is good training and test data - ground-truthed data that involves, for example, someone on the ground identifying whether the fields seen in the imagery are corn or potatoes or something else. There are  publicly available training data sets that can be used to train satellite imagery CNN models. For example, I blogged about the System for Terrrestrial Ecosystem Parameterization (STEP) dataset which has 2000 manually labeled sites covering 17 different land cover types scattered across all continents. A large publicly available training dataset derived from Sentinel 2 imagery contains 30,000 polygons of land use training data for ten classes of land use: annual crop, forest, herbaceous vegetation, highway, industrial, pastures, permanent crop, residential, river and sea or lake.

The Statistics Canada project is scheduled to begin in July with initial work on a proof of concept and CNN testing to begin in August and September.  The communities chosen for the initial phase are  Kitchener-Cambridge-Waterloo, Red Deer, and Iqaliut.  The first phase is scheduled to be completed by winter 2019.  The project plans to use Tensorflow and Keras, open source libraries and a python framework for applying machine learning.  Artificial intelligence (AI) has often suffered from inflated expectations, complex code, heavy processing requirements and not very practical applications. There is a growing body of practical applications in the geospatial domain that shows that as a result of the immense processing power available in the cloud machine learning is able to generate fairly easily practical and useful results.  As Chris Holmes pointed out in his talk at FOSS4GNA, the challenge for Statistics Canda will be to develop reliable and publicly available training and test datasets to enable deep learning models to be created for a broader range of applications using the huge volumes of satellite imagery and other geospatial data that are now available.

A killer application for satellite imagery collected daily for the entire earth

At the GeoIgnite conference in Ottawa, Wade Larsen of Urthecast described what may be a killer application for satellite imagery captured daily. Farmers in the US over-irrigate by 25% per year. A recent study has demonstrated that monitoring vegetation growth on a daily basis can save farmers $50 000 in water per year. With 260,000 pivots operating in the US, this amounts to billions of dollars of savings every year.

Urthecast has been known for having two cameras (Iris and Theia) on the International Space Station and two Deimos earth observation satellites. Last year Urthecast announced the acquisition of Geosys Technology, a software company that processes earth observation imagery for agricultural applications.  Their latest initiative is called UrtheDaily with the objective of capturing nine bands of imagery from six satellites covering the Earth's entire landmass every day. The nine bands are identical to Sentinel-2's visual and near-infrared bands. The cameras are cross-calibrated with Sentinel-2 to assure data compatibility. The six cameras enable a daily revisit for higher latitudes and somewhat less for tropical areas. It captures imagery at the same time, from the same altitude every day.  About 150 terabytes will be acquired and downloaded every day. The imagery has five meter resolution (GSD) and represents scientific grade data quality that will be available on the cloud for user access within hours of acquisition. UrtheDaily is designed to support machine-learning and AI ready geo-analytics applications on a global scale. The UrtheDaily platform will provide the ability to discover data, run user algorithms, and deliver data to customers.

UrtheCast reports that it has already signed contracts primarily in the agriculture sector. An example of a potential killer app in agriculture is monitoring irrigation. Farmers in the U.S. overwater by 25 %. A four year in Texas by HydroBio which assessed the value of frequent monitoring of plant health and growth using satellite data concluded that to be effective this required high-frequency (daily) monitoring with highly calibrated sensors. With its resolution UrtheDaily will be able to monitor vegetation for individual pivots on a daily basis in order to optimize water utilization. It is estimated that this could save farmers $50,000 per pivot per year. (A pivot typically irrigates 120 acres.) There are 260,000 pivot irrigation systems in the U.S. which translates into the potential to save farmers billions of dollars per year.

To date UrtheCast reports that it has signed $350 million worth of contracts in the U.S., Russia, India and other countries. It plans to launch six satellites in 2021 and become operational in 2022.

Automating transmission line inspections with BVLOS drones and machine learning

Transmission line inspections for vegetation management and other purposes are essential in ensuring grid reliability and resilence.  They are generally performed by manned helicopters or by a ground crew.  Data is collected with cameras and analyzed to detect diseased trees that could fall and hit a powerline (fall ins) or trees that were encroaching on powerlines (grow ins).  These inspections are mandated by NERC in North America and are not optional.  Beyond visual line of sight (BVLOS) drones (UAVs) and software analytics using machine learning has made it possible to approach completely automating this expensive process.  Automating transmission line inspections for vegetation management using BVLOS drones not only saves money but also could improve the resiliency and reliability of the transmission grid.

Mapping transmission lines with BVLOS UAVs

At ILMF 2019 in a very interesting talk Paolo Guzzini of Delair described how BVLOS drones has been deployed to map transmission lines in remote areas of Iceland.  Delair conducted two surveys on behalf of EFLA Consulting Engineers, a general engineering and consulting company. EFLA needed to map existing power lines and survey an area where Landsnet, the main transmission system operator in Iceland, planned to build a new power line between two cities. Iceland's airspace regulations allows BVLOS UAVs to be used in areas that are not densely populated. Delair deployed fixed-wing drones to gather two types of data: photography and LiDAR.  The drones were programmed to fly a predetermined flight plan so the pilot had to only monitor the drone's progress and be available for emergencies.  Delair’s drones were able to capture comprehensive data along the planned route and deliver the processed results to the client the next day — all at a fraction of the cost of hiring a helicopter or ground crew.  UAVs with a range of 110-120 minutes flew for up to 10 km per flight. In one case the UAV flew a total of 70 km with a 21.4 megapixel photo camera to capture a corridor of 200 m width to an accuracy of 0.5 m vertical and 0.2 m horizontal accuracy. The actual vertical accuracy achieved was 0.15 m.  In a second mission a UAV flew at 100 m a corridor 10 km in length with a LiDAR camera to capture a corridor 200 m wide with the objective of vertical accuracy of 0.5 m and horizontal accuracy of 0.2 m. The actual accuracy achieved was 4 cm horizontally and 2 cm vertically. The advantage of using LiDAR is that it can achieve better accuracy in location especially in the vertical dimension. In addition the LiDAR approach was able to detect detect conductors and poles.  In both cases the range of the flights was limited by weather conditions, radio range (15 km), and weight of the camera payloads.

Reality capture for vegetation management of transmission lines with BVLOS UAVs

Periodic transmission line inspection is mandated for North American transmission lines by NERC.  Inspections are currently carried out by manned helicopter or ground crews which are expensive and for that reason not done more often than is required by NERC.  Full automation, which is possible with BVLOS UAVs and automated feature extraction of lines, pylons, vegetation and ground, would dramatically reduce the cost of inspection and enable more frequent flyovers.  The major benefits would be lower costs and a reduction in the number and length of outages resulting from vegetation transmission line interactions.

In the U.S. on July 15, 2016, President Obama signed the FAA Extension, Safety, and Security Act of 2016. The 2016 Act requires the FAA to establish a process to allow certain UAV operations related to utilities, pipelines, and oil and gas production to be conducted beyond the visual line of sight (BVLOS) of the operator and either in the daytime or nighttime.  As a result FAA rules permit beyond visual line of sight (BVLOS) UAVs through Part 107 waivers.  I blogged over a year ago about Xcel Energy which obtained one of the first three waiver to fly UAVs for beyond visual line of sight operations.   Paolo related that 32 of these waivers have been granted and that more than 20 companies are operating beyond BVLOS drones but still with some restrictions.  Paolo related that the FAA very recently began allowing BVLOS operation using radar to track the location of the UAV.  Paolo expects to see many long-distance UAV operations in the near future, making drones fully competitive against helicopters.

Over 20 countries have regulations that permit BVLOS UAV flights. In some areas of the world, Delair has even been able to control flights via a 3G data connection, allowing a drone to inspect up to 30 miles in a single flight.

Post-processing

Paolo told me that Delair's post processing software uses a combination of machine learning and other technologies to classify LiDAR point clouds with 85 to 90 % reliability in order to identify potential vegetation encroachments.  I have blogged about Enview and other companies that also offer software solutions for this problem.

Conclusion

Rapidly evolving regulation to permit BVLOS operations, Delair's successful deployment of BVLOS drones with radio or 3G communications and automated post-processing, often cloud-based, to identify potential vegetation encroachment indicate that we are on the cusp of being able to completely automate this type of transmission line inspection.  Using drones and automated feature extraction software not only dramatically reduces the cost of these inspections but makes possible more frequent inspections including on-demand - for example immediately after storms - which would impact positively the reliability and resilience of the transmission grid.

Automating vegetation management for electric power transmission lines using deep learning with LIDAR

With FAA rules permitting beyond visual line of sight (BVLOS) UAVs available through Part 107 waivers the cost of capturing the data required for vegetation management for power lines, whether transmission or distribution, could drop dramatically.  However, manual post processing of the imagery to identify areas of vegetation encroachment, grow-ins and fall-ins, remains an error prone, tedious and time consuming process.  I have come across several firms (for example EPRI, SiteSee, Trimble RealWorks) over the years that have been working towards automating the process for identifying vegetation encroachments including grow-ins and fall-ins, but I have not encountered a utility that is using this technology operationally.  At Distributech 2019 in New Orleans I had a chance to chat with a startup Enview, based in San Francisco,  who described to me how their software has been implemented  over 7,000 miles of transmission lines last year to automate the identification of vegetation encroachment from LiDAR point clouds.  Furthermore, they expect the total distance covered to double this year.

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that utilities even have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment which typically are carried out by manned helicopters.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. I have blogged about Energisa in Brazil who used satellite stereo imagery to prioritize and optimize tree pruning and ComEd's pilot project in Chicago that uses LiDAR imagery to not only prioritize and optimize but also to change the pricing structure for procuring distribution tree trimming.

Enview applies machine learning, computer vision, and geospatial big data analytics to highly reliable feature extraction from LiDAR point clouds to identify the exact location, height and extent of trees and then to determine proximity to power lines and other utility equipment.  Utility GIS data can be integrated and results visualized using colour-coding to identify the level of risk for each tree.  This enables the utility to prioritize their tree trimming and optimize contractor routing.  In environmentally sensitive areas the visualization can be used to inform discussions with the various stakeholders responsible for different land uses.

At Distributech I had a chance to chat with Krassimir Piperkov, COO at Enview, who described how this technology has been deployed on more than 7,000 miles of infrastructure last year, and is on track to be deployed on over twice that much this year. He gave a very recent example of annual vegetation management analytics performed on 230kV transmission lines in California. These lines run through suburban and environmentally-sensitive, heavily forested areas.

3D visualization of a 230kV span (blue lines) with potential grow-in (yellow) and fall-in (red) vegetation that falls within client-determined clearance boundaries.

LIDAR data was acquired by a third party through a helicopter-mounted Teledyne Optech LiDAR sensor package. Four band imagery was co-collected during the LIDAR survey and registered into an orthomosaic of RGB and NIR bands. The data collection generated almost 2GB per linear mile. The Enview data analytics engine processed this data to (i) first classify the massive amounts of LiDAR data and (ii) then automatically calculate potential grow-in and fall-in vegetation points with high speed and accuracy. Insights and data were delivered to the operator not only via GIS-compatible file format for incorporation into existing workflows but also via an intuitive web portal accessible across silos. As a result of the superior accuracy and speed, the client was able to confidently prioritize vegetation prescription work to the areas that need it most and do so much sooner than otherwise possible with traditional methods.

The results being achieved with Enview's technology suggest that we are on the cusp of the widespread adoption and application of deep learning technology to automate specific tasks. The combination of UAVs for data collection and machine learning for feature extraction dramatically reduces the cost of inspections for utilities. This would permit them to be done more frequently than with manned aircraft and would lead to improvement in the reliability of above ground electric power and communications networks.

Machine learning cost-efficiently creates high quality utility GIS data for smart grid applications

Many utilities' GIS contains only a small part of what is required for managing a smart grid.  Often even the data in the GIS is too unreliable to use for smart grid applications.  For example, utility GISs may not include which customers are connected to which transformer, or if they do include this information it is incomplete or incorrect.  Many utility GISs cannot identify the phase(s) each customer is using and they may not include network topology - what is connected to what.  Other missing data may includes street lights, distributed energy generation such as rooftop PV panels, the type and condition of poles, joint use of poles, and encroachment.  To correct these problems typically requires a foot survey which is expensive.  For example, a pole inventory costs at least $20 per pole.  At the Open Geospatial Consortium (OGC) Energy Summit at EPRI in Charlotte, John Simmins of EPRI described the research he is leading at EPRI to use modern technology including machine learning to reduce this cost to pennies per pole using available imagery such as Google Streetview.

I began working in the utility and telecom sector in 1993, specifically in developing software for records management (called network documentation outside of North America), and at that time the magnitude of the AEC+geospatial cultural mismatch was just beginning to be appreciated by utilities and telecoms. Planners tend to use GIS tools, engineers and designers CAD tools, construction folks in the field paper CAD drawings, and asset managers FM, GIS or integrated tools. Fashioning AEC and geospatial data into an efficient data flow from planning through design and construction to operations and maintenance represented a challenge that remains a problem for utilities.  For many utilities the backlog of as-builts and updates waiting to be entered into the GIS stretches into months, with the result that GIS data is permanently out of date inhibiting management and field staff from relying on this data operationally.  The Between The Poles blog is over ten years old and one of the persistent themes from its very beginning in 2006 was the challenge of integrating CAD and GIS data in a common workflow (some examples; 2006, 2007, 2008 ).

There are signs that modern technology is beginning to offer a solution to the problem of developing and maintaining an accurate digital model of electric power grids.

I wrote recently about a utility in western Pennsylvania that intends to implement an Advanced Distribution Systems ( ), but realized that to this requires a network data model. Duquesne Light is typical of many North American small to medium electric power utilities.  It does not have an outage management system or an electrical data model which means it is difficult to connect customers to transformers and to determine upstream devices when customers report power problems.  Currently Duquesne Light's circuit maps are AutoCAD drawings.  Duquesne Light is moving to modernize its grid and plans to deploy an Advanced Distribution Management Systems (ADMS) including outage management (OMS), DMS and DSCADA.  But the ADMS will require developing a complete distribution electrical data model and to that in the traditional way can be expensive.

To quickly, safely and cost efficiently develop a network data model, Duquesne Light looked at several alternatives including reality capture technology.  A pickup truck was equipped with with side mounted photo and IR cameras and LiDAR laser scanners. The reality capture survey could be completed rapidly and safely, with no boots on the pavement  Actual development of the data model was completed in the office without the necessity of going to the field for quality checks because the imagery provided all the data that was required to create and verify the data model.  Duquesne Light found several significant advantages of the reality capture approach.  The first was asset accuracy - both location and asset attributes.  In addition the captured imagery could also be used for other purposes.  For example, it could be used for a joint-use audit.

But this is still basically a manual process requiring a human being to scan images.  At the Energy Summit John Simmins described his research using neural network technology to automatically identify poles and attached equipment from available imagery such as Google Streetview.  Normally applying neural networks requires large quantities of training data.  To reduce this requirement, John has evaluated "biologically inspired", forward pass, neural networks.  In this approach the algorithm first looks (foveates) for vertical things that have the characteristics of a pole, and then for the characteristics of equipment attached to the pole, for example, cross members, street lights, transformers and large cables.  From this analysis is possible to determine pole location, height and width, cross member position on the pole and projected length, street light support strut position and length, position of transformer and large cable connection locations.  The ultimate goal of the research is in addition to automatically detecting each element of equipment to identify the type, size and manufacturer of each piece.

This is not a simple problem and there is no silver bullet.  Using a standard neural network approach requires large volumes of training data.  For poles it requires images of poles with different types of damage under different lighting conditions and in different types of weather.  An Australian company SiteSee claims to be able to do this for communications towers and transmission pylons - identifying the type and manufacturer of each antenna or insulator on a tower or pylon and comparing it to the manufacturer's specification to detect signs of corrosion and other types of damage.  When what you are looking for has a distinctive geometry that differentiates it from the surroundings, a biologically inspired approach may provide a way of identifying equipment without large volumes of training data.  Another area where several different approaches have been applied to automate feature extraction is electric power transmission lines for vegetation management.

Creating a digital twin of an electric grid is essential for developing and maintaining a smart grid.  Utilities have been wrestling with this challenge for decades and in many cases their GIS data is still not much closer to what is required.  If it would be possible to use the technology that John Simmins is investigating or other technology to reduce the cost of improving the quality of their GIS data, it would accelerate the adoption of smart grid technology at many utilities.  In his presentation at the Energy Summit Jared Green described his research in using smart meter and AMI network data to map secondaries from the transformer to the home, identify sources of unmetered load, and use impedance data from smart devices to correct utility GIS data.  Given the history of the challenge to develop and maintain accurate GIS data at many utilities, these approaches represent a beacon of hope which may enable utilities to cost effectively develop a digital twin of their networks that is an essential prerequisite for a  smart grid.

EPRI curates open utility-related training datasets for deep learning applications

Automatic feature extraction or image classification using artificial intelligence (AI) algorithms has been successfully applied in several domains, for example, in recognizing landmarks, dogs, cars and other objects in consumer photos and detecting building footprints in high resolution satellite imagery.  The existing algorithms for doing this are often available in open source implementations and have been democratized to the point where they are available to anyone with a modicum of programming skill.  The challenge is creating and maintaining (curating) the training datasets that are required to turn the basic algorithms into models that can automatically recognize and classify different types of objects.  At the December Open Geospatial Consortium (OGC) Energy Summit at EPRI in Charlotte, Andrew Phillips of EPRI described EPRI's role in the new world of big data which includes curating open training datasets for the electric power industry to be used by the AI industry including universities, startups and established software players to create applications for electric power utilities.

Deep learning is increasingly finding practical applications using different types of georeferenced imagery.  For example, convolutional neural networks (CNN) have been successfully applied to real world problems including detecting blocked canals, differentiating different types of roofs, identifying building footprints and recognizing transportation networks.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded. The basic process in creating these models is to find or create some training data, some data to test the model, and then play with the parameters to optimize the model for the test data. Google is pretty good at recognizing dogs, cars and other objects that are commonly found in consumer photos.  But if you present Google with photos of utility equipment such as a photo of a decayed wooden pole with an insulator precariously hanging off it or a cracked fuse, Google will not recognize these objects correctly.  It is not a problem with Google's algorithm, but a lack of training data - photos whose content has been "labeled" by electric power experts.

An essential piece that is required to open up the data and technology of deep learning to a broader audience is open labeled data, what has been called an "open labeled data commons". There are analogies with OpenStreetMap initiative - something like this is required for labeled data - it needs a repository of open data, tools to capture and manage the data, and a community around this to capture and curate that data.  For example, for automatically classifying land use in satellite imagery the System for Terrrestrial Ecosystem Parameterization (STEP) has 2000 manually labeled sites covering 17 different land cover types that can be used a training data for deep learning applications to land use.

EPRI has already been active in collecting large volumes of training data from members and curating training datasets that can be used for common industry use cases.   For example,  EPRI has pulled together dissolved gases data for 45,000 transformers from 27 utilities which includes a few thousand failures.  The data has been provided to AI vendors to train their models.  Some of the data has been held back to test the models to see how well the vendors have done.

In another experiment EPRI asked 12 experts to review inspection imagery to identify and rank problems with common electric power equipment.  Thousands of labeled images were provided to nine vendors including IBM, Intel, Harris and others to train their deep learning models.   EPRI held back 500 of these images for testing and have just completed running them through an evaluation process to see how well the models do in identifying problems.  For example, it was found that the best models could identify cracked and damaged insulators with 88% confidence (using Matthews correlation coefficient), but cavities in wood poles such as woodpecker holes could only be identified at the 50% confidence level.  The results have revealed more images are needed under different lighting conditions. EPRI is going back to its members to get more images. Another area of interest to EPRI is monitoring transmission lines including vegetation encroachment using imagery from BVLOS UAV flights. 

EPRI's strategy is to be the bridge to the AI community by making the AI community aware of use cases that are important for the electtric power industry, helping the AI community with the physics of electric power and supporting structures, helping its members understand how AI can be used in the electric power industry, and collecting and curating open training datasets to be used by the AI community,

Application of geospatial deep learning to real world problems accelerating

Deep learning is increasingly finding practical applications using geospatial data, specifically, satellite imagery. At HxGNLIve in Las Vegas, Wim Bozelie, Technology Director at Imagem, described some very practical cases in which machine learning, specifically, convolutional neural networks (CNN) were successfully applied to real world problems including detecting blocked canals and differentiating different types of roofs.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded. The basic process to create these models is to find or create some training data, some data to test the model, and then play with the parameters to optimize the model for the test data. There is a long-running competition Imagenet Large-scale Visual Recognition Challenge (ILSVRC) which provides training and test data for evaluating algorithms for object detection and image classification at large scale. A motivation for this public challenge is to allow researchers to compare progress in detection across a wider variety of objects by providing publicly available, relatively expensive data. The validation and test data consist of 150,000 photographs, collected from flickr and other sources of imagery, which have been hand labeled with the presence or absence of 1000 object categories. The object categories are very detailed for the types of objects found on facebook, flicker and othe social media. Some examples,

dial telephone, cell phone, airliner, warplane, airship, balloon, space shuttle,aircraft carrier, Tibetan mastiff, Doberman pinscher, warthog, Arabian camel, dromedary, golden retriever, border collie, boxer, english springer, malamute, malemute, whippet, weimaraner, husky, brittany spaniel, lion, beaver, irish setter...

This training data is not much use for geographic data where typical applications of satellite imagery are for landuse and agricultural identification. Wim talked specifically about a type of deep learning called convolutional neural networks (CNN). For example, remote sensed imagery from a satellite can be used to create a model that will differentiate between corn and potatoes, using factors that can be calculated from satellite imagery such as normalized difference vegetation index (NDVI) (min, max, mean), texture (min, max, mean), vegetation height, and geometric factors such as orientation.  I blogged about how open source code and publicly available training data has been applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil. Wim reported a study in the Netherlands using CNN to identify blocked waterways using overflight imagery. An approach that did not use machine learning was tried first, but it found many false positives and missed some real instances of blocked waterways and the success rate was about 50 %. When machine learning was applied with reliable training data, a success rate to 97 % was achieved.

An important application of satellite imagery is identifying built structures. I have blogged about a competition to use satellite imagery to identify buildings and transportation networks. The winner applied a deep neural network model developed originally for medical image segmentation called U-Net.  Reliable training data was provided by the competition organizers. The winning solution used OpenStreetMap layers and high resolution Worldview multispectral layers as the input of the deep neural network algorithm.  Using the same data NVIDIA demonstrated the ability to automate detection of many road networks using deep learning algorithms and multi-spectral high resolution imagery.  Wim described another application of CNN and imagery in the Netherlands to differentiate residential roofs with dormer windows from those without and discussed how different aspects of the model, such as convolution layers, can be manipulated to improve results.

The essential key to effective application of deep learning is good training and test data - ground-truthed data that involves, for example, someone on the ground identifying whether the fields seen in the imagery are corn or potatoes or something else. There are  publicly available training data sets that can be used to train satellite imagery CNN models. For example, I blogged about the System for Terrrestrial Ecosystem Parameterization (STEP) dataset which has 2000 manually labeled sites covering 17 different land cover types scattered across all continents. Wim described a large publicly available training dataset for Sentinel 2 imagery, which was developed by Eurosat. It contains 30,000 polygons of landuse training data for ten classes of landuse: annual crop, forest, herbaceous vegetation, highway, industrial, pastures, permanent crop, residential, river and sea or lake. In an experiment Wim took 500 training polygons from this dataset for three types of landuse: built-up, pastures and vegetation, and water, to build a CNN model. He applied the resulting model to 25 test areas. Of the 25 test areas the model correctly identified 23, but two were incorrect.

AI has often suffered from inflated expectations, complex code, heavy processing requirements and not very practical applications. There is a growing body of practical applications in the geospatial domain that shows that as a result of the immense processing power available in the cloud machine learning is able to generate fairly easily practical and useful results.  As Chris Holmes pointed out in his talk at FOSS4GNA, the challenge now is to develop reliable and publicly available training and test datasets to enable deep learning models to be created for a broader range of applications using the huge volumes of geospatial data that are now available.

Using machine learning to identify real emergency alarm events

In a very interesting presentation at HxGNLIve in Las Vegas, Chris Klimmm, a subject matter expert with Hexagon Safety and Infrastructure, reported using Microsoft Azure's machine learning capability to develop a model to winnow out likely real burglar alarm events from the 97% of false alarms.

I have blogged about a number of applications involving machine learning.  There are tools on the major cloud service providers for creating and running models.  The ones on Microsoft Azure in particular appear to be relatively easy to use.  The challenge is finding training and testing data.  In this case Hexagon has a mobile response application Called InService that is used to respond to burglar alarm events.  Working with their customers provided validated training data that could be used to train and test a machine learing model which HexagonSI reportedly did quite successfully.

An interesting application of this approach for utilities would be to a social media stream such as twitter to help utilities identify valid tweets that were evidence of outages or other utility-related issue using spatial proximity and keywords to identify candidate tweets.

 

IoT platform for integrating two decades of Hexagon's technology acquisitions announced at HxGNLIve

In May of last year Hexagon acquired a company based in Alpharetta, Georgia called Catavolt that combines internet of things (IoT), analytics and mobility. At HxGNLIve in Las Vegas Ola Rollén in his exhilarating opening keynote - beginning with incredible graphics showing every foot of a young gosling's 400 foot fall down a cliff (which happily it survived) and ending with an incredible slalom ahead of an avalanche punctuated with the skier executing a back flip - announced its new name Xalt and showed a unique example of what it is capable of.  Xalt runs in the cloud on Amazon Web Services and collects data from sensors, which could be a metrology (measuring) device, an Oracle database or SAP. It then analyzes the input stream looking for outliers and raises an alarm that causes something else to happen. It could be informing a human that there is a problem or it could be linking to another machine. In the example that Ola showed a measuring device could be seen measuring the precision of what a robot did and when the precision became worse than a threshold, it raised an alert which caused the robot to recalibrate itself.  Another example of what the Xalt platform is capable of was demonstrated on the exhibition floor where we were invited to develop an app in seven minutes that integrated data from several different sources. The development platform was a laptop and the resulting app ran on an iPad.

Xalt is an internet of things platform supporting many different sensors and types of sensors, big data analytics and different clients especially smart phones. Xalt supports multiple AI data sources including imagery, video and big data for applications such as predictive maintenance, change and anomaly detection. The platform monitors, senses and acts on all connected data inputs in real time. It accesses sensors from remote locations and utilizes the data to optimize business processes. It includes AI capabilities including deep learning with digital neural networks. From Ola's presentation it is clear it is not intended as a standalone product but rather as a platform that integrates the many and disparate devices and systems that Hexagon has acquired since Hexagon began its incredible journey in 2000 from a Swedish contract metals manufacturer and producer of valves, antennas and pressure vessels to a global geolocation and measurement, design and construction application and visualization technology company with $4.11 billion in revenue and 18,315 employees in 2017. During these nearly two decades Hexagon acquired 64 companies many with geospatial capabilities including Leica Geosystems in 2005 and Intergraph in 2010. The big question for observers of Hexagon's meteoric rise has been how was it to integrate these disparate acquisitions into a synergistic whole that is bigger than its component parts.  It is clear from Ola's keynote that Xalt will be the foundation for Hexagon's big leap to achieve this.

Calculating forest loss using open source deep learning tools and publicly available training data

Deep learning algorithms have been developed by academia and as a result the code for the most part is open source but the successful training of deep networks requires thousands of labeled training samples and at the present time this training data is typically not open. In their presentation at this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis Jason Brown and Courtney Whalen, data scientists at astraea, showed how deep learning using publicly available labeled data for training was applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil.

Chris Holmes of Planet Labs, in his insightful talk at FOSS4GNA in St Louis about the application of deep learning to geospatial data, identified a challenge in making this technology open. The deep learning algorithms have been developed by academia and as a result the code for the most part is open source. For example, a deep neural network model developed originally for medical image segmentation called U-Net is open source and has been applied to identifying building footprints. Successful training of deep networks requires thousands of labeled training samples. Labeled data involves people on the ground manually ground-truthing land use types and other features so that the deep learning algorithms can learn what to recognize. At the present time this training data is typically not open source. In this presentation by Jason Brown and Courtney Whalen, both data scientists at astraea, deep learning using publicly available labeled data for training was used to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil.

This is computationally intensive and a distributed engine was used. The computation engine used open source components. Spark is a top level Apache project which enables distributed processing for global scale computation. RasterFrames is a free and open source toolkit allowing scientists, data scientists, and software developers to process and analyze geo

Forest cover in Mato Grosso in 2002

patial-temporal raster data with the same flexibility and ease as any other data type in Spark DataFrames. This is a LocationTech raster project and is built on GeoTrellis. Using this software each year required 6 to 7 hours of computation using 48 cores.

The imagery that was used was captured by the MODIS satellite for the years 2001 through 2017. MODIS monitors the reflection back from ground cover for several bands including red, green, blue, short wave infrared and near infrared. Its cameras have a spatial resolution of 500 by 500 meters and a revisit rate of one to two days. From the bands that it captures the normalized difference vegetation index (NDVI) can be calculated. From the data monthly means and yearly aggregates can be calculated.

Forest cover in Mato Grosso in 2011

The training data used came from the System for Terrrestrial Ecosystem Parameterization (STEP) which has 2000 manually labeled sites covering 17 different land cover types including five forest types scattered across all continents. The model was trained on MODIS 2012 data. 80% of the data was used for training. After training was completed the remaining 20% was used to test the model.

Comparison of rate of forest loss with Global Forest Watch for 2001 to 2016

After training and testing the first application was to Mato Grosso in central Brazil, a large state that has seen a lot of deforestation. The rate of deforestation tracked the rate estimated independently by the Global Forest Watch for the years 2001 to 2017. The major feature, the slowing down of the rate of deforestation in 2011 probably as a result of increased enforcement by the state government, is very clearly discernible.

The successful application of the deep learning technology in Mato Grosso has encouraged astraea to aim at applying this approach globally. They also intend to use satellite data with higher resolution and to handle seasonal differences better.

About STEP

The System for Terrestrial Ecosystem Parameterization (STEP) is a model for deriving vegetation and land surface parameters from remote sensing data for use in remote sensing-based classification of land cover, ecosystems, and vegetation types. The model defines parameters that relate to important ecological and biogeophysical parameters and that can be reliably measured or inferred from remote sensing, collateral, and field plot data. STEP is maintained as a database of training polygons drawn on high spatial resolution imagery that can be extracted with GIS to produce a global land cover classification. STEP is periodically reviewed to filter out inconsistent sites and augmented to fill gaps in biogeographical coverage. The database was originally created to follow the International Geosphere-Biosphere Programme (IGBP) land cover legend but it has since evolved to support any number of additional classifications.