Between the Poles

Automated feature recognition helps democratize utility mapping for producing infrastructure digital twins

In jurisdictions worldwide the low quality of underground location data has been identified as a top cause of underground utility damage. Existing underground utility data, most of which is stored in network owners' records, is often inaccurate, out of date or simply missing.  In North America in a survey of 402 locate professionals better facility maps received nearly unanimous agreement as a top strategy for improving the efficiency and quality of locates.  Singapore's Digital Underground project is unequivocal about the economic and social impact of low quality underground location data stating that "virtually all stakeholders are aware that much of the available information is unreliable and that this has repeatedly led to losses of time, money and opportunities."

Challenges facing the utility mapping industry

The root cause of the low quality of underground utility data is how utility mapping is structured as an industry. The primary activities of the utility mapping industry are data collection and how we keep and share this data with other parties.

Data collection

The way we collect is slowly changing, but predominantly we're still using traditional survey and locate methods. The survey methodology we are using hasn't changed in a hundred years. It's a very time and materials based, manual and analog process. Survey processes are slow and expensive to execute and require specialized equipment and skill sets, and safety is a big concern.

Locating underground utilities is one of the biggest causes of delays in construction schedules, service disruptions are a common occurrence, safety is always a major concern. and locate operations are slow to execute. Accuracy and timeliness are major issues. Locate operations require specialized equipment and skill sets which restricts locators to a small number of industry insiders.  A major shortfall is that underground utility location data is rarely available during the crucial planning and design stages of civil engineering projects.

Democratization of utility mapping

The utility mapping industry needs some fundamental changes. We need to move away from the monopoly that exists right now. There is just a handful of companies that are able to go out and collect underground utility data. We need to democratize data collection so that many more people can collect this data, targeting a crowd sourced approach and enabling multiple creators or collectors providing this data. This distributes the responsibility for the data as well. The more distributed the responsibilities, the lower risk and the lower the cost of failures will be.

We also need to change the skill level required. With current technology, we should be able to automate much of routine locating and thus enable many more people to do it. Using inexpensive technology will lower the cost of collecting and aggregating locate data. As data becomes lower cost to capture, data retention will become less expensive which will encourage people to retain the location data they acquire rather than simply painting/marking the ground and not capturing the data digitally.

Democratization is the real future of 3D infrastructure mapping. Right now data is still very siloed. We'll always have the traditional mapping companies, survey companies, line locating companies and asset owners going out and collecting their own data. But through crowdsourcing data from routine day to day locating and mapping, infrastructure projects, and other sources, aggregating that data and distributing access to that data is how utility mapping will really come to the forefront.  It should be easy to upload data into a central register where it can be made accessible to multiple stakeholders. Compliance with Open Geospatial Consortium (OGC) standards provides a basis for full interoperability enabling open access to every single conduit, cable, pipe, or other piece of infrastructure in real time from your phone, your tablet, your laptop or other applications.

Infrastructure digital twin

We think the best paradigm for producing and capturing asset data is through collecting it into an infrastructure digital twin. Digital twins have made a lot of progress in a variety of industries. Most importantly, for the utility mapping industry, it should be noted that creating a digital twin is much easier than it was even just a few years ago. We think the time is right for an infrastructure digital twin and it can be just at the asset level for one particular company, but we believe it should be at a regional level. Cities, municipalities and regional governments around the world are beginning to develop comprehensive digital twins of all of their physical infrastructure; buildings, roads, trees, power lines, and subsurface utilities and they're pouring resources into it.

And we think that Canada and the United States should start doing the same thing. And for us in the utility industry, data collection costs are lower than they ever have been. There's clearly interest among stakeholders and governments to start dealing with infrastructure and utility mapping problems, particularly to lower the number and costs of strikes, injuries and the loss of life. We now know a lot more about how to create data from other data sets, using different types of analysis and artificial intelligence to piece disparate data sets together. Using AI machine learning in the cloud is much less inexpensive compared to just a few years ago. And nowadays one of the things that makes infrastructure digital twins readily accessible to the utility industry is the fact that storing huge volumes of data in the cloud has become very inexpensive.

What is infrastructure digital twin ? First of all, a digital twin needs to result in effective action. It's not just enough say we have a digital twin philosophy The purpose of having a digital twin is to result in effective action that improves your business. It needs to use real time and historical data, which is paramount for us in the utility mapping industry, and it's going to be motivated by outcomes. For us in the utility industry, it is to reduce and even eliminate utility strikes during construction and the associated risks of injuries and fatalities.

The infrastructure digital twin has to be stored and managed in a relational database so it can be queried, analyzed and improved on through time, and related to other data sets. Processes for collecting data and maintaining data must be simple so that we can break the barrier of restricting underground utility mapping to a specialized group of people. If we truly want to democratize and compile data from disparate data sets, we need very simple processes to get data into the database, analyzed and maintained.

An infrastructure digital twin should be captured and maintained in 3D, even though most utilities in the ground are within just a couple of meters of the top of the ground. The 3D component is vital, because part of analyzing a data set is that much ancillary data is 3D such as overhead power lines that come down the side of a pole into conduits below the ground,

Imagery that's taken looking straight down at the ground, like traditional satellite imagery, can be visualized on a map and incorporated directly into a GIS database. But other types of imagery such as oblique, 360-degree, street-side, and inspection imagery need to be treated differently.  Oriented Imagery has a range of possible applications. For example, a construction company might use it to review asset inventory photos taken by field workers at a construction site. A telecom company or electric utility would use it to examine close range inspection images of electrical and telecom towers from many different angles. Oriented imagery applications enable you to see exactly where the image is on the map as you pan or zoom in the image. There are spatial navigation tools designed to work specifically with oriented imagery. You can query oriented images using spatial and temporal filters.

Finally, it is very important to have standardized formats, including storage formats and cloud formats.  It improves our ability to inexpensively use machine learning to do associative analyses of disparate data sets to make a larger, more conflated data sets.

Automated feature extraction

Maintaining accurate, up-to-date, comprehensive data about assets, above ground and below ground, has been a challenge for utilities for decades. Now new technologies including machine learning are being applied to automate real-time maintenance of utility network information and provide a foundation for infrastructure digital twins.

Automated feature extraction has advanced dramatically in the last few years.  This is an example where high density LiDAR and ultra high resolution imagery was used on a very low, very small diameter gas distribution install from an open trench. You can clearly see the ease with which the LiDAR produces a 3D representation of all of the local utility infrastructure.

Much of the the advancement in in automated 3D mapping have derived from the autonomous vehicle industry which have trickled down to other parts of the GIS industry.  The hardware and software are vastly easier to use and produce vastly better data. You can see from these ortho images collected from our system and the LiDAR overlaid on it. Feature extraction was used to automatically define top of pipe, top of ditch and actual joints in the pipe. Automated classification of LiDAR removes the need for someone to go out there with a Pogo stick and find the top of pipe. It makes it possible to simply drive alongside and scan with a mobile LiDAR system an open trench containing a newly installed pipe to produce a detailed 3D model with centimetre accuracy without having a person present. It is much safer - no boots on the pavement - and it drastically reduces the cost.

We prefer mobile mapping systems as opposed to handhelds because mobile mapping systems produce a tremendous amount more data and its registration tends to be vastly superior. It is much safer because it eliminates a need for someone to get out of a vehicle - so no boots on the pavement. Anytime you're still getting out, whether you're using handhold laser handhold mapping system, or a Pogo stick, you still have a person outside the vehicle walking around a street. With a mobile system you eliminate that risk, and you also drastically decrease the time required to do the collection. Here is another example where automated feature extraction was used to identify the electro fused coupling. You can see the butt joints, you can see the Ts and so forth. All this can be done very simply now with automated feature extraction.

Data registers

While surveyors may have had experience with a provincial or other types of survey register, utility location data identified during routine locate (one call) operations in general is only used to mark the ground and is rarely recorded, digitally or otherwise. Practically this means locators going back to the same location they were at just a few weeks ago to repeat the locate operation for another project. People in the locate industry are very familiar with this and the  amount of schedule and cost expense it causes. Exposing people to relocate or remap an area that was just done not too long ago is doubling their risk. Secondly, because the data is not retained or kept in easily accessible ways, we can't analyze it.  For example, we might want to confirm that we have correctly identified the type of utility, the accuracy of its location, or determine if there have been changes in its location, type, or current status since we last detected it. And because the data is not being retained and centrally managed at a global level or at a large scale level, there's no real opportunity for organic data enhancements or improvements.

The CGA Next Practices Report to Industry has made the case that a comprehensive national GIS map of buried infrastructure is among the opportunities for systemic improvement with the greatest ROI for industry. The benefits of a national register are greater safety, enhanced construction efficiency; reduced design, construction, and maintenance costs; greater regulatory compliance; improved locate efficiency; fewer project delays and budget overruns; fewer service disruptions to the public, less traffic disruptions and loss of custom and other inconveniences for the public.

Sharing maps of underground infrastructure is not totally new to Canada. For example, the City of Calgary's Joint Utility Mapping Project (JUMP) system, the ICI Society in British Columbia, the City of Toronto's Digital Map Owners Group (DMOG) and others have been sharing underground utility information for years.

The Canadian Underground Infrastructure Register (CUIR) is a new national initiative started at the beginning of the year. The CUIR initiative is intended to enhance current practices in a way that enables all stakeholder groups to benefit from the initiative; construction contractors, network owners, government transportation agencies, design engineers, subsurface utility engineering (SUE) engineers, surveyors, locators, workers in the field and the public. The CUIR initiative has been inspired by the National Underground Asset Register (NUAR) project currently being implemented in the UK. The objective of NUAR is to develop a register of location data for all underground infrastructure in England, Wales and Northern Ireland that is accessible to all network owners and government agencies.

This post is based on Joseph Hlady's (Lux Modus) talk at Subsurface Utility Mapping Strategy Forum (SUMSF).

Automating scan-to-BIM for brownfield substations

With about 55,000 substations in the U.S. and the average age of the transformers in these substations over 40 years old, many substations are going to be refurbished, retrofitted, extended or upgraded. Utilities are opting to create digital twins of their existing substations as part of this process to enable maintenance and upgrading activities to be performed more efficiently. For older substations records may consist in outdated 2D paper drawings or documents may be missing altogether.  New digital scanning technology together with machine learning is making it possible to semi-automate the process of creating intelligent BIM models from point clouds derived from laser scanning or photography.

primtech is a CAD-based design tool for designing high voltage substations.  An important recent enhancement Optical Symbol Recognition (OSR) allows point clouds from laser scanning or photography to be imported and then to apply machine learning to semi-automate the process of creating an intelligent BIM model. 

The software includes an extensive library, which contains more than four thousand intelligent 2D and 3D electrical devices from different manufacturers, steel structures, foundations, and wires and tubes.  The library has been growing for over 15 years and includes new equipment and elements as well as for legacy assets.  Each symbol (device, steel or foundation, etc.) has a specific geometric signature in the database in addition to meta data, file links, connection points and defined object behaviours. Together these comprise smart objects that enables objects in point clouds to be recognized and converted into an intelligent substation information model. The resulting intelligent model provides the basis for the calculations required in the substation engineering process.

The OSR process consists of two phases; first conductors are created and then equipment is added. A wire can be created by simply selecting three points, first anchor point, maximum sag point and end anchor point. For tubes, two or more points can be selected. Wires and tubes are available in the library for different conductor types according to international standards.  After conductors has been created and placed, the OSR tool provides a list of compatible library symbols for each symbol type. It is up to the user to choose which symbol should be selected. The user can select the perfect matching symbol or another one in the list. A similar workflow applies to all the other symbols like support structures and foundation heads. The system then combines the symbols to create assemblies (electrical device, support structure, foundation head) and places them in the model. Once an assembly has been created, the OSR tool will recognize identical assemblies in the point cloud and propose it for those positions in the substation. The wires are automatically attached to the respective electrical connection points on the equipment upon the insertion of the assemblies.  After the OSR tool has examined all objects in the point cloud and they have been mapped to an assembly in the model, a logical and intelligent primtech BIM model has been created. The model can be used as a full-fledged template for extension or retrofit - to create parts lists and plans as well as to carry out calculations including phase checking, clearance calculation, lightning protection calculation, sag and short-circuit effects calculations.

The new OSR technology is not fully automatic but drastically improves efficiency and reliability of converting point clouds into intelligent 3D models.This is an important advance that could dramatically improve the process of scanning brownfield substations to intelligent models by making the process more efficient and less error-prone.

M. Kokorus, F. Pizarro, W. Eyrich and S. Heuser, "From Optical Symbol Recognition (OSR) of Point Clouds to the Substation Information Model," 2018 IEEE/PES Transmission and Distribution Conference and Exposition (T&D), Denver, CO, USA, pp. 1-4., 16-19 April, 2018

Automating vegetation management for electric power transmission lines using deep learning with LIDAR

With FAA rules permitting beyond visual line of sight (BVLOS) UAVs available through Part 107 waivers the cost of capturing the data required for vegetation management for power lines, whether transmission or distribution, could drop dramatically.  However, manual post processing of the imagery to identify areas of vegetation encroachment, grow-ins and fall-ins, remains an error prone, tedious and time consuming process.  I have come across several firms (for example EPRI, SiteSee, Trimble RealWorks) over the years that have been working towards automating the process for identifying vegetation encroachments including grow-ins and fall-ins, but I have not encountered a utility that is using this technology operationally.  At Distributech 2019 in New Orleans I had a chance to chat with a startup Enview, based in San Francisco,  who described to me how their software has been implemented  over 7,000 miles of transmission lines last year to automate the identification of vegetation encroachment from LiDAR point clouds.  Furthermore, they expect the total distance covered to double this year.

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that utilities even have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment which typically are carried out by manned helicopters.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. I have blogged about Energisa in Brazil who used satellite stereo imagery to prioritize and optimize tree pruning and ComEd's pilot project in Chicago that uses LiDAR imagery to not only prioritize and optimize but also to change the pricing structure for procuring distribution tree trimming.

Enview applies machine learning, computer vision, and geospatial big data analytics to highly reliable feature extraction from LiDAR point clouds to identify the exact location, height and extent of trees and then to determine proximity to power lines and other utility equipment.  Utility GIS data can be integrated and results visualized using colour-coding to identify the level of risk for each tree.  This enables the utility to prioritize their tree trimming and optimize contractor routing.  In environmentally sensitive areas the visualization can be used to inform discussions with the various stakeholders responsible for different land uses.

At Distributech I had a chance to chat with Krassimir Piperkov, COO at Enview, who described how this technology has been deployed on more than 7,000 miles of infrastructure last year, and is on track to be deployed on over twice that much this year. He gave a very recent example of annual vegetation management analytics performed on 230kV transmission lines in California. These lines run through suburban and environmentally-sensitive, heavily forested areas.

3D visualization of a 230kV span (blue lines) with potential grow-in (yellow) and fall-in (red) vegetation that falls within client-determined clearance boundaries.

LIDAR data was acquired by a third party through a helicopter-mounted Teledyne Optech LiDAR sensor package. Four band imagery was co-collected during the LIDAR survey and registered into an orthomosaic of RGB and NIR bands. The data collection generated almost 2GB per linear mile. The Enview data analytics engine processed this data to (i) first classify the massive amounts of LiDAR data and (ii) then automatically calculate potential grow-in and fall-in vegetation points with high speed and accuracy. Insights and data were delivered to the operator not only via GIS-compatible file format for incorporation into existing workflows but also via an intuitive web portal accessible across silos. As a result of the superior accuracy and speed, the client was able to confidently prioritize vegetation prescription work to the areas that need it most and do so much sooner than otherwise possible with traditional methods.

The results being achieved with Enview's technology suggest that we are on the cusp of the widespread adoption and application of deep learning technology to automate specific tasks. The combination of UAVs for data collection and machine learning for feature extraction dramatically reduces the cost of inspections for utilities. This would permit them to be done more frequently than with manned aircraft and would lead to improvement in the reliability of above ground electric power and communications networks.

LiDAR changes the cost structure for utility distribution vegetation management

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that utilities may have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. I blogged previously about Energisa in Brazil who used satellite stereo imagery to prioritize and optimize tree pruning. At Distributech 2019 in New Orleans, Sainab Ninalowo and Dan Marron from ComEd described a pilot project in Chicago to use LiDAR imagery to not only prioritize and optimize but also to change the pricing structure for distribution tree trimming.

Vegetation SAIDI

Currently ComEd trims 25% of their service area every year so that their entire service area is covered every four years. In addition they perform reactive trimming - when they have a vegetation-related problem reported they dispatch a crew to resolve the problem. Statistics show that the number of vegetation-related outages (vegetation SAIDI) has been declining year over year. The contractors who do this work are currently paid by the circuit mile independent of tree density, so that a contractor working in a rural or suburban area with many trees is paid the same per mile as a contractor in an urban area with few trees.

Fall-ins: yellow doughnuts Grow-ins: orange within 0.5 ft, amber within 0.5-1.5 ft, light yellow within 1.5 to 5 ft

ComEd has had experience using LiDAR for vegetation management for its transmission lines and decided on conducting a pilot project to apply the technology to its distribution network. It chose circuits in four areas in its service territory with a range of tree densities ranging from rural to urban. ComEd put the capture of LiDAR data and post-processing of the point clouds out to competitive bid and selected two vendors. For the LiDAR imagery acquisition fixed wing aircraft were used. Post-processing involved combining the distribution network from their GIS with the vegetation identified in the point cloud imagery and modeling cylinders around the cables to reflect the tolerances for vegetation encroachment (grow-ins and fall-ins).

Using the volume of vegetation encroachment in the cylinders it was possible to calculate the volume of tree trimming required. This made it possible to procure tree trimming services with a different pricing model, based on per cubic foot instead of per circuit mile.  This change in procurement modus enables contractors to more accurately estimate labor and equipment costs and ComEd to prioritize their tree trimming based on the needs revealed by the data.

In addition to LiDAR cameras the aircraft were equipped with IR and hyperspectral sensors. The IR imagery made it possible to successfully identify weakened and unhealthy trees. The hyperspectral data was used to try to identify different tree species, but the results were not able to this reliably enough to recommend using this technology operationally.

The pilot has been deemed to be a success, although there are not yet direct cost comparison figures publicly available. In addition to the perceived benefits for vegetation management, ComEd has identified other significant benefits which can be derived from the LiDAR data. Based on the results of the pilot it is being recommended that LiDAR be phased in gradually for the entire network with the goal of conducting a LiDAR scan of the entire distribution network annually.

This is another example of how geospatial technology is fundamentally changing the utility industry, in this case how vegetation management services are procured.

Machine learning cost-efficiently creates high quality utility GIS data for smart grid applications

Many utilities' GIS contains only a small part of what is required for managing a smart grid.  Often even the data in the GIS is too unreliable to use for smart grid applications.  For example, utility GISs may not include which customers are connected to which transformer, or if they do include this information it is incomplete or incorrect.  Many utility GISs cannot identify the phase(s) each customer is using and they may not include network topology - what is connected to what.  Other missing data may includes street lights, distributed energy generation such as rooftop PV panels, the type and condition of poles, joint use of poles, and encroachment.  To correct these problems typically requires a foot survey which is expensive.  For example, a pole inventory costs at least $20 per pole.  At the Open Geospatial Consortium (OGC) Energy Summit at EPRI in Charlotte, John Simmins of EPRI described the research he is leading at EPRI to use modern technology including machine learning to reduce this cost to pennies per pole using available imagery such as Google Streetview.

I began working in the utility and telecom sector in 1993, specifically in developing software for records management (called network documentation outside of North America), and at that time the magnitude of the AEC+geospatial cultural mismatch was just beginning to be appreciated by utilities and telecoms. Planners tend to use GIS tools, engineers and designers CAD tools, construction folks in the field paper CAD drawings, and asset managers FM, GIS or integrated tools. Fashioning AEC and geospatial data into an efficient data flow from planning through design and construction to operations and maintenance represented a challenge that remains a problem for utilities.  For many utilities the backlog of as-builts and updates waiting to be entered into the GIS stretches into months, with the result that GIS data is permanently out of date inhibiting management and field staff from relying on this data operationally.  The Between The Poles blog is over ten years old and one of the persistent themes from its very beginning in 2006 was the challenge of integrating CAD and GIS data in a common workflow (some examples; 2006, 2007, 2008 ).

There are signs that modern technology is beginning to offer a solution to the problem of developing and maintaining an accurate digital model of electric power grids.

I wrote recently about a utility in western Pennsylvania that intends to implement an Advanced Distribution Systems ( ), but realized that to this requires a network data model. Duquesne Light is typical of many North American small to medium electric power utilities.  It does not have an outage management system or an electrical data model which means it is difficult to connect customers to transformers and to determine upstream devices when customers report power problems.  Currently Duquesne Light's circuit maps are AutoCAD drawings.  Duquesne Light is moving to modernize its grid and plans to deploy an Advanced Distribution Management Systems (ADMS) including outage management (OMS), DMS and DSCADA.  But the ADMS will require developing a complete distribution electrical data model and to that in the traditional way can be expensive.

To quickly, safely and cost efficiently develop a network data model, Duquesne Light looked at several alternatives including reality capture technology.  A pickup truck was equipped with with side mounted photo and IR cameras and LiDAR laser scanners. The reality capture survey could be completed rapidly and safely, with no boots on the pavement  Actual development of the data model was completed in the office without the necessity of going to the field for quality checks because the imagery provided all the data that was required to create and verify the data model.  Duquesne Light found several significant advantages of the reality capture approach.  The first was asset accuracy - both location and asset attributes.  In addition the captured imagery could also be used for other purposes.  For example, it could be used for a joint-use audit.

But this is still basically a manual process requiring a human being to scan images.  At the Energy Summit John Simmins described his research using neural network technology to automatically identify poles and attached equipment from available imagery such as Google Streetview.  Normally applying neural networks requires large quantities of training data.  To reduce this requirement, John has evaluated "biologically inspired", forward pass, neural networks.  In this approach the algorithm first looks (foveates) for vertical things that have the characteristics of a pole, and then for the characteristics of equipment attached to the pole, for example, cross members, street lights, transformers and large cables.  From this analysis is possible to determine pole location, height and width, cross member position on the pole and projected length, street light support strut position and length, position of transformer and large cable connection locations.  The ultimate goal of the research is in addition to automatically detecting each element of equipment to identify the type, size and manufacturer of each piece.

This is not a simple problem and there is no silver bullet.  Using a standard neural network approach requires large volumes of training data.  For poles it requires images of poles with different types of damage under different lighting conditions and in different types of weather.  An Australian company SiteSee claims to be able to do this for communications towers and transmission pylons - identifying the type and manufacturer of each antenna or insulator on a tower or pylon and comparing it to the manufacturer's specification to detect signs of corrosion and other types of damage.  When what you are looking for has a distinctive geometry that differentiates it from the surroundings, a biologically inspired approach may provide a way of identifying equipment without large volumes of training data.  Another area where several different approaches have been applied to automate feature extraction is electric power transmission lines for vegetation management.

Creating a digital twin of an electric grid is essential for developing and maintaining a smart grid.  Utilities have been wrestling with this challenge for decades and in many cases their GIS data is still not much closer to what is required.  If it would be possible to use the technology that John Simmins is investigating or other technology to reduce the cost of improving the quality of their GIS data, it would accelerate the adoption of smart grid technology at many utilities.  In his presentation at the Energy Summit Jared Green described his research in using smart meter and AMI network data to map secondaries from the transformer to the home, identify sources of unmetered load, and use impedance data from smart devices to correct utility GIS data.  Given the history of the challenge to develop and maintain accurate GIS data at many utilities, these approaches represent a beacon of hope which may enable utilities to cost effectively develop a digital twin of their networks that is an essential prerequisite for a  smart grid.

EPRI curates open utility-related training datasets for deep learning applications

Automatic feature extraction or image classification using artificial intelligence (AI) algorithms has been successfully applied in several domains, for example, in recognizing landmarks, dogs, cars and other objects in consumer photos and detecting building footprints in high resolution satellite imagery.  The existing algorithms for doing this are often available in open source implementations and have been democratized to the point where they are available to anyone with a modicum of programming skill.  The challenge is creating and maintaining (curating) the training datasets that are required to turn the basic algorithms into models that can automatically recognize and classify different types of objects.  At the December Open Geospatial Consortium (OGC) Energy Summit at EPRI in Charlotte, Andrew Phillips of EPRI described EPRI's role in the new world of big data which includes curating open training datasets for the electric power industry to be used by the AI industry including universities, startups and established software players to create applications for electric power utilities.

Deep learning is increasingly finding practical applications using different types of georeferenced imagery.  For example, convolutional neural networks (CNN) have been successfully applied to real world problems including detecting blocked canals, differentiating different types of roofs, identifying building footprints and recognizing transportation networks.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded. The basic process in creating these models is to find or create some training data, some data to test the model, and then play with the parameters to optimize the model for the test data. Google is pretty good at recognizing dogs, cars and other objects that are commonly found in consumer photos.  But if you present Google with photos of utility equipment such as a photo of a decayed wooden pole with an insulator precariously hanging off it or a cracked fuse, Google will not recognize these objects correctly.  It is not a problem with Google's algorithm, but a lack of training data - photos whose content has been "labeled" by electric power experts.

An essential piece that is required to open up the data and technology of deep learning to a broader audience is open labeled data, what has been called an "open labeled data commons". There are analogies with OpenStreetMap initiative - something like this is required for labeled data - it needs a repository of open data, tools to capture and manage the data, and a community around this to capture and curate that data.  For example, for automatically classifying land use in satellite imagery the System for Terrrestrial Ecosystem Parameterization (STEP) has 2000 manually labeled sites covering 17 different land cover types that can be used a training data for deep learning applications to land use.

EPRI has already been active in collecting large volumes of training data from members and curating training datasets that can be used for common industry use cases.   For example,  EPRI has pulled together dissolved gases data for 45,000 transformers from 27 utilities which includes a few thousand failures.  The data has been provided to AI vendors to train their models.  Some of the data has been held back to test the models to see how well the vendors have done.

In another experiment EPRI asked 12 experts to review inspection imagery to identify and rank problems with common electric power equipment.  Thousands of labeled images were provided to nine vendors including IBM, Intel, Harris and others to train their deep learning models.   EPRI held back 500 of these images for testing and have just completed running them through an evaluation process to see how well the models do in identifying problems.  For example, it was found that the best models could identify cracked and damaged insulators with 88% confidence (using Matthews correlation coefficient), but cavities in wood poles such as woodpecker holes could only be identified at the 50% confidence level.  The results have revealed more images are needed under different lighting conditions. EPRI is going back to its members to get more images. Another area of interest to EPRI is monitoring transmission lines including vegetation encroachment using imagery from BVLOS UAV flights. 

EPRI's strategy is to be the bridge to the AI community by making the AI community aware of use cases that are important for the electtric power industry, helping the AI community with the physics of electric power and supporting structures, helping its members understand how AI can be used in the electric power industry, and collecting and curating open training datasets to be used by the AI community,

Enabling the rest of us to find actionable information in huge volumes of satellite data

There is a huge volume of satellite imagery that is being captured and has been captured over the past couple of decades.  The challenge is to automatically find objects such as ships, trees, buildings, or containers in this vast amount if data and track changes in them over time.  This morning at at this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis, Chris Holmes of Planet Labs, gave an insightful overview of the application of deep learning to geospatial data, identifying opportunities for the open source community to lead in the application of this data and technology to addressing this challenge.

Outside of Google, Planet Labs, DigitalGlobe and some others deep learning has been applied primarily to recognizing people in selfies and dogs and cats from peoples photos on online repositories such as Flickr. I have blogged previously about competitions DigitalGlobe (which has 100 petabytes of imagery) and Nvidia have hosted into automating building footprint and road network recognition. Planet Labs has a computer vision and analytics team that has been applying the results of academic research to geospatial data, primarily imagery from Planet's constellation of 150+ satellites. For example, they have used deep learning to detect building footprints and road networks in Dar es Salaam. Chris showed examples of airplane, ship and container recognition achieved using deep learning. But these are research pilots. One company has applied deep learning to scale for a practical application. Google has been developing and applying the technology to building recognition - automatically identifying building footprints and features such as aerials. The deep learning algorithms have been developed by academia and as a result the code for the most part is open source.  For example, a  deep neural network model developed originally for medical image segmentation called U-Net  has been applied to identifying building footprints.  Successful training of deep networks requires many thousand labeled training samples and this training data is not open source. Labeled data involves people on the ground manually ground-truthing building footprints and other features so that the deep learning algorithms can learn what to recognize.

Chris believed that it is time to open these models to a broader group. There are some technical things that need to happen to make this possible. We need new formats for exchanging raster and vector data. Cloud optimized GeoTiff (COG) makes it possible to stream imagery from cloud sources such as Amazon S3. Planet, Google Earth, and QGIS already support COGs and there are other open source tools such as COG-Explorer that support them as well. Chris and others are working on WFS 3.0, a completely new developer-friendly version of an existing OGC (Open Geospatial Consortium) standard for exchanging vector data over the web.

Another technical advance that is required is a common way to query satellite data based on date, time, location, type of bands, and so on.  Chris and others have been working on a standard for searching satellite imagery for objects. The SpatioTemporal Asset Catalog, known as STAC, is an open specification that came about when fourteen different organizations came together to increase the interoperability of searching for satellite imagery. Currently when a user wants to search for all the imagery in their area and time of interest they can’t make just one search — they have to use different tools and connect to API’s that are similar but all slightly different. The STAC specification aims to make that much easier, by providing common metadata and API mechanics to search and access geospatial data.  Chris emphasized that standards are essential for opening up satellite data to processing and visualization and encouraged the open source community to get involved with the OGC to further developer-friendly standards.

Currently in order to use imagery data, a lot of data preparation is required of the end user such as lining up pixels, assigning a projection, removing clouds and mist, and other corrections. We need to move toward analysis ready data where data derived from different sources can be combined for analysis.

Because of the huge volumes of computational intensity we need cloud-native engines to perform the analyses. Google, DigitalGlobe (Maxar), Planet have these deep learning engines, but Chris encouraged the open source community to get more involved in developing these engines.

An essential piece that is required to open up the data and technology to a broader audience is open labeled data, Chris referred to it as a "open labeled data commons". There are analogies with OpenStreetMap initiative. Chris suggested that something like this is required for labeled data - it needs a repository of open data, tools to capture and manage the data, and a community around this to capture and curate that data.

The other key piece is converting this data into actionable information.  For example, dashboards that would alert municipal governments to un-permitted development, forestry enforcement to illegal forest cutting, or commodity price monitoring agencies to changes into the flow of ships into harbours or to large changes in containers at container ports.

Taken together Chris suggestions and work underway represent a roadmap for the open source geospatial community to lead in making the huge volumes of satellite imagery and other data queryable and able to generate actionable information.

Deep learning enables automated extraction of building footprints and road networks from satellite imagery

Automated feature extraction from satellite imagery has made major progress in the last year. Accurate building footprints extracted from high resolution satellite imagery are available from companies such as Ecopia (which has just announced a partnership with DigitalGlobe).  Also NVIDIA has demonstrated the ability to automate detection of many road networks using sophisticated algorithms and multi-spectral high resolution imagery.

Building footprints

SpaceNet is a repository commercial satellite imagery and labeled training data being made available through Amazon Web Services at no cost to the public to foster innovation in the development of computer vision algorithms to automatically extract information from remote sensing data.  It is the result of a collaboration between DigitalGlobe, CosmiQ Works, and NVIDIA.  Two competitions for automated building footprint extraction from high resolution satellite imagery and a third for road network extraction have been held based on SpaceNet data. 

The first SpaceNet Challenge was launched in November 2016.  Imagery of Rio de Janeiro from the WorldView 2 satellite at 50cm resolution using eight spectral bands was made publicly available on Amazon Web Services (AWS).  42 developers competed to create algorithms that extract building footprints from the imagery.  The source code of the winning algorithms was made available on the SpaceNetChallenge GitHub repository. 

The winning implementation was developed by Brazilian developer using an implementation based on random forests with brute force polygon search and did not leverage deep learning frameworks.  The organizers' conclusion based on the results of Round 1 was that automated building footprint extraction remained a challenge.

Round 2 winner. Most building footprints are precisely detected. Number on each building footprint represents the overlap between the prediction and the ground truth.

Round 2 of the SpaceNet challenge involved higher resolution 30 cm imagery from WorldView 3 for four cities across the globe. The competition challenged developers to improve performance from the first competition using the higher-resolution imagery and more geographically diverse training data samples. SpaceNet on AWS held imagery and building footprints for Las Vegas, Paris, Shanghai, and Khartoum with over 180,000 buildings181,619 footprints.  In addition additional bands were made available (panchromatic, 3-band pan-sharpened, and 8-band pan-sharpened).

Results from Round 2 winning solution: top left: Las Vegas, top right: Las Vegas, lower right: Khartoum, and lower left: Paris. Blue outlines: ground truth, green outlines: correct identifications, red outlines: false positives and yellow outlines: false negatives.

The winner of Round 2 applied a  deep neural network model developed originally for medical image segmentation called U-Net.  Successful training of deep networks requires many thousand annotated training samples.  Training was aided significantly by augmenting the imagery with OpenStreetMap data.  The winning solution used OpenStreetMap layers and Worldview-2 multispectral layers as the input of the deep neural network algorithm.  In Las Vegas, the winning solution was outstanding in outlining suburban homes, but also did well in identifying odd-shaped buildings. In Khartoum, the solution was very good in identifying the footprints of stand-alone apartment complexes, but had trouble with small buildings, with building footprints that are close together, and with horizontally large buildings.  One of the problems was inconsistent ground truth which affected training. The solutions offered by the developers in the competition were sufficiently successful that the organizers concluded that the winning algorithms achieved performance with potential for automated mapping tasks such as keeping maps up-to-date and to assist first responders during natural disasters.  The source code for the winning implementations can be found in the SpaceNetChallenge GitHub repository.

DigitalGlobe has just announced a partnership with Ecopia, which has established an automated process to create building footprints quickly and at scale by leveraging DigitalGlobe's Geospatial Big Data platform (GBDX) and advanced machine learning in combination with DigitalGlobe’s imagery library. The two companies plan to automatically extract accurate 2D building footprints globally, then refresh the datasets periodically to find and track changes over time.  These datasets would be valuable to municipal governments for permitting purposes and for first responders after disasters.   Ecopia is developing a database for all of Australia (7.6 million square kilometers) which will include every building with a roof area greater than 9 square meters.  The database will contain building footprints for about 15 million buildings. The database includes linkages to other datasets including geocoded address, property data and administrative boundaries. Ecopia intend to update the dataset regularly to be able to track changes, especially urban sprawl.

Road networks

In November 2017 CosmiQ Works, Radiant Solutions, and NVIDIA announced a third round competition, “Road Detection and Routing Challenge” to explore automated methods for extracting map-ready road networks and routing information from high-resolution satellite imagery.  

Detecting road networks in Paris. Photo image, ground truth, and the results achieved with three deep learning algorithms

NVIDIA has released the results of several deep learning algorithms that illustrate just how difficult identifying road networks is and the sophistication of the available tools.  These include using multi-band spectral imagery to identify the material properties of the road surface itself (asphalt, gravel, packed earth) and using more sophisticated processing including conditional random fields (CRF), percolation theory, and reinforcement learning. 

Detecting road networks in Las Vegas. Photo image, ground truth, and the results achieved with three deep learning algorithms

NVIDIA utilized high performance GPU compute resources provided by the NVIDIA GPU Cloud and Amazon Web Services.

High resolution satellite imagery helps innovative electric utility optimize vegetation management

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that some utilities have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. At Distributech 2018 Pedro Lemes, and engineer at Energisa (Brazil) described an innovative project which used high resolution satellite imagery to optimize vegetation management. Energisa is the sixth largest Brazilian power utility with 6.6 million customers.

Vegetation is the source of many outages at Energisa and the budget for vegetation management represents a significant proportion of its operating budget.   To reduce the number of outages due to vegetation and the costs of vegetation management, Energisa initiated a project to develop a tool that would allow it to identify and map high risk areas for vegetation encroachment and secondly to optimize routes and crew allocation for tree pruning.  The objective of the project was to decrease the number of outages due to vegetation encroachment, reduce vegetation management costs especially third-party firms hired to identify risk areas for vegetation encroachment, reduce the level of risk associated with vegetation near power lines, and to bring more IT into operations at Energisa.

The project innovated in using satellite imagery together with GIS data to solve the problem of identifying high risk areas for vegetation encroachment in a distribution grid.  By analyzing satellite imagery it was possible to distinguish vegetation from other objects, to differentiate trees from grass and bushes, measure the height of trees, and using GIS data to measure the distance between trees and poles and cables. Energisa contracted with Airbus Pléiades to provide 50-cm resolution, colour, orthorectified as well as near infrared (NIR) imagery. The contract with Airbus specified that all imagery supplied to Energisa had 15% or less clouds.  The imagery enabled identification of vegetation using the RGB and near infrared (NIR) bands.  In addition Pleiades provides high-resolution stereo imagery enabling generating digital surface and terrain models.  Trees heights could be determined by comparing surface and terrain models.    Trees were separated from other vegetation by a minimum height threshold.  Even when there were many trees together, it was possible to separately identify individual trees. NDVI (Normalized Difference Vegetation Index) was used to differentiate between vegetation and other objects and to distinguish healthy and diseased trees.  Coconut palms were the most challenging because of the lack of a central crown.  Energisa's GIS provided the location of line segments, switches, poles and structures.  Based on the information derived from the two sources it was possible to calculate the distance between trees and cables and identify high risk areas of vegetation encroachment.

The second part of the project involved developing a pruning management system to optimize pruning.  Once the risk areas had been identified, based on the available budget, team locations, the desired level of risk and the cost of pruning a tree, the algorithm was able to determine an optimized route and schedule for pruning operations for each team.  The algorithm was sufficiently robust to be able to be applied to a neighbourhood or an entire city.

Nowadays there is a plethora of satellite imagery with resolution as high as a foot and a frequency of several times a day.  One company is now imaging the entire landmass of the Earth every day. Satellite imagery is becoming increasingly competitive which is lowering the cost of imagery.  Energisa's creative use of satellite imagery and another innovative application that uses satellite-based radar interferometry to detect subsidence, which often can be sourced to a water leak, are two examples demonstrating how satellite imagery can be applied by innovative utilities to improve performance and reduce costs.

Fully automated classification of road and highway features for autonomous vehicles

Point clouds are fine for visualization to provide context, but to be really useful they need to be classified.  Feature extraction, as the process is often referred to,  is currently a mostly manual process that has been used for many years in the remote sensing sector. A trained person visually inspects the point cloud and graphically selects features which are identified by type of feature, sign, light pole, lane marking, and so on. 

Automated classification of point clouds

Image processing of 3D point clouds has advanced to the point where it has been possible to semi-automate this tedious feature extraction process.  The reliability of the automation process depends on the point density of point cloud, often expressed as points per square meter or per square centimeter.  Point density depends on the technology used to capture the point cloud. Terrestrial lidar, for example, typically captures points at the rate of hundreds of thousands to millions of point per second and is capable of millimeter to centimeter precision. Phodar generates lower point densities and is less precise. Aerial lidar captures many points per second but with less precision because it is covering much greater areas. Single photon lidar is a new technology that can collect 6 million points per second which means that planes can fly quite high, cover very broad areas and still collect 12 - 30 points per square meter.

At the Year in Infrastructure 2017 (YII2017) conference in Singapore, Sharad Oberoi of Sanborn gave an insightful overview of the state of the art for automated feature extraction from combined aerial and mobile scans of highways and adjacent areas.  Different types of laser scanners are appropriate for capturing different types of features. Terrestrial and mobile laser scanners can capture point clouda that are suitable for extracting curbs, gutter drains, traffic signals, road and local terrain information, parking meters, walls, obstructions in the right of way, manholes, sidewalks, overhead clearances, lights, utility wires and connections, garbage cans, benches, fences, guardrails and barriers, retaining walls, and line of sight vegetation obstructions. Aerial lidar will only be able to get gutter drains, road and local terrain information, walls, obstructions in the right of way, and sidewalks with limited ability to get curbs, lights, utility wires and connections, retaining walls, and vegetation.  Lidar does not do well for metal structures such as communications towers and fir these a combination of lidar and phodar is preferred.

Sanborn has developed feature extraction technology that extracts transportation objects from combined aerial and mobile data sets in a semi-automated manner. The mobile and aerial data sets meet the accuracy and information content required for geospatial information for mapping applications as well as the additional information that can be mined for potential asset inventory and infrastructure information content. Mobile technology allows for the low-risk, rapid collection of geospatial information, limiting safety impacts to workers. The aerial LiDAR allows for the rapid collection of elevation information for detailed surface modeling as well as feature extraction. Both technologies collect enormous amounts of point-cloud data that can be merged using the Terrascan capability in Bentley MicroStation. The combined data set is then passed through the automated feature extraction modules to identify transportation objects.  Sanborn feature extraction modules achieve a high proportion of correct feature extraction for road and highway scans.

In the future Sharad expects that it will be possible to improve data collection efficiency by using terrestrial robots or autonomous UAVs equipped with SLAM technology to enable them to operate in GPS-denied areas such as under bridges and in tunnels.  Sanborn has been working with deep learning to improve the automated extraction process.  So far this has yielded mixed results, but Sharad expressed optimism that with AI full automation of the classification process will be achieved.

High precision maps for autonomous vehicles

An application of this technology is high precision maps for the autonomous vehicle market.

Sanborn develops high precision maps for the autonomous vehicle market.  Autonomous vehicles will require high precision maps which contain significantly more detailed information and true-ground-absolute accuracy than current road maps. Sanborn has developed proprietary mapping technology that leverages aerial imagery, aerial lidar data, and mobile lidar data to create standardized, high-precision 3D base-maps focusing specifically on the self-driving vehicle market. These maps include detailed inventories of all stationary physical assets related to roadways such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and all other critical data needed for the safe navigation of roadways and intersections by autonomous vehicles.