Between the Poles

Toward automating scan-to-BIM for brownfield substations

With about 55,000 transmission substations in the U.S. and the average age of the transformers in these substations over 40 years old, many substations are going to be refurbished, retrofitted, extended or upgraded. Utilities are opting to create digital twins of their existing substations as part of this process to enable maintenance and upgrading activities to be performed more efficiently. For older substations records may consist in outdated 2D paper drawings or documents may be missing altogether.  New digital scanning technology together with machine learning is making it possible to semi-automate the process of creating intelligent BIM models from point clouds derived from laser scanning or photography.

Companies are recognizing the benefits of an integrated geospatial and building information modeling (BIM) approach to design and construction. I blogged about the winner of the Year in Infrastructure 2019 award who applied BIM processes and reality modeling on a brownfield substation expansion project. The integrated approach enabled the organization to cut design time, minimize site visits, reduce land requirement and material waste, and, most importantly, eliminate change orders during construction.

Now reality capture and deep learning have been incorporated in a substation design product to semi-automate the scan-to-BIM process for brownfield substations.

primtech is a CAD-based design tool for high voltage substations.  An important recent enhancement Optical Symbol Recognition (OSR) allows point clouds from laser scanning or photography to be imported and then to apply machine learning to semi-automate the process of creating an intelligent BIM model. 

The software includes an extensive library, which contains more than four thousand intelligent 2D and 3D electrical devices from different manufacturers, steel structures, foundations, and wires and tubes.  The library has been growing for over 15 years and includes new equipment and elements as well as for legacy assets.  Each symbol (device, steel or foundation, etc.) has a specific geometric signature in the database in addition to meta data, file links, connection points and defined object behaviors. Together these comprise smart objects that enables objects in point clouds to be recognized and converted into an intelligent substation information model. The resulting intelligent model provides the basis for the calculations required in the substation engineering process.

The OSR process consists of two phases; first conductors are created and then equipment is added. A wire can be created by simply selecting three points, first anchor point, maximum sag point and end anchor point. For tubes, two or more points can be selected. Wires and tubes are available in the library for different conductor types according to international standards.  After conductors has been created and placed, the OSR tool provides a list of compatible library symbols for each symbol type. It is up to the user to choose which symbol should be selected. The user can select the perfect matching symbol or another one in the list. A similar workflow applies to all the other symbols like support structures and foundation heads. The system then combines the symbols to create assemblies (electrical device, support structure, foundation head) and places them in the model. Once an assembly has been created, the OSR tool will recognize identical assemblies in the point cloud and propose it for those positions in the substation. The wires are automatically attached to the respective electrical connection points on the equipment upon the insertion of the assemblies.  After the OSR tool has examined all objects in the point cloud and they have been mapped to an assembly in the model, a logical and intelligent primtech BIM model has been created. The model can be used as a full-fledged template for extension or retrofit - to create parts lists and plans as well as to carry out calculations including phase checking, clearance calculation, lightning protection calculation, sag and short-circuit effects calculations.

The new OSR technology is not fully automatic but drastically improves efficiency and reliability of converting point clouds into intelligent 3D models.This is an important advance that could dramatically improve the process of scanning brownfield substations and generating  intelligent models by making the process more efficient and less error-prone.

M. Kokorus, F. Pizarro, W. Eyrich and S. Heuser, "From Optical Symbol Recognition (OSR) of Point Clouds to the Substation Information Model," 2018 IEEE/PES Transmission and Distribution Conference and Exposition (T&D), Denver, CO, USA, pp. 1-4., 16-19 April, 2018

Read more about substation design

Applying machine learning to identifying housing starts from satellite imagery

Deep learning is increasingly finding practical applications using geospatial data, specifically, satellite imagery. At the GeoIgnite conference in Ottawa, Nicolas Martinez of Statistics Canada described a project scheduled to kick-off in July to use machine learning to identify housing starts across Canada.  The goal of the project is to improve the accuracy and coverage of housing starts survey, specifically to fill data gaps for smaller remote and indiginous communities which are generally excluded from the current survey program. 

Canada Mortgage and Housing Corporation (CMHC) currently employs a significant number of people at their head office in addition to field agents across the country who physically confirm residential starts and completions.  The objective of the project is to apply convolutional neural network (CNN) machine learning to satellite imagery to extract information about housing starts and completions. The imagery will be preprocessed to prepare it to be used in developing a machine learning processing model.  The largest challenge will be developing training and testing imagery data sets.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded.  More recent applications of satellite imagery targeted landuse and agricultural identification. For example, remote sensed imagery from a satellite can be used to create a model that will differentiate between corn and potatoes, using factors that can be calculated from satellite imagery such as normalized difference vegetation index (NDVI) (min, max, mean), texture (min, max, mean), vegetation height, and geometric factors such as orientation.  I blogged about how open source code and publicly available training data has been applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil. A study in the Netherlands used CNN to identify blocked waterways using overflight imagery with a success rate of 97 %.

An important application of satellite imagery is identifying built structures. I have blogged about how satellite imagery has been used to identify buildings and transportation networks by applying a neural network model together with OpenStreetMap layers and high resolution Worldview multispectral imagery.  NVIDIA has demonstrated the ability to automate detection of many road networks using deep learning algorithms and multi-spectral high resolution imagery.  Another application of CNN and imagery in the Netherlands was able to differentiate residential roofs with dormer windows from those without.

The essential key to effective application of deep learning is good training and test data - ground-truthed data that involves, for example, someone on the ground identifying whether the fields seen in the imagery are corn or potatoes or something else. There are  publicly available training data sets that can be used to train satellite imagery CNN models. For example, I blogged about the System for Terrrestrial Ecosystem Parameterization (STEP) dataset which has 2000 manually labeled sites covering 17 different land cover types scattered across all continents. A large publicly available training dataset derived from Sentinel 2 imagery contains 30,000 polygons of land use training data for ten classes of land use: annual crop, forest, herbaceous vegetation, highway, industrial, pastures, permanent crop, residential, river and sea or lake.

The Statistics Canada project is scheduled to begin in July with initial work on a proof of concept and CNN testing to begin in August and September.  The communities chosen for the initial phase are  Kitchener-Cambridge-Waterloo, Red Deer, and Iqaliut.  The first phase is scheduled to be completed by winter 2019.  The project plans to use Tensorflow and Keras, open source libraries and a python framework for applying machine learning.  Artificial intelligence (AI) has often suffered from inflated expectations, complex code, heavy processing requirements and not very practical applications. There is a growing body of practical applications in the geospatial domain that shows that as a result of the immense processing power available in the cloud machine learning is able to generate fairly easily practical and useful results.  As Chris Holmes pointed out in his talk at FOSS4GNA, the challenge for Statistics Canda will be to develop reliable and publicly available training and test datasets to enable deep learning models to be created for a broader range of applications using the huge volumes of satellite imagery and other geospatial data that are now available.

Deep learning is increasingly finding practical applications using geospatial data, specifically, satellite imagery. At the GeoIgnite conference in Ottawa, Nicolas Martinez of Statistics Canada described a project scheduled to kick-off in July to use machine learning to identify housing starts across Canada.  The goal of the project is to improve the accuracy and coverage of housing starts survey, specifically to fill data gaps for smaller remote and indiginous communities which are generally excluded from the current survey program. 

Canada Mortgage and Housing Corporation (CMHC) currently employs a significant number of people at their head office in addition to field agents across the country who physically confirm residential starts and completions.  The objective of the project is to apply convolutional neural network (CNN) machine learning to satellite imagery to extract information about housing starts and completions. The imagery will be preprocessed to prepare it to be used in developing a machine learning processing model.  The largest challenge will be developing training and testing imagery data sets.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded.  More recent applications of satellite imagery targeted landuse and agricultural identification. For example, remote sensed imagery from a satellite can be used to create a model that will differentiate between corn and potatoes, using factors that can be calculated from satellite imagery such as normalized difference vegetation index (NDVI) (min, max, mean), texture (min, max, mean), vegetation height, and geometric factors such as orientation.  I blogged about how open source code and publicly available training data has been applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil. A study in the Netherlands used CNN to identify blocked waterways using overflight imagery with a success rate of 97 %.

An important application of satellite imagery is identifying built structures. I have blogged about how satellite imagery has been used to identify buildings and transportation networks by applying a neural network model together with OpenStreetMap layers and high resolution Worldview multispectral imagery.  NVIDIA has demonstrated the ability to automate detection of many road networks using deep learning algorithms and multi-spectral high resolution imagery.  Another application of CNN and imagery in the Netherlands was able to differentiate residential roofs with dormer windows from those without.

The essential key to effective application of deep learning is good training and test data - ground-truthed data that involves, for example, someone on the ground identifying whether the fields seen in the imagery are corn or potatoes or something else. There are  publicly available training data sets that can be used to train satellite imagery CNN models. For example, I blogged about the System for Terrrestrial Ecosystem Parameterization (STEP) dataset which has 2000 manually labeled sites covering 17 different land cover types scattered across all continents. A large publicly available training dataset derived from Sentinel 2 imagery contains 30,000 polygons of land use training data for ten classes of land use: annual crop, forest, herbaceous vegetation, highway, industrial, pastures, permanent crop, residential, river and sea or lake.

The Statistics Canada project is scheduled to begin in July with initial work on a proof of concept and CNN testing to begin in August and September.  The communities chosen for the initial phase are  Kitchener-Cambridge-Waterloo, Red Deer, and Iqaliut.  The first phase is scheduled to be completed by winter 2019.  The project plans to use Tensorflow and Keras, open source libraries and a python framework for applying machine learning.  Artificial intelligence (AI) has often suffered from inflated expectations, complex code, heavy processing requirements and not very practical applications. There is a growing body of practical applications in the geospatial domain that shows that as a result of the immense processing power available in the cloud machine learning is able to generate fairly easily practical and useful results.  As Chris Holmes pointed out in his talk at FOSS4GNA, the challenge for Statistics Canda will be to develop reliable and publicly available training and test datasets to enable deep learning models to be created for a broader range of applications using the huge volumes of satellite imagery and other geospatial data that are now available.

Automating vegetation management for electric power transmission lines using deep learning with LIDAR

With FAA rules permitting beyond visual line of sight (BVLOS) UAVs available through Part 107 waivers the cost of capturing the data required for vegetation management for power lines, whether transmission or distribution, could drop dramatically.  However, manual post processing of the imagery to identify areas of vegetation encroachment, grow-ins and fall-ins, remains an error prone, tedious and time consuming process.  I have come across several firms (for example EPRI, SiteSee, Trimble RealWorks) over the years that have been working towards automating the process for identifying vegetation encroachments including grow-ins and fall-ins, but I have not encountered a utility that is using this technology operationally.  At Distributech 2019 in New Orleans I had a chance to chat with a startup Enview, based in San Francisco,  who described to me how their software has been implemented  over 7,000 miles of transmission lines last year to automate the identification of vegetation encroachment from LiDAR point clouds.  Furthermore, they expect the total distance covered to double this year.

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that utilities even have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment which typically are carried out by manned helicopters.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. I have blogged about Energisa in Brazil who used satellite stereo imagery to prioritize and optimize tree pruning and ComEd's pilot project in Chicago that uses LiDAR imagery to not only prioritize and optimize but also to change the pricing structure for procuring distribution tree trimming.

Enview applies machine learning, computer vision, and geospatial big data analytics to highly reliable feature extraction from LiDAR point clouds to identify the exact location, height and extent of trees and then to determine proximity to power lines and other utility equipment.  Utility GIS data can be integrated and results visualized using colour-coding to identify the level of risk for each tree.  This enables the utility to prioritize their tree trimming and optimize contractor routing.  In environmentally sensitive areas the visualization can be used to inform discussions with the various stakeholders responsible for different land uses.

At Distributech I had a chance to chat with Krassimir Piperkov, COO at Enview, who described how this technology has been deployed on more than 7,000 miles of infrastructure last year, and is on track to be deployed on over twice that much this year. He gave a very recent example of annual vegetation management analytics performed on 230kV transmission lines in California. These lines run through suburban and environmentally-sensitive, heavily forested areas.

3D visualization of a 230kV span (blue lines) with potential grow-in (yellow) and fall-in (red) vegetation that falls within client-determined clearance boundaries.

LIDAR data was acquired by a third party through a helicopter-mounted Teledyne Optech LiDAR sensor package. Four band imagery was co-collected during the LIDAR survey and registered into an orthomosaic of RGB and NIR bands. The data collection generated almost 2GB per linear mile. The Enview data analytics engine processed this data to (i) first classify the massive amounts of LiDAR data and (ii) then automatically calculate potential grow-in and fall-in vegetation points with high speed and accuracy. Insights and data were delivered to the operator not only via GIS-compatible file format for incorporation into existing workflows but also via an intuitive web portal accessible across silos. As a result of the superior accuracy and speed, the client was able to confidently prioritize vegetation prescription work to the areas that need it most and do so much sooner than otherwise possible with traditional methods.

The results being achieved with Enview's technology suggest that we are on the cusp of the widespread adoption and application of deep learning technology to automate specific tasks. The combination of UAVs for data collection and machine learning for feature extraction dramatically reduces the cost of inspections for utilities. This would permit them to be done more frequently than with manned aircraft and would lead to improvement in the reliability of above ground electric power and communications networks.

Fully automated classification of road and highway features for autonomous vehicles

Point clouds are fine for visualization to provide context, but to be really useful they need to be classified.  Feature extraction, as the process is often referred to,  is currently a mostly manual process that has been used for many years in the remote sensing sector. A trained person visually inspects the point cloud and graphically selects features which are identified by type of feature, sign, light pole, lane marking, and so on. 

Automated classification of point clouds

Image processing of 3D point clouds has advanced to the point where it has been possible to semi-automate this tedious feature extraction process.  The reliability of the automation process depends on the point density of point cloud, often expressed as points per square meter or per square centimeter.  Point density depends on the technology used to capture the point cloud. Terrestrial lidar, for example, typically captures points at the rate of hundreds of thousands to millions of point per second and is capable of millimeter to centimeter precision. Phodar generates lower point densities and is less precise. Aerial lidar captures many points per second but with less precision because it is covering much greater areas. Single photon lidar is a new technology that can collect 6 million points per second which means that planes can fly quite high, cover very broad areas and still collect 12 - 30 points per square meter.

At the Year in Infrastructure 2017 (YII2017) conference in Singapore, Sharad Oberoi of Sanborn gave an insightful overview of the state of the art for automated feature extraction from combined aerial and mobile scans of highways and adjacent areas.  Different types of laser scanners are appropriate for capturing different types of features. Terrestrial and mobile laser scanners can capture point clouda that are suitable for extracting curbs, gutter drains, traffic signals, road and local terrain information, parking meters, walls, obstructions in the right of way, manholes, sidewalks, overhead clearances, lights, utility wires and connections, garbage cans, benches, fences, guardrails and barriers, retaining walls, and line of sight vegetation obstructions. Aerial lidar will only be able to get gutter drains, road and local terrain information, walls, obstructions in the right of way, and sidewalks with limited ability to get curbs, lights, utility wires and connections, retaining walls, and vegetation.  Lidar does not do well for metal structures such as communications towers and fir these a combination of lidar and phodar is preferred.

Sanborn has developed feature extraction technology that extracts transportation objects from combined aerial and mobile data sets in a semi-automated manner. The mobile and aerial data sets meet the accuracy and information content required for geospatial information for mapping applications as well as the additional information that can be mined for potential asset inventory and infrastructure information content. Mobile technology allows for the low-risk, rapid collection of geospatial information, limiting safety impacts to workers. The aerial LiDAR allows for the rapid collection of elevation information for detailed surface modeling as well as feature extraction. Both technologies collect enormous amounts of point-cloud data that can be merged using the Terrascan capability in Bentley MicroStation. The combined data set is then passed through the automated feature extraction modules to identify transportation objects.  Sanborn feature extraction modules achieve a high proportion of correct feature extraction for road and highway scans.

In the future Sharad expects that it will be possible to improve data collection efficiency by using terrestrial robots or autonomous UAVs equipped with SLAM technology to enable them to operate in GPS-denied areas such as under bridges and in tunnels.  Sanborn has been working with deep learning to improve the automated extraction process.  So far this has yielded mixed results, but Sharad expressed optimism that with AI full automation of the classification process will be achieved.

High precision maps for autonomous vehicles

An application of this technology is high precision maps for the autonomous vehicle market.

Sanborn develops high precision maps for the autonomous vehicle market.  Autonomous vehicles will require high precision maps which contain significantly more detailed information and true-ground-absolute accuracy than current road maps. Sanborn has developed proprietary mapping technology that leverages aerial imagery, aerial lidar data, and mobile lidar data to create standardized, high-precision 3D base-maps focusing specifically on the self-driving vehicle market. These maps include detailed inventories of all stationary physical assets related to roadways such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and all other critical data needed for the safe navigation of roadways and intersections by autonomous vehicles.