Between the Poles

Automated feature recognition helps democratize utility mapping for producing infrastructure digital twins

In jurisdictions worldwide the low quality of underground location data has been identified as a top cause of underground utility damage. Existing underground utility data, most of which is stored in network owners' records, is often inaccurate, out of date or simply missing.  In North America in a survey of 402 locate professionals better facility maps received nearly unanimous agreement as a top strategy for improving the efficiency and quality of locates.  Singapore's Digital Underground project is unequivocal about the economic and social impact of low quality underground location data stating that "virtually all stakeholders are aware that much of the available information is unreliable and that this has repeatedly led to losses of time, money and opportunities."

Challenges facing the utility mapping industry

The root cause of the low quality of underground utility data is how utility mapping is structured as an industry. The primary activities of the utility mapping industry are data collection and how we keep and share this data with other parties.

Data collection

The way we collect is slowly changing, but predominantly we're still using traditional survey and locate methods. The survey methodology we are using hasn't changed in a hundred years. It's a very time and materials based, manual and analog process. Survey processes are slow and expensive to execute and require specialized equipment and skill sets, and safety is a big concern.

Locating underground utilities is one of the biggest causes of delays in construction schedules, service disruptions are a common occurrence, safety is always a major concern. and locate operations are slow to execute. Accuracy and timeliness are major issues. Locate operations require specialized equipment and skill sets which restricts locators to a small number of industry insiders.  A major shortfall is that underground utility location data is rarely available during the crucial planning and design stages of civil engineering projects.

Democratization of utility mapping

The utility mapping industry needs some fundamental changes. We need to move away from the monopoly that exists right now. There is just a handful of companies that are able to go out and collect underground utility data. We need to democratize data collection so that many more people can collect this data, targeting a crowd sourced approach and enabling multiple creators or collectors providing this data. This distributes the responsibility for the data as well. The more distributed the responsibilities, the lower risk and the lower the cost of failures will be.

We also need to change the skill level required. With current technology, we should be able to automate much of routine locating and thus enable many more people to do it. Using inexpensive technology will lower the cost of collecting and aggregating locate data. As data becomes lower cost to capture, data retention will become less expensive which will encourage people to retain the location data they acquire rather than simply painting/marking the ground and not capturing the data digitally.

Democratization is the real future of 3D infrastructure mapping. Right now data is still very siloed. We'll always have the traditional mapping companies, survey companies, line locating companies and asset owners going out and collecting their own data. But through crowdsourcing data from routine day to day locating and mapping, infrastructure projects, and other sources, aggregating that data and distributing access to that data is how utility mapping will really come to the forefront.  It should be easy to upload data into a central register where it can be made accessible to multiple stakeholders. Compliance with Open Geospatial Consortium (OGC) standards provides a basis for full interoperability enabling open access to every single conduit, cable, pipe, or other piece of infrastructure in real time from your phone, your tablet, your laptop or other applications.

Infrastructure digital twin

We think the best paradigm for producing and capturing asset data is through collecting it into an infrastructure digital twin. Digital twins have made a lot of progress in a variety of industries. Most importantly, for the utility mapping industry, it should be noted that creating a digital twin is much easier than it was even just a few years ago. We think the time is right for an infrastructure digital twin and it can be just at the asset level for one particular company, but we believe it should be at a regional level. Cities, municipalities and regional governments around the world are beginning to develop comprehensive digital twins of all of their physical infrastructure; buildings, roads, trees, power lines, and subsurface utilities and they're pouring resources into it.

And we think that Canada and the United States should start doing the same thing. And for us in the utility industry, data collection costs are lower than they ever have been. There's clearly interest among stakeholders and governments to start dealing with infrastructure and utility mapping problems, particularly to lower the number and costs of strikes, injuries and the loss of life. We now know a lot more about how to create data from other data sets, using different types of analysis and artificial intelligence to piece disparate data sets together. Using AI machine learning in the cloud is much less inexpensive compared to just a few years ago. And nowadays one of the things that makes infrastructure digital twins readily accessible to the utility industry is the fact that storing huge volumes of data in the cloud has become very inexpensive.

What is infrastructure digital twin ? First of all, a digital twin needs to result in effective action. It's not just enough say we have a digital twin philosophy The purpose of having a digital twin is to result in effective action that improves your business. It needs to use real time and historical data, which is paramount for us in the utility mapping industry, and it's going to be motivated by outcomes. For us in the utility industry, it is to reduce and even eliminate utility strikes during construction and the associated risks of injuries and fatalities.

The infrastructure digital twin has to be stored and managed in a relational database so it can be queried, analyzed and improved on through time, and related to other data sets. Processes for collecting data and maintaining data must be simple so that we can break the barrier of restricting underground utility mapping to a specialized group of people. If we truly want to democratize and compile data from disparate data sets, we need very simple processes to get data into the database, analyzed and maintained.

An infrastructure digital twin should be captured and maintained in 3D, even though most utilities in the ground are within just a couple of meters of the top of the ground. The 3D component is vital, because part of analyzing a data set is that much ancillary data is 3D such as overhead power lines that come down the side of a pole into conduits below the ground,

Imagery that's taken looking straight down at the ground, like traditional satellite imagery, can be visualized on a map and incorporated directly into a GIS database. But other types of imagery such as oblique, 360-degree, street-side, and inspection imagery need to be treated differently.  Oriented Imagery has a range of possible applications. For example, a construction company might use it to review asset inventory photos taken by field workers at a construction site. A telecom company or electric utility would use it to examine close range inspection images of electrical and telecom towers from many different angles. Oriented imagery applications enable you to see exactly where the image is on the map as you pan or zoom in the image. There are spatial navigation tools designed to work specifically with oriented imagery. You can query oriented images using spatial and temporal filters.

Finally, it is very important to have standardized formats, including storage formats and cloud formats.  It improves our ability to inexpensively use machine learning to do associative analyses of disparate data sets to make a larger, more conflated data sets.

Automated feature extraction

Maintaining accurate, up-to-date, comprehensive data about assets, above ground and below ground, has been a challenge for utilities for decades. Now new technologies including machine learning are being applied to automate real-time maintenance of utility network information and provide a foundation for infrastructure digital twins.

Automated feature extraction has advanced dramatically in the last few years.  This is an example where high density LiDAR and ultra high resolution imagery was used on a very low, very small diameter gas distribution install from an open trench. You can clearly see the ease with which the LiDAR produces a 3D representation of all of the local utility infrastructure.

Much of the the advancement in in automated 3D mapping have derived from the autonomous vehicle industry which have trickled down to other parts of the GIS industry.  The hardware and software are vastly easier to use and produce vastly better data. You can see from these ortho images collected from our system and the LiDAR overlaid on it. Feature extraction was used to automatically define top of pipe, top of ditch and actual joints in the pipe. Automated classification of LiDAR removes the need for someone to go out there with a Pogo stick and find the top of pipe. It makes it possible to simply drive alongside and scan with a mobile LiDAR system an open trench containing a newly installed pipe to produce a detailed 3D model with centimetre accuracy without having a person present. It is much safer - no boots on the pavement - and it drastically reduces the cost.

We prefer mobile mapping systems as opposed to handhelds because mobile mapping systems produce a tremendous amount more data and its registration tends to be vastly superior. It is much safer because it eliminates a need for someone to get out of a vehicle - so no boots on the pavement. Anytime you're still getting out, whether you're using handhold laser handhold mapping system, or a Pogo stick, you still have a person outside the vehicle walking around a street. With a mobile system you eliminate that risk, and you also drastically decrease the time required to do the collection. Here is another example where automated feature extraction was used to identify the electro fused coupling. You can see the butt joints, you can see the Ts and so forth. All this can be done very simply now with automated feature extraction.

Data registers

While surveyors may have had experience with a provincial or other types of survey register, utility location data identified during routine locate (one call) operations in general is only used to mark the ground and is rarely recorded, digitally or otherwise. Practically this means locators going back to the same location they were at just a few weeks ago to repeat the locate operation for another project. People in the locate industry are very familiar with this and the  amount of schedule and cost expense it causes. Exposing people to relocate or remap an area that was just done not too long ago is doubling their risk. Secondly, because the data is not retained or kept in easily accessible ways, we can't analyze it.  For example, we might want to confirm that we have correctly identified the type of utility, the accuracy of its location, or determine if there have been changes in its location, type, or current status since we last detected it. And because the data is not being retained and centrally managed at a global level or at a large scale level, there's no real opportunity for organic data enhancements or improvements.

The CGA Next Practices Report to Industry has made the case that a comprehensive national GIS map of buried infrastructure is among the opportunities for systemic improvement with the greatest ROI for industry. The benefits of a national register are greater safety, enhanced construction efficiency; reduced design, construction, and maintenance costs; greater regulatory compliance; improved locate efficiency; fewer project delays and budget overruns; fewer service disruptions to the public, less traffic disruptions and loss of custom and other inconveniences for the public.

Sharing maps of underground infrastructure is not totally new to Canada. For example, the City of Calgary's Joint Utility Mapping Project (JUMP) system, the ICI Society in British Columbia, the City of Toronto's Digital Map Owners Group (DMOG) and others have been sharing underground utility information for years.

The Canadian Underground Infrastructure Register (CUIR) is a new national initiative started at the beginning of the year. The CUIR initiative is intended to enhance current practices in a way that enables all stakeholder groups to benefit from the initiative; construction contractors, network owners, government transportation agencies, design engineers, subsurface utility engineering (SUE) engineers, surveyors, locators, workers in the field and the public. The CUIR initiative has been inspired by the National Underground Asset Register (NUAR) project currently being implemented in the UK. The objective of NUAR is to develop a register of location data for all underground infrastructure in England, Wales and Northern Ireland that is accessible to all network owners and government agencies.

This post is based on Joseph Hlady's (Lux Modus) talk at Subsurface Utility Mapping Strategy Forum (SUMSF).

Democratizing reality capture for underground pipeline and utility infrastructure

A revolution is happening in how we map new underground infrastructure. Instead of the paper as-builts submitted after the completion of construction for a new pipeline or other infrastructure, network owners may now request a LiDAR scan of the newly installed equipment before the trench is filled.  There are several reasons for this.  First and foremost is that LIDAR scanning has become so easy and fast that it is literally accessible to everyone. Secondly, a LIDAR scan not only captures the survey-grade location in 3D of new infrastructure, but a lot more information besides.  Thirdly, LIDAR scanning has become so inexpensive that it does not really add to construction costs. With the democratization of modern reality capture technology the availability of comprehensive, high accuracy 3D location data about underground pipelines, utilities and other infrastructure is growing.

When Leica Geosystems released the BLK360 laser scanner, this was revolutionary for several reasons;  it was small,   it only had one button, its price was unheard of in the world of professional laser scanners, and the result of a scan could be seen a few minutes later on an iPAD. This device was all about democratizing reality capture. The BLK360 was Leica's play to make professional-quality reality capture available to anyone who wants it.

Utilities have used LiDAR and photogrammetry for a number of applications involving above ground infrastructure, for example, for vegetation management and to reduce safety issues and costs when conducting a high accuracy inventory of above ground infrastructure.  But this changed in 2018 when Lux Modus entered the pipeline mapping market with the objective to change field mapping fundamentally by making the capture of high accuracy maps of underground infrastructure accessible to anyone and at a low cost. Consisting of high-density digital LiDAR, three 13 megapixel cameras, with GNSS with base station correction the Lux Modus device only requires one touch to initiate scanning which frees the user from having to manage the system while driving around the collection area.The device can be mounted on a rig on a pickup truck and by simply driving the truck along the line prior to filling the trench captures survey-grade location information and a lot more information such as depth of cover, location of weld points, and nearby utilities. The point cloud and images are uploaded for cloud processing and within a few minutes a 3D image of the new section of pipeline can be viewed by anyone with a browser that supports HTML5 including mobile devices. 

There are two use cases where LiDAR scanning can be applied two underground infrastructure. The first is when new utility infrastructure is installed and before the trench is covered.  Some jurisdictions are already mandating survey-grade accurate as-builts for new installations.  The second case is when existing infrastructure is exposed for repair or relocation.

The underground mapping industry has been facing two historical challenges.  The first is that when new network infrastructure is installed by a contractor, the so-called as-builts that are submitted to the utility after completion of construction rarely reflect the actual location of what went in the ground. The problem is that calling a survey crew and then redlining the original design to indicate where equipment was actually installed is expensive in time and cost. Furthermore, contractors are remunerated based on how rapidly they can get new equipment in the ground, not on the accuracy of the as-builts they submit.

The second problem is the result of the first. Most of the existing location data for underground utilities contained in utility and telecom network records is inaccurate, out of date or just missing. In responding to one call locate requests, utility locators have to rely on these same utility records augmented with remote detection techniques to try to detect and identify their network equipment. Contractor often have to  conduct daylighting operations using vacuum or hydraulic excavation equipment to verify what utility locators believe they have have identified and painted on the ground. Line locating and daylighting requires special equipment, software and training to do things efficiently. But even with experience detecting and identifying underground network facilities remains a slow, error prone process that can expose workers to the hazards of traffic and other safety risks and can play havoc with construction schedules.

A number of jurisdictions, realizing that the capture of point clouds is accessible and low cost, are dispensing with traditional paper as-builts and requesting LiDAR scans instead.  As a result of Lux Modus this is happening in the pipeline industry in Western Canada. Lux Modus started mapping pipelines because that was the hardest use case they could find. Now Lux Modus is expanding into other sectors such as rail, power transmission, and urban mapping first in Canada and the United States and soon in Europe. Recently in Denmark two water utilities in the Greater Copenhagen and Aarhus areas are efficiently generating point clouds of underground utility infrastructure using a handheld phone and accurately surveyed control points.  These point clouds are captured for both newly installed infrastructure and for infrastructure exposed during repairs of existing infrastructure.  Novafos, a water company in the Greater Copenhagen area, already has thousands of these point clouds which capture not only high accuracy location of the water utility's underground infrastructure, but a lot more information including the location of nearby utilities which would be a concern for any excavation involving the water lines. 

This post is based on Joseph Hlady's keynote at the Canadian Underground Forum (CUF). You can listen to all the talks at CUF on the GeoIgnite CUF Youtube channel.

Cloud service for capturing and sharing point clouds of underground infrastructure

Modern reality capture technology is not only providing a high accuracy alternative to paper as-builts, but is also being used to capture the location of underground utilities and other infrastructure exposed during during excavation or exploratory digging.  LiDAR scanning or taking photos or videos with a handheld together with high accuracy GNSS equipment such as RTK or accurately surveyed control points makes it possible to achieve survey-grade accuracy for underground utility infrastructure without a survey team on site. In France a new commercial GéoCassini cloud service is available from e-Cassini that captures, processes, integrates with GIS and other geodata sources and shares accurately georeferenced point clouds of underground utilities. Furthermore the e-Cassini MarketPlace enables customers to publish and market their data stored in a GéoCassini database to the broader construction community.

Background

Without accurate maps of underground infrastructure, every construction project has the potential to become a disaster site. Incidents of underground utility damage are more frequent than many people realize. In the U.S. there are between 500,000 and 800,000 incidents every year. Every year there are injuries and fatalities attributable to hitting underground infrastructure during excavations. Inaccurate and missing information about underground infrastructure is also responsible for construction project schedule and budget overruns. According to the Federal Highway Administration (FHWA) missing or inaccurate location information about underground utilities is a major source of highway construction project delays.

Utilities have used LiDAR and photogrammetry for a number of applications involving above ground infrastructure, for example to reduce safety issues and costs when conducting a high accuracy inventory of above ground infrastructure.  Until recently modern reality capture techniques have not been applied to underground infrastructure.  But in the last few years this has begun to change. Since 2018 Lux Modus has applied LiDAR together with an accurate georeferencing device to achieve survey-grade location accuracy in "near real-time" as pipelines are constructed. As another example AVUS have developed a reality capture solution that allows the creation of as-builts accurate to ± 5 centimetres from a video taken with an Android smartphone. Together with GNSS + RTK base station or previously surveyed control points.

Commercial cloud service

In France a new GéoCassini cloud service is available from e-cassini that commercially captures accurately georeferenced point clouds of underground utilities and allows this data to be integrated with other geo data such as orthophotos, traditional GIS maps, panoramic imagery, above ground imagery from LiDAR and photogrammetry, satellite and aerial imagery, and associated metadata. I had the opportunity to speak with Patrick Maïore, founder and director of e-Cassini.

GeoCassini is a cloud-based marketplace for various types of geospatial data including underrground utilities.  The data is owned by the utility which captured it and is made commercially available to transportation agencies, construction contractors, other utilities via the GéoCassini cloud services.  For example, standard part of the planning exercise for an engineering design firm would be to consult GéoCassini to see what underground and other data is available for the intended construction site.

Furthermore GéoCassini data can be integrated and visualized in augmented reality using vGIS. e-Cassini has a technology partnership with vGIS. For example, vGIS has been used to visualize underground networks and property boundaries stored in GéoCassini for a new housing development.

Applications

One of the most recent applications of GéoCassini is capturing LiDAR scans of open trench utilities.  These could be newly installed utilities or utilities exposed during repair or relocation work.  For example, GéoCassini has been used to capture using a handheld and share location information about above and below ground data for a 200 km water network for a water utility operating in several regions of France (Herblay, Mery-sur-Oise, Saint Leu, Nanterre, Vaucresson, and Suresnes.) The video capture was performed with a handheld app that was originally developed for Android and will soon be available for iOS. The data includes point clouds captured by mobile LiDAR and utility GIS maps of underground utilities.

The process used for reality capture is simple; the field operator walks along the trench and captures a video of the trench and the video is uploaded to GéoCassini where it is transformed into a point cloud. A few minutes later an analyst in the office receives a notification that the point cloud is available. The analyst georeferences the point cloud using control points captured in parallel with the video or from preregistered control points. Accurate GNSS+RTK points can be captured at the same time as the video is recorded. Alternatively, accurate control point scan be included in the field of vision of the video. Some public works companies carry out a preliminary survey of the site using LiDAR to develop a cloud of control point clouds. Another alternative is a service offered by GéoCassini to subscribers are control point clouds in the municipalities where they regularly work. The geo-referenced point clouds are saved in the GeoCassini database and all projects that have been linked to the point clouds are automatically updated.  The information is shared amongl all stakeholders on each project.

The point clouds are used in many applications; drawing a proofing plan of the construction site, visualizing underground networks in 3D using augmented reality, and correcting and updating 2D data in the utility GIS are typical. But a big benefit is that they provide survey-grade location for underground infrastructure which enables precision engineering design with respect to the underground and the above ground.

e-Cassini has participated in two pilots in which the GéoCassini approach has been applied. In the first pilo,t which involved replacing aging house connections to the water network, after the new equipment was installed and before the trench was filled, a field technician captured a video of the newly installed equipment. The video was uploaded to GéoCassini where it was transformed into a point cloud and accurately georeferenced. The resulting georeferenced point cloud provides a high accuracy 3D image of the newly installed equipment available to all stakeholders of the project and potentially commercially available to other interested parties via the e-Cassini MarketPlace.  The second pilot involved installation of an entirely new water network for a new development. The network was installed in sections. After each section installed and prior to filling the trench before re-filling the trench, a video was recorded, uploaded to GeoCassini, converted to a point cloud and geoferenced.

e-Cassini's technology has been implemented by Veolia, a large multinational water company, and by Bouygues, a large construction company.  Veolia has conducted pilots using the technology in three regions of France, Normandy, Marseilles, and the Loire.  Veolia has applied for and is expecting regulatory approval from the local D.S.P.s, which are equivalent to public utility commissions in North America.

Bouygues Energie et Service is currently experimenting with a GéoCassini solution for the renewal of public lighting network equipment in the municipality of Pornichet. GéoCassini is being applied to improve diagnostics of existing equipment and to optimize the location and type of new equipment.

Conclusion

A benefit of the GéoCassini cloud service is that it enables the capture and sharing of survey-grade as-built records of underground infrastructure. For new infrastructure this provides a reliable record of what was installed in the ground, in other words, survey-grade "as-constructed" instead of "as-designed".  For existing infrastructure this enables discrepancies and omissions to be reported back to asset owners.

Another important benefit of the GéoCassini service is that it if very efficient and does not add to construction costs. Compared to the time required to call a surveyor to the site, video recording requires negligible time. Furthermore, the video approach does not interrupt construction work or affect scheduling and it works in most weather conditions, even at night in artificial lighting.

Applying inertial locating technology to mapping and condition assessment of large deep assets

Knowing where things are underground has become important enough that in a number of jurisdictions around the world including the Netherlands, Singapore, and the U.K., initiatives are underway to create national digital twins which include below-ground infrastructure.  Recently there have been important technical advances that are have brought the feasibility of cost-effective mapping of underground infrastructure within reach. The most accurate way to detect the location of underground infrastructure is to (carefully) dig a hole to expose it and then bring in a survey team to survey the location.  But this is time consuming and expensive. It has been estimated that the average direct and indirect cost of a pothole is $30,000. Depending on the technology used to dig the hole it can also he hazardous. Current best practice for remote locating underground infrastructure is walking the site with electromagnetic wands (EMI) or ground penetrating radar (GPR) pushcarts. This is slow, expensive, and hazardous.

Most construction projects are only concerned with the underground utility and other facilities that lie within a few meters of the surface.  But there is another class of subterranean objects that are large and lie deeper.  These include rail, large scale water transmission, and large diameter storm, sanitary and combined sewer systems. The traditional approach to investigating such facilities is CCTV.  These are TV cameras connected to a cable for power and image transmission that are guided down pipes.  While such systems are low cost, they don't capture location data, they have limited range requiring multiple deployments from manholes, and assessment and reporting is manual and arduous.

I have blogged previously about inertial locating and mapping, which relies of accelerometers, gyroscopes,  on devices with memory for recording output from the sensors and a battery to power the device.  Such devices do not require a cable connection for power and data transmission. Inertial mapping technology is composed of an Orientation Measurement Unit (OMU) that contains inertial sensors. The OMU is battery powered and the data logged is stored internally during a measurement run. Unlike other technologies for tracking underground facilities the device does not need to be traced from above ground as it progresses through a pipe.  The device also contains odometers recording the speed of travel.  After a run post-processing software converts the data recorded by the OMU and odometers to an accurate 3D model.

Now a recent a startup in the UK, Headlight AI, has developed a system that builds on inertial technology but adds significant capabilities for mapping and condition assessment of large deep assets.  I had a chance to chat with Dr. Puneet Chhabra, Co-founder and CTO at Headlight AI. Headlight AI's product Telesto integrates inertial technology with wireless communication, LiDAR, cameras, and lights to capture location and high precision 3D point cloud data that allows assets to be mapped and high-resolution LiDAR and imagery to be captures at high frequencies as the device progresses through an underground asset.   Wireless communication enables real-time monitoring of the progress of the device at shorter ranges. In addition, a machine learning application has been developed which helps automate the detection of different materials and identifying and locating defects such as corrosion, calcification, wall deformations, and other common problems.

Photograph of deployment of the system during the data collection phase of the BCIS survey and (right) examples of 3D point cloud produced from the trials

An example where the technology has been deployed is the Bournemouth Coastal Interceptor Sewer (BCIS) project, which is 8 km 2.2 m diameter sewer deep underground with flowing water in some sections.  The BCIS is a critical asset managed by Wessex Water. Its concrete structure was constructed between 1964 and 1971 with an internal diameter of 1,800mm. Due to its location and its surrounding geology it is particularly vulnerable to water infiltration and biogenic corrosion from hydrogen sulfide.

The BCIS is regularly surveyed by Wessex Water to ensure its effective operation. Surveying methods have included CCTV condition assessments, laser profiling surveys, and calibrated LiDAR surveys. However, this traditional surveying method requires person entry and traversing of the sewer by a team of operatives due to the need for calibration targets and accurate registration of the 3D point cloud. Hence, this method requires substantial assessment and planning in order to accommodate the higher health and safety risks imposed on workers.

Wessex Water’s priority in the management of its sewers is health and safety. In many cases access shafts into sewers can be located every 1000m or so, which is concerning if operatives are to enter such dangerous environments. Therefore, person entry into the confined spaces of a live sewer, such as the BCIS, are minimized or avoided wherever possible to ensure enhanced process safety.

3D point cloud of section of BCIS; the image shows how the software can focus on specific parts of the point cloud and measurements can be taken by the user.

Using Headlight AI's technology required 2 weeks to collect the data across 7,692 m of the BCIS, including 13 shaft scans at various positions and depths along the sewer. For the sewer alone 250 GB of data and over 2 billion points were collected. Headlight AI's software made it possible to automatically distinguish different wall materials in the point cloud using a combination of traditional and deep learning algorithms. Furthermore, machine learning helped to identify and locate common defects for follow-up assessment.

The project has enabled approximately 8 km of the Bournemouth CIS to be mapped in 3D without the need of operatives to traverse the sewer. The cost of the Telesto survey was similar in cost to laser profiling and CCTV surveys, while being significantly more informative and less hazardous to workers health and safety. This project has demonstrated that Telesto has the potential to improve the quality and the health and safety of sewer inspections across the water and wastewater industry.

In a very recent development it is now it possible by combining Telesto information with terrestrial surveys and shaft scans to map the key sections of deep sewers in x, y, and z. Further future benefits will be the ability to compare how the 3D structure evolves over time and to conduct change detection. This will enhance predictive maintenance and allow water utilities to make more informed decisions regarding both operational and capital expenditure.

Automating scan-to-BIM for brownfield substations

With about 55,000 substations in the U.S. and the average age of the transformers in these substations over 40 years old, many substations are going to be refurbished, retrofitted, extended or upgraded. Utilities are opting to create digital twins of their existing substations as part of this process to enable maintenance and upgrading activities to be performed more efficiently. For older substations records may consist in outdated 2D paper drawings or documents may be missing altogether.  New digital scanning technology together with machine learning is making it possible to semi-automate the process of creating intelligent BIM models from point clouds derived from laser scanning or photography.

primtech is a CAD-based design tool for designing high voltage substations.  An important recent enhancement Optical Symbol Recognition (OSR) allows point clouds from laser scanning or photography to be imported and then to apply machine learning to semi-automate the process of creating an intelligent BIM model. 

The software includes an extensive library, which contains more than four thousand intelligent 2D and 3D electrical devices from different manufacturers, steel structures, foundations, and wires and tubes.  The library has been growing for over 15 years and includes new equipment and elements as well as for legacy assets.  Each symbol (device, steel or foundation, etc.) has a specific geometric signature in the database in addition to meta data, file links, connection points and defined object behaviours. Together these comprise smart objects that enables objects in point clouds to be recognized and converted into an intelligent substation information model. The resulting intelligent model provides the basis for the calculations required in the substation engineering process.

The OSR process consists of two phases; first conductors are created and then equipment is added. A wire can be created by simply selecting three points, first anchor point, maximum sag point and end anchor point. For tubes, two or more points can be selected. Wires and tubes are available in the library for different conductor types according to international standards.  After conductors has been created and placed, the OSR tool provides a list of compatible library symbols for each symbol type. It is up to the user to choose which symbol should be selected. The user can select the perfect matching symbol or another one in the list. A similar workflow applies to all the other symbols like support structures and foundation heads. The system then combines the symbols to create assemblies (electrical device, support structure, foundation head) and places them in the model. Once an assembly has been created, the OSR tool will recognize identical assemblies in the point cloud and propose it for those positions in the substation. The wires are automatically attached to the respective electrical connection points on the equipment upon the insertion of the assemblies.  After the OSR tool has examined all objects in the point cloud and they have been mapped to an assembly in the model, a logical and intelligent primtech BIM model has been created. The model can be used as a full-fledged template for extension or retrofit - to create parts lists and plans as well as to carry out calculations including phase checking, clearance calculation, lightning protection calculation, sag and short-circuit effects calculations.

The new OSR technology is not fully automatic but drastically improves efficiency and reliability of converting point clouds into intelligent 3D models.This is an important advance that could dramatically improve the process of scanning brownfield substations to intelligent models by making the process more efficient and less error-prone.

M. Kokorus, F. Pizarro, W. Eyrich and S. Heuser, "From Optical Symbol Recognition (OSR) of Point Clouds to the Substation Information Model," 2018 IEEE/PES Transmission and Distribution Conference and Exposition (T&D), Denver, CO, USA, pp. 1-4., 16-19 April, 2018

Automating transmission line inspections with BVLOS drones and machine learning

Transmission line inspections for vegetation management and other purposes are essential in ensuring grid reliability and resilence.  They are generally performed by manned helicopters or by a ground crew.  Data is collected with cameras and analyzed to detect diseased trees that could fall and hit a powerline (fall ins) or trees that were encroaching on powerlines (grow ins).  These inspections are mandated by NERC in North America and are not optional.  Beyond visual line of sight (BVLOS) drones (UAVs) and software analytics using machine learning has made it possible to approach completely automating this expensive process.  Automating transmission line inspections for vegetation management using BVLOS drones not only saves money but also could improve the resiliency and reliability of the transmission grid.

Mapping transmission lines with BVLOS UAVs

At ILMF 2019 in a very interesting talk Paolo Guzzini of Delair described how BVLOS drones has been deployed to map transmission lines in remote areas of Iceland.  Delair conducted two surveys on behalf of EFLA Consulting Engineers, a general engineering and consulting company. EFLA needed to map existing power lines and survey an area where Landsnet, the main transmission system operator in Iceland, planned to build a new power line between two cities. Iceland's airspace regulations allows BVLOS UAVs to be used in areas that are not densely populated. Delair deployed fixed-wing drones to gather two types of data: photography and LiDAR.  The drones were programmed to fly a predetermined flight plan so the pilot had to only monitor the drone's progress and be available for emergencies.  Delair’s drones were able to capture comprehensive data along the planned route and deliver the processed results to the client the next day — all at a fraction of the cost of hiring a helicopter or ground crew.  UAVs with a range of 110-120 minutes flew for up to 10 km per flight. In one case the UAV flew a total of 70 km with a 21.4 megapixel photo camera to capture a corridor of 200 m width to an accuracy of 0.5 m vertical and 0.2 m horizontal accuracy. The actual vertical accuracy achieved was 0.15 m.  In a second mission a UAV flew at 100 m a corridor 10 km in length with a LiDAR camera to capture a corridor 200 m wide with the objective of vertical accuracy of 0.5 m and horizontal accuracy of 0.2 m. The actual accuracy achieved was 4 cm horizontally and 2 cm vertically. The advantage of using LiDAR is that it can achieve better accuracy in location especially in the vertical dimension. In addition the LiDAR approach was able to detect detect conductors and poles.  In both cases the range of the flights was limited by weather conditions, radio range (15 km), and weight of the camera payloads.

Reality capture for vegetation management of transmission lines with BVLOS UAVs

Periodic transmission line inspection is mandated for North American transmission lines by NERC.  Inspections are currently carried out by manned helicopter or ground crews which are expensive and for that reason not done more often than is required by NERC.  Full automation, which is possible with BVLOS UAVs and automated feature extraction of lines, pylons, vegetation and ground, would dramatically reduce the cost of inspection and enable more frequent flyovers.  The major benefits would be lower costs and a reduction in the number and length of outages resulting from vegetation transmission line interactions.

In the U.S. on July 15, 2016, President Obama signed the FAA Extension, Safety, and Security Act of 2016. The 2016 Act requires the FAA to establish a process to allow certain UAV operations related to utilities, pipelines, and oil and gas production to be conducted beyond the visual line of sight (BVLOS) of the operator and either in the daytime or nighttime.  As a result FAA rules permit beyond visual line of sight (BVLOS) UAVs through Part 107 waivers.  I blogged over a year ago about Xcel Energy which obtained one of the first three waiver to fly UAVs for beyond visual line of sight operations.   Paolo related that 32 of these waivers have been granted and that more than 20 companies are operating beyond BVLOS drones but still with some restrictions.  Paolo related that the FAA very recently began allowing BVLOS operation using radar to track the location of the UAV.  Paolo expects to see many long-distance UAV operations in the near future, making drones fully competitive against helicopters.

Over 20 countries have regulations that permit BVLOS UAV flights. In some areas of the world, Delair has even been able to control flights via a 3G data connection, allowing a drone to inspect up to 30 miles in a single flight.

Post-processing

Paolo told me that Delair's post processing software uses a combination of machine learning and other technologies to classify LiDAR point clouds with 85 to 90 % reliability in order to identify potential vegetation encroachments.  I have blogged about Enview and other companies that also offer software solutions for this problem.

Conclusion

Rapidly evolving regulation to permit BVLOS operations, Delair's successful deployment of BVLOS drones with radio or 3G communications and automated post-processing, often cloud-based, to identify potential vegetation encroachment indicate that we are on the cusp of being able to completely automate this type of transmission line inspection.  Using drones and automated feature extraction software not only dramatically reduces the cost of these inspections but makes possible more frequent inspections including on-demand - for example immediately after storms - which would impact positively the reliability and resilience of the transmission grid.

Automating vegetation management for electric power transmission lines using deep learning with LIDAR

With FAA rules permitting beyond visual line of sight (BVLOS) UAVs available through Part 107 waivers the cost of capturing the data required for vegetation management for power lines, whether transmission or distribution, could drop dramatically.  However, manual post processing of the imagery to identify areas of vegetation encroachment, grow-ins and fall-ins, remains an error prone, tedious and time consuming process.  I have come across several firms (for example EPRI, SiteSee, Trimble RealWorks) over the years that have been working towards automating the process for identifying vegetation encroachments including grow-ins and fall-ins, but I have not encountered a utility that is using this technology operationally.  At Distributech 2019 in New Orleans I had a chance to chat with a startup Enview, based in San Francisco,  who described to me how their software has been implemented  over 7,000 miles of transmission lines last year to automate the identification of vegetation encroachment from LiDAR point clouds.  Furthermore, they expect the total distance covered to double this year.

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that utilities even have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment which typically are carried out by manned helicopters.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. I have blogged about Energisa in Brazil who used satellite stereo imagery to prioritize and optimize tree pruning and ComEd's pilot project in Chicago that uses LiDAR imagery to not only prioritize and optimize but also to change the pricing structure for procuring distribution tree trimming.

Enview applies machine learning, computer vision, and geospatial big data analytics to highly reliable feature extraction from LiDAR point clouds to identify the exact location, height and extent of trees and then to determine proximity to power lines and other utility equipment.  Utility GIS data can be integrated and results visualized using colour-coding to identify the level of risk for each tree.  This enables the utility to prioritize their tree trimming and optimize contractor routing.  In environmentally sensitive areas the visualization can be used to inform discussions with the various stakeholders responsible for different land uses.

At Distributech I had a chance to chat with Krassimir Piperkov, COO at Enview, who described how this technology has been deployed on more than 7,000 miles of infrastructure last year, and is on track to be deployed on over twice that much this year. He gave a very recent example of annual vegetation management analytics performed on 230kV transmission lines in California. These lines run through suburban and environmentally-sensitive, heavily forested areas.

3D visualization of a 230kV span (blue lines) with potential grow-in (yellow) and fall-in (red) vegetation that falls within client-determined clearance boundaries.

LIDAR data was acquired by a third party through a helicopter-mounted Teledyne Optech LiDAR sensor package. Four band imagery was co-collected during the LIDAR survey and registered into an orthomosaic of RGB and NIR bands. The data collection generated almost 2GB per linear mile. The Enview data analytics engine processed this data to (i) first classify the massive amounts of LiDAR data and (ii) then automatically calculate potential grow-in and fall-in vegetation points with high speed and accuracy. Insights and data were delivered to the operator not only via GIS-compatible file format for incorporation into existing workflows but also via an intuitive web portal accessible across silos. As a result of the superior accuracy and speed, the client was able to confidently prioritize vegetation prescription work to the areas that need it most and do so much sooner than otherwise possible with traditional methods.

The results being achieved with Enview's technology suggest that we are on the cusp of the widespread adoption and application of deep learning technology to automate specific tasks. The combination of UAVs for data collection and machine learning for feature extraction dramatically reduces the cost of inspections for utilities. This would permit them to be done more frequently than with manned aircraft and would lead to improvement in the reliability of above ground electric power and communications networks.

LiDAR changes the cost structure for utility distribution vegetation management

Vegetation is a significant source of outages for many utilities. In some regions from one quarter to one half of all outages can be ascribed to vegetation.  Vegetation encroachment initiated the massive North American northeastern outage of 2003.  It is so critical for utility operations that utilities may have a vice president responsible for vegetation management.  It can be costly for utilities to identify areas of high risk vegetation encroachment.  Scheduling and routing crews to prune the trees that require attention is often sub-optimal adding to the cost of vegetation management.  Optimizing these processes can produce significant benefits in the form of fewer outages and reduced costs. I blogged previously about Energisa in Brazil who used satellite stereo imagery to prioritize and optimize tree pruning. At Distributech 2019 in New Orleans, Sainab Ninalowo and Dan Marron from ComEd described a pilot project in Chicago to use LiDAR imagery to not only prioritize and optimize but also to change the pricing structure for distribution tree trimming.

Vegetation SAIDI

Currently ComEd trims 25% of their service area every year so that their entire service area is covered every four years. In addition they perform reactive trimming - when they have a vegetation-related problem reported they dispatch a crew to resolve the problem. Statistics show that the number of vegetation-related outages (vegetation SAIDI) has been declining year over year. The contractors who do this work are currently paid by the circuit mile independent of tree density, so that a contractor working in a rural or suburban area with many trees is paid the same per mile as a contractor in an urban area with few trees.

Fall-ins: yellow doughnuts Grow-ins: orange within 0.5 ft, amber within 0.5-1.5 ft, light yellow within 1.5 to 5 ft

ComEd has had experience using LiDAR for vegetation management for its transmission lines and decided on conducting a pilot project to apply the technology to its distribution network. It chose circuits in four areas in its service territory with a range of tree densities ranging from rural to urban. ComEd put the capture of LiDAR data and post-processing of the point clouds out to competitive bid and selected two vendors. For the LiDAR imagery acquisition fixed wing aircraft were used. Post-processing involved combining the distribution network from their GIS with the vegetation identified in the point cloud imagery and modeling cylinders around the cables to reflect the tolerances for vegetation encroachment (grow-ins and fall-ins).

Using the volume of vegetation encroachment in the cylinders it was possible to calculate the volume of tree trimming required. This made it possible to procure tree trimming services with a different pricing model, based on per cubic foot instead of per circuit mile.  This change in procurement modus enables contractors to more accurately estimate labor and equipment costs and ComEd to prioritize their tree trimming based on the needs revealed by the data.

In addition to LiDAR cameras the aircraft were equipped with IR and hyperspectral sensors. The IR imagery made it possible to successfully identify weakened and unhealthy trees. The hyperspectral data was used to try to identify different tree species, but the results were not able to this reliably enough to recommend using this technology operationally.

The pilot has been deemed to be a success, although there are not yet direct cost comparison figures publicly available. In addition to the perceived benefits for vegetation management, ComEd has identified other significant benefits which can be derived from the LiDAR data. Based on the results of the pilot it is being recommended that LiDAR be phased in gradually for the entire network with the goal of conducting a LiDAR scan of the entire distribution network annually.

This is another example of how geospatial technology is fundamentally changing the utility industry, in this case how vegetation management services are procured.

Towards a digital twin of infrastructure: efficient reality capture of new pipelines

Traditionally when a new pipeline is put in the ground, either its location and depth is only approximately known or it has to be surveyed by a registered surveyor.  Now there is another alternative. At GeoAlberta 2018 is Edmonton Joseph Hlady of Lux Modus described how his company has developed a rig and software running in the cloud that allows this information to be captured in "near real-time" as the pipeline is constructed.  The rig includes LiDAR and photo cameras which capture the trench and the pipeline by driving along each section before the trench is filled to record a point cloud and photos.  The point cloud and images are uploaded for cloud processing and within minutes to hours a digital twin of the pipeline can be viewed by anyone with a browser that supports HTML5 including mobile devices.

By automatically detecting the top of the ditch and the top of the pipe, the location (x,y) and depth (z) of the pipe can be computed in real-time by the LuxGear or later in a GIS.  Lux Modus uses the Unity gaming engine to enable 3D walk-throughs in virtual reality (VR) of the pipeline after the ditch has been covered.  For example, this makes it possible for QA/QC people to use VR to identify weld points in the office rather than in the field.

Documenting construction projects with a consumer 360° camera

Many see handheld scanners as revolutionizing the construction sector.  Portable and handheld laser scanners offer millimeter accuracy and can be used for engineering purposes.  Cameras with less precision are being used for rapidly documenting a construction site for a variety of purposes.   Most recently a couple of startups Cupix and HoloBuilder are lowering the bar by offering cloud software for the AEC industry to enable images of a construction site to be captured with a consumer grade 360° camera like the Ricoh Theta S or V. Walking through a construction site capturing 360° images along the way and uploading the images to the web is all that is required to create 3D walk-throughs of a site.  The 3D walk-through can be used to measure dimensions, it can be annotated, it can be compared with a BIM model, and 3D images captured at different times can be compared to monitor construction progress.

 

For engineering purposes, handheld laser scanners provide the requisite precision but are relatively expensive.  The scanning process requires time and produces large point cloud files and imagery that need to be managed and processed for visualization.

• The Leica Geosystems BLK360 weighs a mere 2.2 lbs (1 kg), is incredibly small 6.5 x 4 inches and lists for US$16,000.  It is a full laser scanner and imager, capturing 360,000 points/second with a range of 0.6 - 60 meters.  The scanner can be attached to a Microsoft Surface Pro tablet to visualize what you have scanned. Up to 16 of these can be used at the same time.
• Dot Product DPI-8 and DPI-8X (short range) are multi-sensor devices (IR laser and photo cameras).  Targeted at professionals, especially in the engineering and construction sector, the devices incorporate SLAM technology and works with Android handhelds.  All the processing is done locally on the handheld - no cloud processing is required.
• The Faro Freestyle can scan objects within a range of half a meter to 3 meters away with a resolution of  better than 1.5mm with a image point density of up to 45,000 points/m² at 0.5m and up to 10,500 points/m² at 1m.  The scanner is attached to a Microsoft Surface Pro tablet for visualizing the point cloud as you scan.

For rapid site documentation, Matterport allows you to scan a floor of a building in an hour or less, upload the captured data to the cloud (Amazon Web Services) and a couple of hours later have a fully-rendered 3D walk-through that can be accessed over the web.  It was originally targeted on the real estate sector but recently has been showing up at construction industry and other events including the recent BIMForum in Las Vegas.  It provides a dollhouse view to see a whole floor at once, an inside view for walk-throughs, and a floor plan view for a traditional top-down perspective. It relies on IR and photo cameras and multiple scans in each room - scanning every 1.5 to 2.5 meters.  It is capable of rough measurement, but not for engineering or construction purposes.  Its biggest advantages are the speed with which you can scan a site and the small size of the resulting file which has to be uploaded to Matterport's cloud site.  In addition to real estate and construction site documentation, it is used by FM professionals who need a 3D context for facilities management.  For example, it can provide a mechanical room walk through with tags on different pieces of equipment and a way of directing anyone to any location in the facility.  The camera is several thousand dollars and cloud processing requires a subscription.

 

The most recent additions to the technologies that can be used to document a construction site are Cupix and Holobuilder which lower the bar by only requiring a consumer grade 360° camera such as a Ricoh Theta.  In the case of Cupix cloud processing requires a subscription.  Similarly to the Matterport, you walk through the site capturing images with 360° camera as you go.  The resulting images are uploaded to the Cupix site and the result is 3d tours or walk-throughs accessible over the web.  You can compare 3D tours taken on different days to monitor construction progress, compare a 3D walk-through with a BIM model (Revit or IFC) you have uploaded, and measure dimensions with a precision of 1% or better at 20 meters.  To reach this level of precision in measurement requires including a marker when taking pictures.  Alternatively if a dimension is known it can be used to scale the entire walk-through.  Cupix can use images captured indoors and outdoors.  Cupix says that if GPS metadata is available from the photos, they can georeference the walk-throughs and generate a KML file. I used Cupix with a low cost 360Fly4K camera and a Samsung Note 5 phone to capture the interior and a bit outside of a cabin in a nearby park.  It took a couple of attempts to get the correct spacing between camera placements in capturing 360° images, but after that it was straightforward. I uploaded a rough picture of a floor plan and I was able to walk-through the resulting 3D tour using the Cupix web viewer.  Without doing anything additional I was able to get rough measurements accurate to about 5%.  Cupix only requires a consumer grade 360° camera.  A subscription is necessary for cloud processing. Cupix is targeting a broad market including travel and hospitality, real estate, and arts and culture in addition to the AEC industry

 

I just came across HoloBuilder at the recent BIMForum in Las Vegas.  Like Cupix it only requires a consumer 360° camera. At first glance it appears to be focused on the AEC industry and makes creating and managing 3D tours of construction sites very easy.  It offers 360° SiteStream which is a unique live streaming solution for real-time construction progress monitoring in 3D that runs as a plug-in app on the Ricoh Theta V.