Between the Poles

An infrastructure digital twin provides a dynamic environment for managing underground assets of a multi-utility campus

For organizations managing a campus such as educational facilities, healthcare facilities, commercial real estate, utilities, and municipalities managing public works, storm sewer systems, and parks which are characterized by having many different utilities and and other infrastructure, an infrastructure digital twin provides owners, operators and third-parties with a streamlined, cloud-based solution that maps, manages and shares a wide range of subsurface and above ground assets, including fibre, pipes, electric power cables, lighting, conduits, and other structures enabling campus asset owners to not only track their assets but to provide all the information about their assets; their history, their replacement cost and their current cost without which you cannot justify your financial demands, be it a budget request for next year or a bid bond for a major project.

Infrastructure digital twin

A major challenge for legacy asset management systems is that documents are hard to access. Problems in the field are hard to identify because it takes too long to access the relevant documents and other information in the field. It's hard to share architecture and utility drawings with contractors and other stakeholders. People are really copying large blueprints or as-builts and delivering a set of documents to a new contractor who's just been onboarded and the contractor has to spend weeks studying these documents and comparing this information to what's there, what's underground, what's above ground and matching it together.

The way we address this challenge is by creating an infrastructure digital twin which provides accessible information to all stakeholders through a subscription service. A full digital twin of your campus, is a big, costly project, but an infrastructure digital twin is simpler and much faster. It involves site verification and infrastructure mapping as a service including drone imaging, site surveying, walking the campus and verifying objects. If the customer is using some type of infrastructure detection, the accurate information from that system plugs into our system, so it remains live. We support Open Geospatial Consortium standards to ensure interoperability. We onboard all of the data and we do that for any new updates on the campus. If something is wrong in the map and it's not in the site, or in site and not in the map, or if the drone gave us a better accuracy, we detect that and make corrections. For example, customers often hire utility engineering firms to conduct subsurface utility surveys (SUE).  We load that information and match it with everything we have in the system about their underground and other assets. Full onboarding requires 40 days or less. And more importantly, all the data is owned by the client all the time.

We've loaded over four million assets for our customers. Those assets are being used by thousands of users, including facilities, teams, construction managers, engineering and architecture firms, local utilities, fire departments and others.

You can share data immediately with anyone you want. And there are no limits on the number of users in any of our systems. If you onboard a new construction company for a new project, you just provide them with a login, specify what they can access and see, and they are onboard.

Dynamic view of your assets

The system provides a dynamic view of all your assets. It's not a static image, GIS or CAD files which you update every several months. It's real time, live presentation of information coming from a database and it is integrated with all your other systems. You can drill down on every single conduit, cable, pipe, or other piece of infrastructure and query all its attributes and see the sources of the data. You have access to the data in real time from your phone, your tablet, or your laptop because it is a fully cloud solution. We are able to do that because we are compliant with the Open Geospatial Consortium (OGC) and follow OGC standards. We are using open systems in terms of representing all the attributes about every item.  For a typical campus environment, ranging from a few blocks to a college campus or hospital complex, for example, you may have 20,000 assets. All of those assets and their associated attributes are captured and live in that database. We're building IOT integration, so you will have real time data on top of their various asset layers. You can see the water flow and the temperature and the pressure from the sensors. To go back to the updates. We are building integration with augmented reality. so that all underground assets can be visualized in 3D AR.

We provide a dynamic way of having base maps. You can use Google Maps, Esri World Imagery, or OpenStreetMap or even a mix of these. You can add your own imagery. 

In addition to the dynamic view of the locations and attributes of all your assets, you also have a plan room. All your documents are there, your engineering drawings, as-builts, survey and SUE reports, legacy maps, and other sources of underground data. You can search them by date or by the contractor or surveyor who supplied this data to you. These documents are accessible to you at any time from any device.

Managing asset budgeting

More importantly, the system provides an executive summary of all your assets. For example, in this site, which we have built as an example (we cannot share real data publicly) we have included about $4 million worth of assets. FM people need to deliver projects on time, minimize disruption of services to those people who are using these facilities, and improve the sustainability of their facilities. To do this requires that you can justify your CAPEX and OPEX budget. For some capital projects, you might be able to look at the total cost for adding a new building. But for budgets for operations and maintenance, retrofitting a fiber network, changing your irrigation network, it's really difficult to do that without having access to everything about your assets, their history, their replacement cost and their current cost. Without this information you cannot justify your financial demands, be it a budget request for next year or a bid bond for a major project.

The key to managing budgets is by replacement costs. If you want to replace your electric network, you need to know how you built the electric network in the first place. The database provides access to all this information. You can break your assets down by type of structure; electric power, conduits, lighting, water, wastewater and gas pipe networks. You can see their life expectancy, aggregate them based on their expected time of expiry and then decide on remedial action. You also need to worry about the impact of potential accidents and most importantly your liabilities. If a major underground gas line is damaged, then you might face heavy liabilities because it might interrupt gas service not just to your campus but for the entire area or even an entire municipality.

Our system has an ROI calculator so you can compute your potential savings. This has helped customers secure grants because they were able to compute the information needed to submit a bid bond.

Conclusion

An infrastructure digital twin provides immediately accessible information to all your assets for all stakeholders through a subscription service. It's a real time, live presentation of information coming from a database and it is integrated with all your other systems. For critical budgeting operations and maintenance, you have immediate access to everything about your assets; their history, their replacement cost and their current cost that enables you to justify your financial demands, be it a budget request for next year or a bid bond for a major project.

This post is based on Moufid Charafeddine's (BetterGIS) talk at Subsurface Utility Mapping Strategy Forum (SUMSF).

Eliminating the as-built backlog by automating field utility data capture

The as-built update backlog is the time it takes for as-builts submitted from the field after completion of construction for newly installed network equipment to get entered into your utility GIS.  It is generally measured in months which is appalling today when we have real-time or near real-time data about so many aspects of the real world from traffic to weather.

Field sourcing

The key to solving this problem is to enable crowdsourcing of updates from the people who actually do the construction, the field workers, so construction crews can make those updates where and when the work happens. In order to do that, we really need data capture tools that are much faster and simpler, in other words, more automated than today's solutions.

There are a number of enabling technologies that can change the way that we approach this problem; computer vision, machine learning, augmented reality, voice interaction, and reality capture can all play a role. A lot of these things are being driven by other industries outside the geospatial technology industry. Part of the challenge in bringing together a solution for the as-builting process is looking broadly at these different types of technologies and where they're applied and bringing them together into a solution for utilities. You need to enable the people doing the construction to capture accurate location and attribute data quickly and efficiently. Anyone should be able to use the system with minimal or no training and it should use inexpensive commodity equipment.

Automating feature recognition

As an example, we're in a new subdivision construction site and we're going to do some as-built data capture of some electric equipment using an iPad. We point our mobile device at a piece of utility equipment and the application automates the process of detecting what the equipment is. The user just taps the screen to add the feature, an electrical pedestal. Next, we point our mobile device at the neighbouring electric meter. And again, it's automatically recognized and we tap to confirm that we want to add it.  We can then scan the meter to add more detailed attributes according to rules that we've set up.

Now we want to add a cable. If the trench was still open so the cable was visible, we could automatically recognize it like we did with the other features, but since it's hidden, we select it from a list and then just tap to add points along the path of the cable. As we walk, you can see the cumulative distance of the cable path being shown at the top. Lastly, we'll add an electric cabinet in the same way and scan its attributes, including the transformer number and manufacturer on the front side. Then the KVA from the yellow 50 on the side. We zoom in more closely to get additional attributes like the serial number, work order number from the smaller label.

Once we have completed scanning the various objects, the data gets pushed automatically to the GIS with no user intervention needed. We can see the features that we just captured on a map with their attributes plus photos. The newly captured data about the equipment is immediately available in any GIS client. For example, we can see the newly captured equipment data on a handheld mobile device or it can be viewed on any web browser. The data can then be pushed on from the GIS to real-time systems like an ADMs outage management system or network management system.

The app is able to automatically recognize features and attributes using machine learning based image recognition, to tell what type of feature it is, read barcodes and QR codes, and use text recognition to capture detailed attributes. The additional attribute information also reinforces the feature recognition in some cases. YYou can also use voice recognition as a backup if there's a scenario where you have to correct something that was misrecognized or something that can't be identified correctly.

Lastly is an example of capturing very detailed attributes quickly. We captured 22 attributes in just over a second from an electric meter.

Location accuracy

Location accuracy is particularly important for underground assets. AR gives much greater relative accuracy than consumer GPS and phones do. It measures accurate distances and angles in 3D for recording of relative asset layout. In a range of tests we found AR is accurate to about 1%. As a simple example. we measured a 10 foot distance next to a tape measure. We've added a virtual pipe and you can see the measured distance in the augmented reality app is 10 inches and it's within an inch of the 10 foot mark or within about 1%. Tests of up to about 100 feet have found the same 1% accuracy. AR provides much better accuracy than you could get with anything except the very highest precision GPS.

Visual positioning system

Apple's GeoAnchors is their implementation of the AR Cloud. The idea is that as you're moving your mobile phone around among 3D objects captured in a street view scene, it uses the camera view to match what the camera is seeing against the highly accurate street view type imagery, which Apple calls look around imagery and that refines the position.

Conclusion

Field sourcing of data is key.. You need to enable the people doing the construction to capture the data, and these new technologies really open up some very exciting new ways that we can provide systems to the construction workers to enable them to capture accurate location data. We foresee that visual positioning systems are very likely going to supersede high precision GPS in a lot of scenarios.

This post is based on Peter Batty's (SSP Innovations) talk at the Subsurface Utility Mapping Strategy Forum (SUMSF).

Reducing damage to underground utilities: the French DT DICT system

In France a national system for reducing damage to underground utilities during construction was initiated in 2013.  The key features of the system are the requirement to notify utilities with the details of a proposed construction project during the planning phase, a process that improves location accuracy of existing underground infrastructure, mandatory reporting of incidents of damage, and the Observatoire national DT-DICT which publishes annual statistics. The annual statistics include the number of projects, the number of damage incidents and the ratio expressed as a percentage of damages per project.  The statistics show that while the number of projects has increased since 2013 by about 25 %, the damage per project has decreased from 0.42 % in 2013 to 0.26% in 2020. This represents a significant decline in underground damage even though construction activity has increased.

Background

In early 2000, the French government investigated damages in utility networks, of which there are something like 60,000 incidents every year. It found that the average direct cost of each damage incident was roughly 1,000 euro, the total cost including direct+indirect+social costs was estimated to reach 32,000 euro for each damage incident, and in addition there were injuries and fatalities to workers and the public. The total cost to the French economy is estimated to be on the order to two billion euros annually. According to professionals involved in public construction project, the main causes of incidents of damage to utility networks are unreliable information about the location of underground utility networks,  lack of practical and efficient tools to measure accurate locations for underground utilities, and the inability to provide real-time visualization for underground utilities.

After many very serious accidents involving fatalities in the late 2000s, the government began an initiative entitled Construire sans détruire (Building Without Destroying.)  In 2011 this resulted in damage reduction regulations. It required the implementation of an online service where all network operators were required to  to provide information about their overhead, underground and underwater utilities including their location. Furthermore they were required to classify all of their network equipment according to three quality classes. The target class is Class A which means that the location of the network facility is known with an accuracy better than 40 cm for rigid equipment like pipes and better than 50 cm for flexible facilities like cables. Class B is assigned to network equipment whose location is known to an accuracy between 40 cm and one meter. Class C is assigned to equipment who location is either not known or is known to an accuracy of one metre or lower.  The objective for achieving the Class A quality standard for all critical infrastructure (gas, electric power, hazardous fluids and water under high pressure) in urban areas was 2020. The target for all critical infrastructure in urban and non-urban locations is 2026. The target for knowing the location of all network infrastructure in France to 40 cm or better in 2032.

The DT - DICT system

At the inception of a construction project the project manager is required to prepare a DT (Déclaration de projet de travaux) a standardized document where the construction project is described.  It can be submitted online or on paper. Among other things it involves drawing a map of the proposed construction site which can be a shape file or a sketch on paper.  Once the DT is submitted all the operating companies who own network facilities in the area, have nine days to provide maps showing the location of their network equipment with the quality level (A,B, or C) assigned to each piece of equipment. The project manager will likely receive a response from 8 -12 network operating companies including electric power, gas and fuel, clean water, waste water, telecom, and other utilities.These responses enables the project manager when planning and designing the project to know where underground infrastructure is located and the reliability of the data. Unfortunately current regulations do not restrict the way this information is provided to the project manager and they can take the form of paper, PDF or a variety of digital vector formats. 

When a project has reached the stage of excavation, the project manager must submit a second document referred to as DICT (Déclaration d'intention de commencement de travaux) declaring the intention to begin work.  The DICT provides a detailed schedule of the planned construction work. In response to the DICT the network operating companies are required to notify the excavator of any changes to the network since the submission of the DT.  They also are required to verify the technical compatibility of the project with the existing networks and check the technical safety recommendations. 

The objective of the DT DICT system is to provide the project manager with maps of all underground utilities accurate to 40 cm or better (Class A). One of key activities of the project manager is to review the quality levels assigned to all the network facilities in the construction area to identify any equipment that has not achieved Class A. For any equipment assigned B or C Class, the project manager may require additional investigation to precisely position the network equipment to achieve Class A. There are rules for who is responsible for funding the extra investigations. For equipment that is assigned Class B, the project is responsible for covering the cost of any further locate work.  For equipment assigned Class C, the network operator and the project split the cost of additional locate operations 50%/50%.  If it is suspected or revealed that the maps provided by the network operator are wrong, the operating company must perform and cover the cost of additional investigations to locate the noncomplisnt equipment accurately.

Impact of the DT DICT system on underground damage

The DT DICT system has been operational since 2013.  One of the requirements of the regulations is that it is mandatory to report all incidents of underground damage.  Based on these data statistics are compiled and published annually for the Observatoire national DT-DICT.  There are several things to notice in these statistics.  First of all construction activity has increased by about 25 % since 2013.  Secondly in 2020 most of the submissions were made online.  Only 7 % of the DT submission were manual, and these were probably made my relatively small operators.

Year	Projects	Damage rate	Submitted online
2013	4,502,440	0.42%	29%
2020	5,628,251	0.26%	93%

Since the DT DICT system was implemented, the damage rate prorated to construction activity (number of damage incidents per project) has decreased. The damage rate in 2013 was 0.42%. Seven years later, the statistics show that the damage rate has decreased to 0.26% which represents a large improvement. This contrasts with the damage rates for one call systems in the Netherlands, United States, and Ontario which have remained nearly constant for the last few years,

One of the important features of the DT DICT system that differentiates it from one call systems is that the process improves the quality of the records maintained by network operators.  That together with the requirement of notifying all affected utilities of planned construction projects are probably the most important reasons for the decline in the rate of underground damage.

Challenges

The DT DICT system places heavy responsibilities on the construction industry.  These include the preparation and management of complex documents, challenging scheduling, especially for surveyors, risk of delays, and increased demand for surveyors because of the high accuracy required by the DT DICT system.  This is occurring at a time where there is already a shortage of surveyors in France.

The challenge for surveyors is to increase productivity. One way is to avoid site visits whenever possible.  Modern technology in the form of a smartphone and GNSS+RTK makes it possible for the construction crew to record the location of newly installed or exposed underground utilities.  The collected data can be reviewed later in the office by a certified surveyor.  The new technology makes it possible for contractors and network operators to comply with the DT DICT regulations without adding significantly to costs.

The future of utility surveying is that the certified surveyor no longer needs to travel to construction sites. The location of utilities in open trenches can be recorded to survey accuracy and uploaded by the construction crew who are already onsite.  With modern LiDAR and photogrammetric technology it is possible to record a point cloud of the open trench which can be stored in the cloud and contains much more information than a traditional set of survey points.  If the location of all the underground utilities in the proposed construction site is Class A, known to 40 cm accuracy or better, augmented reality technology provides a reliable way of visualizing this data in the field on a handheld device.

This post is based on Patrice Lemire's talk at the Subsurface Utility Mapping Strategy Forum (SUMSF). Patrice Lemire is the Managing Director of Geosystems France.

Keeping utility GISs up to date using automated real-time reality capture

Maintaining accurate, up-to-date, comprehensive data about assets, above ground and below ground, has been a challenge for utility and communications firms for decades. Now new technologies including machine learning and computer vision - based feature extraction and text recognition and high accuracy location capture using GPS and augmented reality clouds are being applied to automate the real-time capture of utility network information in the field during construction, when new utilities are installed and when utilities are exposed and relocated.  When fully implemented this will automate real-time maintenance and update of utility network information and form the foundation for utility digital twins.

Background

The decades-old as-builting process remains marred by manual paper-based processes that is characterized by inefficiency that causes critical processes such as outage restoration to require more time than they should and low data quality which leads to incomplete, out of date and inaccurate underground utility records that are responsible for frequent construction delays, injuries and fatalities for workers and the public, and a drag of billions of dollars on national economies.

One of the persistent themes from the very beginning of the Between The Poles was the challenge of integrating CAD and GIS data in a common workflow (some examples; 2006, 2007, 2008 ).  Critical metrics for utility readiness for the smart grid is the as-built backlog, field update backlog and the location accuracy of the assets in utility GISs.  As-built backlogs typically stretch into months, updates from the field take months to be entered into the records database (if they are entered at all), and the recorded location of assets in the GIS can be meters and even tens of meters from their actual location.   The challenge is to replace conventional paper-based workflows with real-time digital work flows that can provide the basis for utility digital twins.

Real-time reality capture using machine learning and augmented reality clouds

New technologies that have been developed for consumer-based applications include machine learning for image and text recognition and more recently augmented reality clouds for recording accurate relative locations and distance measurements are now being applied for capturing utility network information.  This week at the Mirrored World: Role of Digital Twins in the Utilities Industry virtual event, Skye Perry and Peter Batty of SSP Innovations, discussed and demonstrated how these technologies are being applied to the efficient, real-time reality capture of information about utility network facilities in the field during construction. 

Peter Batty demonstrated functionality being developed for a new product SSP Vision that supports the semi-automated capture of information about newly installed electric network facilities.  At an actual construction site Peter demonstrated automated feature recognition of electric and telecom pedestals, automated capture of attribute information from cabinets, scanning bar and QR codes and text for work order number, serial number, manufacturer and other attribute information, and using augmented reality + GPS to record the location of equipment and linear features such as a cable from the pedestal to the transformer.  All of this information was captured and pushed up in real-time and was available in seconds in the utility GIS.

The key features that are being developed for SSP Vision are automatically recognizing equipment such as pedestals and cabinets, automatically detecting and capturing attributes from bar codes, QR codes and text recognition using computer vision and machine learning. The recording of accurate locations is enabled using augmented reality clouds and GPS.  Augmented reality clouds add much greater relative accuracy including distance measurements and work both outdoors and indoors.  At this point properties like feature type can be specified manually if automated feature recognition is unable to detect the type of feature or if a new feature is being entered as part of a new design being carried out using AR in the field. AR also enables capturing linear features with support for snapping to features such as pedestals and cabinets.

Foundation for keeping a utility digital twin up to date

This is truly ground-breaking and shows how these technologies can be applied now in a semi-automated fashion to dramatically improve the efficiency of capturing location and attribute information about utility facilities during construction. The longer term vision is that the capture of this information could be fully automated by simply mounting a camera on a construction crew member's hard hat and feature recognition and attribute identification and capture fully automated requiring no manual input from the crew. This will automate the real-time maintenance and update of utility network information and form the foundation for utility digital twins.

A user platform for digital twins of underground utility infrastructure

The foundation for digital twins of cities, regions and nations are digital geometric models.  Developing these models have become a priority for a number of jurisdictions around the world. To date the emphasis has been on above ground models, primarily because of the availability of a variety of technologies for efficiently capturing high accuracy location data for above ground infrastructure. However, there is growing awareness of the critical importance of below ground infrastructure.  Several jurisdictions are integrating below ground infrastructure into their digital models.  Models of underground infrastructure are characterized by important distinguishing features that differentiate them from above ground digital models and that need to be taken into account when maintaining and using the geometric models. Digital twins are living models and need to reflect the changes in real time of the underlying assets. An on-line platform that that has been used primarily in large-scale simulations and games has a number of unique features suited for digital twins by enabling data providers including construction workers in the field, developers and users of analytics and simulations and citizen users to collaborate in real time in maintaining and using subsurface digital twins.

Background

Many cities (Rotterdam, Helsinki, Pilsen, Athens), regions (Flanders), and nations (Netherlands, Estonia, Singapore, UK) are developing 3D models as a basis for digital twins.  3D models of above-ground infrastructure include buildings, transportation networks, parks, and other infrastructure typically captured in 2D or 3D imagery from overflights or in 2D GIS maps. But for several reasons underground infrastructure is often neglected, even though it is recognized that water and wastewater, energy (gas, electric power, and district heating),  and communications (fibre and copper) networks provide the life blood of the city.

There is a fundamental difference in what jurisdictions are able to do with representations of above and below ground infrastructure. Above ground, whatever can be photographed or scanned in the public space; from the street, from an airplane or drone, or from a satellite, in 2D or 3D, is generally unrestricted in application.  It is necessary to remove recognizable people and vehicles for privacy reasons and there may be national security restrictions, but in general this data can be used without restriction to create an open, publicly available digital model or digital twin.  The accuracy of the above-ground data in digital twins is generally high - for digitally LiDAR scanned data accuracy even reaches the mm level.

Underground the situation is different. Typically there are restrictions on access to underground infrastructure location data.  In many jurisdictions some or all of the underground infrastructure in the public right-of-way is privately owned.  Where cities own water and sewer services, this data may be open and publicly accessible. But electric power, gas, and telecom are often privately owned.  Private ownership imposes restrictions on public access to this data. Accessing the data requires a legal data sharing agreement between the utility or telecom company and an entity requiring access to the data. For example, the City of Rotterdam has decided that a key requirement of its digital twin is that it is open and accessible to the public, but for below ground infrastructure there are restrictions.  About 2/3 of the underground utility data is open and accessible, but for the remaining 1/3 there is restricted access because these represent privately owned facilities.

Another problem specific to underground infrastructure is that in general each utility has a different data model, vocabulary, symbology, projection, base map, standards for layout, and data protection guidelines and harmonizing these for a shared underground utility map requires collaboration among data providers.

Jurisdictions such as the City of Rotterdam, Estonia and the U.K. are already developing geometric models that include underground and above-ground infrastructure.  For underground utilities the Open Geospatial Consortium's MUDDI underground standards project has identified important use cases for underground infrastructure location data.  The priority use cases are reducing utility damage during construction and efficiency of planning, engineering design and construction.  Other use cases include emergency response, disaster planning, and strategic planning for smart cities. Each use case has different requirements for data access, quality, currency and level of detail.

Existing databases of underground geospatial data, most of which were developed and are maintained by utilities and telecoms, are not suitable as geometric models on which digital twins can be based for a number of reasons. They are comprised of a 2D topographic map, typically with road centre lines, water bodies, building footprints and other recognizable features. Utilities either use a national base map, such as MasterMap in the UK, or have acquired a base map from a commercial mapping company. In North America in general each utility has a different basemap on which the location of utility infrastructure is superimposed. Transferring underground location data to a different base map can be a complicated process (conflation). Furthermore, each utility uses its own terminology and has developed its own data model and symbology for its displaying infrastructure maps. Most underground infrastructure data is 2D and lacks depth coordinates. Access to utility infrastructure data is generally restricted to the utility's own staff and contractors. In addition the accuracy of the data is low. It is low precision, often incomplete, and out of date. It is cumbersome to use in the field and is unintelligible to non-engineering stakeholders.

These current geometric models of underground infrastructure will require important changes to enable it to be used as a basis for a digital twin. Databases of real time, high precision underground infrastructure data are required and this is motivating fundamental changes to how data is captured and maintained. Live updates during construction and 3D reality capture with survey-grade precision are essential. Technologies that enable near real time precision data capture include LiDAR scanners, photogrammetry using handheld devices, ground penetrating radar and other underground remote sensing technology, with high precision GNSS using a base station or RTK. Being able to simultaneously capture above and below data facilitates connecting above ground visible features such as manholes, drains, risers, gas connections, hydrants, and so on with below the ground equipment. The database includes all planning, engineering and construction documents and support the integration of geospatial data from sources such as GIS, CAD and BIM.

User platform for subsurface digital twins

A platform specifically for data providers and users of a subsurface digital twin has been developed that is intended to be used after a comprehensive underground utility digital model has been built. It is not intended for the creating and building the geometric model or for new designs. It supports importing digital building and infrastructure models and provides tools for multi-user collaboration to maintain those models and keep them up to date whenever there is a change, as a result of maintenance or if there has been damage. A key requirement for this platform is scale-ability, it must support a large-scale 3D environment on the scale of a city. The amount of 3D data is growing exponentially driving a requirement for tools that will allow massive volumes of 3D data to be manipulated and visualized. The new generation of engineers are familiar with and expect three dimensional virtual environments, whether it's gaming or simulation software. Within the next 10, 20 years, they will be the ones who manage this infrastructure, so providing them with an interactive 3D environment including mixed reality where they feel comfortable provides them with an environment that best enables their skills, has to be undertaken now, because building out digital twins in three dimensional view is going to take years. 

The platform does not limit the number of simultaneous contributors who could be construction workers on the ground who want to update the model, city planners and managers running analytics and simulations, and citizen users who want to visualize and understand the results of the analytics and simulations. For all users the platform provides the most up-to-date and accurate information to enable workers to use it in near real-time. It can connect to live feeds from IoT sensors in the field to be able to monitor conditions in real-time.   Data is versioned to enable tracking changes in the data over time. The platform provides an interactive environment supporting mixed reality applications.  It runs in the cloud where it is universally accessible in a fully protected, secure environment. In addition data protection rules protect data custodians and prevent viewing of network information that could provide competitors information about a company's network.

Building a persistent environment is extremely important, because every participant, whether a user or a data provider, needs to be able to contribute to growing the environment and seeing in real time what everyone else is seeing. For example, if a telecom company begins installing a new fibre cable and at the same time the local water company is looking at that model when designing a new water pipe, the water designers will be able to see the new fibre cable in near real time. Being able to see edits in real-time makes it possible to edit the model with several users collaborating simultaneously.  Everyone can see everyone else's changes and conflicts can be avoided.  Some utilities are collecting point clouds captured with LiDAR or photogrammetrically of newly installed equipment before covering the trench and these can be used in a mixed reality environment to maintain and make modifications to existing infrastructure.

The benefits of a living digital model are better and faster decision making.  According to the CEO of one Swiss multi-utility company, not knowing exactly where the pipes are means that "As soon as we start digging, about 90% of plans have to be thrown away or redrawn, because we find stuff we didn't know actually existed underground." This refers to Switzerland where there is a very rigid replacement cycle requiring about 3% of roads to be replaced and refurbished every year, but even with this they are still missing quite a lot of information. A reliable, up to date model allows you plan for maintaining or replacing existing infrastructure, precisely where there is free space to avoid unnecessary relocations and avoiding disruptions to existing infrastructure that is delivering vital services to the communities. Better planning means fewer disruptions of services to the public. Better planning also results in faster project startup, fewer and less extensive utility relocations during construction, fewer injuries to workers and the public, fewer project delays and more on time and on budget completions for construction projects. Over time this produces substantial cost and time savings. Reduced risk of utility damage also has indirect benefits. For example, it can lower insurance costs.  A scaleable, 3D platform with high precision, real time data can also support other use cases such as proactive equipment maintenance, disaster planning, emergency response and other things that can improve the resiliency of infrastructure. Such a platform supporting users of a subsurface digital twin will minimize the impact on the local communities from any kind of disruptions that are potentially going to interrupt services.

Digital twins of underground infrastructure are living models of real world objects and require a platform and tools that enable a large number of data providers and users including construction workers in the field, developers of analytics and simulations, and citizens to collaborate in maintaining and using the models.  The key is to integrate live data from all assets and ensure that the geospatial systems modelling these networks are a ‘living document’ constantly drawing on current data and intelligence from the field.

This post is on Alex Shalash's and Antoine Castel's talk at the Canadian Underground Forum (CUF). You can listen to all the talks at CUF on the GeoIgnite CUF Youtube channel.

Study shows augmented reality improves productivity of underground utility locators

Mixed reality applications for visualizing underground utilities in 3D have been available from many companies for years, but have not been widely used. There are signs this is changing. For example, vGIS is an augmented reality (AR) visualization application that integrates traditional BIM, CAD, GIS and other types of 2D and 3D data including Autodesk BIM and CAD models, Bentley BIM designs and Esri ArcGIS data, into augmented reality visuals to provide survey-grade accuracy on consumer handheld devices.  A recent study has quantitatively assessed the benefits of applying AR to underground utility locate operations.

Background

In most jurisdictions locating underground utilities is performed just prior to excavation by network operator staff or by commercial professional locators in response to an information request submitted to the local one call centre. Locates involve remote sensing devices such as ground penetrating radar, electromagnetic wands, and other devices augmented by maps of underground utilities provided by network operators. The result of a locate operation is marks on the ground, paint or flags. The locate may be independently validated in a quality assurance process, often by the contractor. It is estimated that $10 billion is spent annually in the U.S. on underground utility locates. Any technology such as augmented reality that can improve the efficiency and effectiveness of underground locates needs to be seriously assessed. The application of AR to locate work provides an opportunity to improve the process and outcomes including productivity, accuracy and safety.

While it may seem obvious that being able to visualize underground utilities in 3D on-site would be highly beneficial, there are practical obstacles that hinder realizing the full benefits of the technology. A major issue is that much utility GIS data is incomplete, out-of-date and and inaccurate. Secondly, environmental conditions, terrain, and accessibility specific to a locate job may make the application of AR impractical.

Assessing the benefits of AR for underground utility locates

To assess the practical benefits of applying augmented reality to underground utility locates a leading Canadian locate services provider, Promark-Telecon, has joined with the developer of a leading augmented reality solution vGIS, to initiate a study to measure the quantitative benefits of applying AR to underground utility locates. The study was conducted in the city of Toronto, a major metropolitan setting containing a diverse mix of personal housing, industrial and commercial buildings. In addition to different types of buildings, Toronto possesses an assortment of new and aging infrastructure, some well-documented and others sporadically documented. The infrastructure located included gas, water and sewer. The study was conducted from January 2018 through May 2018. During that period, the locators participating in the study recorded the time that it took to perform locate tasks with and without AR. In addition to the time measurements for each locate job, the study participants recorded the technology’s impact on several other factors including safety, locate accuracy, and any issues encountered.

Augmented reality involves a smartphone or a tablet that has a screen, camera, GPS and an accelerometer that detects camera motion and orientation.  The real world "background" is comprised of data, including location, of real world objects or visible objects .  For example, in the context of construction virtual objects representing a model of underground infrastructure can be superimposed on a map showing existing conditions or a map of surface structures such as roads and buildings.  The augmented reality application superimposes the virtual design objects or the underground infrastructure on the map of existing objects or the surface map. In this case 2D data showing the location of underground assets are projected onto a photographic view of the surface.

Using an iPad, locators in the field used the vGIS application to access city 2D GIS data and overlay it on the 3D view of the surroundings captured by the on-board camera.  A calibration process was used to ensure that the GIS data and the view captured by the camera are aligned. 

The Toronto study evaluated the impact of AR visualization technologies on utility locates conducted in real-life conditions. By collecting empirical data over the course of several months, the study measured the effectiveness of applying augmented reality technology to utility locates using multifaceted criteria, including productivity, safety and locate accuracy.

Time savings from using AR for underground utility locates

The time savings identified by the Promark locators ranged from negligible to four hours. For just under 90% of the jobs, the time savings were significant ranging from 10 minutes to more than 2 hours. The average time saved was 30 minutes per job.

Time savings from using AR as a percentage of the total job duration

Perhaps more meaningful is the proportion of total job duration saved by using AR. In 40% of jobs AR reduced job duration by a half. In a third of the jobs locators realized even larger time savings of 66% or more.

The study did not identify statistically significant differences based on the infrastructure type (gas, water and sewer). Informal feedback suggested that the more accurate and complete the location information about underground utilities was, the greater the time savings. In other words accurate utility records amplified the benefits of the AR visualization system.

The study evaluated differences in savings based on different types of jobs, such as industrial, residential, and other types of locates. The comparison revealed that the time savings for validation jobs were significantly higher than for other types of jobs, ranging from 66% to 90%, suggesting that AR is exceptionally effective for quality assurance jobs.

On average, utility locators using AR reduced the time required to complete jobs by 50%. At the same time, QA validation time was reduced by 66–85%. This translates to cumulative savings of 12–20 hours per locator per month.

The impact on the ease of doing a job was assessed by asking participants to evaluate each job using a three-level scale whether AR made the job easier, made the job harder or had no impact. In 84% of jobs, participants reported that AR made the job easier.

AR and safety

Underground utility locates can be hazardous work, especially in heavy track.  This study assessed the impact on safety of using AR for underground utility locates. For 85% of jobs participants reported that AR had no impact on safety. In 11% of jobs, AR increased safety, while in 4%, AR had a negative impact. The notes provided by the participants suggested that the negative impact on safety was experienced when locators strayed into hazardous situations while concentrating on tracing lines using AR.

Regarding the issue prevention aspect, participants reported that AR helped prevent issues in 49% of cases. Issue prevention was experienced during validation locates where supervisors would “test” validated locates using AR, giving them confidence about the completed job. In 4% of jobs locators reported issues resulting from inaccurate utility data and unreliable base maps that misled locators.

Conclusion

The study has quantified the practical benefits of visualizing existing underground infrastructure in augmented reality for underground utility locates. The benefits were found to be significant; utility locators using AR reduced the time required to complete jobs by 50% and QA validation time by even more 66–85%. The approximately 10% productivity boost recorded by the study confirms the potential of augmented reality for the locate industry including network operators. There was no apparent downside to using AR in the field, for example, AR had no significant impact on safety. The quality of utility GIS data was found to be a key factor in maximizing the benefits of applying AR to underground utility locates.

Augmented reality enables simplified design workflows

Engineering design is typically done on a desktop computer, using CAD or BIM software. Models are viewed in 3D or 2D on a monitor. Increasingly they can be viewed in mixed reality.  At a SPAR3D conference three companies in the construction sector, Gensler (a collaborative design firm), McCarthy (a commercial construction firm), and Martin Brothers Construction (a heavy civil construction firm), demonstrated how they were using mixed reality on real-world projects.  For example, Martin Brothers demonstrated building a bathroom pod with Hololens. The bathroom pod was designed using a standard design application such as Revit or Sketchup, then projected onto the reality of a shop floor were it was constructed using the full-scale virtual reality projection of the design into the shop. The next step is bidirectional between the mixed reality environment and the authoring design tool. This makes it possible for changes made in the mixed reality environment to be visible in Revit or Sketchup where they can be incorporated into the design. It is also possible to reverse this process where design changes made in Revit or Sketchup are simultaneously visible in the mixed reality environment.

Stéphane Côté of Bentley has taken this one step further and suggested that the design and visualization process should really be a continuum involving design, markup, and visualization.  To enable this to happen Stéphane's team developed a simple set of drawing tools along with features that enhance drafting accuracy, such as angle, length and element snapping.  But most innovative was the ability to snap to real world objects in augmented reality.  This is illustrated in a video by designing an extension to a table.  To do this the designer simply snaps to the table in augmented reality and then designs the extension seeing exactly what it looks like including accurate measurements such as lengths and angles.

As a real world example, when an engineer needs to design a bypass system between two existing pipes, normally he/she would have to take measurements of the pipes or scan them using a laser scanner. These measurements would then be imported into a CAD application where they would be used as the basis for the design.  It might even be necessary to iterate this process once or twice to ensure the accuracy of the design.  With an augmented reality design tool such as that created by Stéphane's team the design could be done on-site, avoiding the initial measurement step and being able to see immediately if the design fits accurately between the existing pipes.

AR enables better grid damage assessments and improves quality of utility GIS

Maintaining accurate, up-to-date GIS data has been a challenge for utility and communications firms for decades.  Inefficient CAD/GIS workflows result in utility GIS quality degrading with time.  Now with business drivers such as implementing a smart grid, determining service availability and other applications making GIS quality a top priority at an increasing number of utilities and communications firms, new technical solutions are being applied to implement new business processes that raise the quality of utility GIS data over time.  These include CAD/GIS integration; integrated engineering design, GIS and enterprise services; extensive geospatial open source libraries  for viewing, editing and analyzing geospatial and other data; and reality capture often together with AI for inspections, distribution and transmission vegetation management, asset inventories and creating network data models. Another example of a technical advance that can contribute to improving GIS quality over time is a recent proof of concept by Duke Energy, Verizon and EPRI in which augmented reality (AR) was applied to improve the reliability of damage assessments after major events.  While successful in making damage assessments more reliable, Duke found an unexpected result that AR also improved the locational accuracy and description of assets in the GIS.

Broken business processes degrades GIS data quality

 Over the years the lack of CAD/GIS interoperability has cost communications and utility companies hundreds of millions of dollars trying to cleanup their GIS data and streamline their CAD+GIS workflows.  For many utilities and communications firms traditional GIS/CAD business processes, which have been the standard approach for maintaining utility GIS for the past few decades, has simply not been able to maintain cost-effective, real-time, high quality asset location and associated data.  Instead GIS data quality tends to degrade with time.

Challenge: improving the reliability of damage assessments

At Distributech 2019 Aleksandar Vukojevic of Duke Energy together with Paul Giers Duke Energy, Micah Tinklepaugh EPRI, Norm McCollough EPRI, and John Simmins (now Associate Provost for Research and Economic Development at Alfred University) reported on a proof of concept at Duke Energy the primary objective of which was to apply AR to improving the reliability and trust-worthiness of damage assessments after major disasters, but found an unexpected benefit in contributing to improving GIS data quality.

The objectives of the trial were to investigate the application of AR to reduce errors in identifying and characterizing missing and damaged equipment and as a result improve trust in the assessments by the work crews assigned to carry out the repairs required to restore power. It was envisaged that AR could improve damage assessments by visualizing before-storm images over post-storm damage thereby simplifying identifying missing or damaged assets. Currently field technicians visit the area of the outage and use paper and pencil to assess and record damage. Some of the challenges associated with this process is that field technicians, especially less younger, less experienced ones or office staff seconded to the field, may not be knowledgeable about the secondary power grid and in assessing damage to it. The result is missed and incorrect identification of missing or damages equipment. In addition paper processes are slow and error prone making ordering the required replacement equipment a major source of delay in restorating power.  The result is a lack of trust in damage assessments by field crews.

Proof of concept: Applying AR to improve reliability of damage assessments

The trial involved a safety helmet mounted RealWear HMT-1 Android computer with an eye-level display and microphone. It incorporated a GPS so that location was recorded automatically. The HMT-1 was connected via WiFi to router in a truck that connected via a Verizon wide area network to a cloud server.  The field engineer views the "before picture" based on the location data and is guided to align the high resolution, stabilized HMT-1 camera to match the same view to capture a corresponding "after picture". Voice commands and voice responses to damage assessment questions presented in the application are utilized for a fully hands-free user interface. As the engineer performs the work, data is immediately sent to the back office for evaluation and action. The images and data are used to create a list of equipment and inventory required to restore missing and damaged equipment.  The trial did not involve an actual outage but simulated a major outage.  17 different sites were involved in the exercise. 

Results of the proof of concept

Early results indicated that nearly 100% accuracy rate was achieved in damage assessments using AR.  This represents a 50 % improvement over the existing, paper-based process. This resulted in much greater trust by work crews in the damage assessments.  In addition financial benefits were estimated to be significant. The first 12 hours response spent manually recording and transferring assessment reports was sped up dramatically. It was estimated that for an outage affecting 250,000 customers over a four day period, the time to assess and restore power could be reduced by 12 hours with a cost savings of over $ 8 million. One of the big advantages is that training only took 10 minutes even for a worker with no damage assessment experience.  Assessments were delivered to the back office in near real-time, much faster than for paper reports and eliminating the "middle man" who transcribed paper reports into a computer based system reduced the number of errors.

Side benefit of using AR: Improved GIS data quality

While Duke assessed the proof-of-concept to be successful for damage assessment, they also reported an unexpected but important result of the proof of concept.  The process using AR for identifying assets helped improve the quality of utility GIS data compared to the existing paper-based process that could actually result in degrading the quality of the GIS.

In conclusion Duke was extremely happy with the proof of concept. Achieving 100% accuracy even for inexperienced assessors impressed the Duke Energy team and created much greater trust by the work crews in the assessment reports. But AR had an important side benefit - Duke found that using AR to accurately locate and identify assets had the potential to contribute to improving the reliability of asset data stored in the GIS.  This is another example of the application of technology which change business processes and result in improving the quality of GIS data over time.

Augmented reality aids disaster damage assessment by inexperienced utility workers

One of the challenges of disaster response for utilities is situation assessment which involves sending utility personnel to the areas affected by a storm to assess the state of utility infrastructure. Skilled staff are often in short supply at such times and office and other staff without field experience are often assigned to perform damage assessment. These staff find it difficult to perform a reliable situation assessment because of unfamiliarity with the equipment they are looking for. At Distributech 2019 in New Orleans Rodney James of Duke Energy and John Simmins of EPRI described research they have been doing to use augmented reality (AR) to help with damage assessment by using specially equipped glasses to superimpose a map of the utility's infrastructure on the real world making it easy for an inexperienced staff member to see where equipment has been destroyed or otherwise affected by a storm.

Augmented reality used to require large heavy headsets or helmets but now there are a number of manufacturers of specially equipped light-weight glasses that can be used for AR. Duke Energy uses Realwear HMT-1 glasses that can be attached to the helmet that utility field workers are required to wear and that are safety rated. The glasses are smart running the Android operating system and are able to connect via a wide area network for real-time video and audio making it possible for someone back in the office to view what the staff member in the field is seeing. However, this can be a drain on the battery, so the setup can work with just audio communications as well.

I have blogged previously about Duke Energy's MapView mobile GIS application which enables anyone with a web browser to see a queryable map of all of the utility's infrastructure. A worker equipped with glasses sees MapView superimposed on the real world showing the location of all utility infrastructure with appropriate labelling. This enables an untrained staff member to determine very easily, quickly and reliably which equipment has been carried away by a flood, for example.  Of course this assumes that the utility's GIS data is accurate and uptodate.  I have blogged about this problem many times and about the recent research at EPRI to apply deep learning to improve the quality of utility GIS data.

Rodney James described other AR applications of the technology as well such as supervised routine field maintenance enabling new field workers to be mentored from the office by an experienced engineer.  It appears that we may soon see some practical applications of AR in the utility space deployed in the field.  A question then will be will all field staff including linesmen be equipped with glasses or just some.  In Duke's experience field staff have been enthusiastic about the glasses.

Intriguing new technology developments for improving construction productivity

Throughout media day at the Year in Infrastructure 2017 (YII2017) conference I was on the edge of my chair because there were such tanatalizing hints of spatial technologies that might address major challenges of 21st Century construction. We were inundated with a phenomenal amount of information about new technology developments and by examples of how these developments are actually being used by customers in the real world.  It is breathtaking to experience how rapidly new technology is being developed, rolled out and applied in the real world. This is happening in diverse disciplines as evidenced by the projects nominated for the roughly 20 different domains that are broken out for Be Inspired Awards categories at YII2017.

Today's presentations and the latest press releases absolutely intrigued me and I was tantalized by what I heard. The presentations stimulated many questions about the new technologies that Bentley was developing and I was very relieved to be able to sit down and chat with Francois Valois, Senior Director of Software Development Reality Modeling at Be‎ntley. Real world experience has shown that reality modeling is relevant to every part of the construction lifecycle. As a result reality modeling has grown rapidly in importance at Bentley and now has its own software development group. It joins four other groups which reflect the construction business cycle; design,  analytics, project delivery, and operations.  We chatted about a number of different topics that had been discussed during the day's sessions.

Comparing reality and design

During a highway construction project, estimating how much earth has been cut and filled is critical because it determines for a large part payments to contractors. With laser scanning and UAVs it is now becoming possible to scan an entire project with greater frequency. Converting the scanned point clouds to reality meshes produces a manageable 3D image of the status of the construction project at a given time. Seeing changes from one week to the next, for example, requires a tool that can compare two reality meshes and identify where differences exceed a specified threshold. Depending on the type of project and the 3D scanning technology this could be millimeters or centimeters. Bentley has developed a product called ContectCaptureEdit which allows two reality meshes to be compared. The difference between two reality meshes captured a week apart can be used to compute quantitatively the volumes of cut and fill on a highway or rail construction project in the last week. This approach can also be used to monitor construction progress for buildings and civil projects. Bentley's vision for the future is to enable comparing not only two reality meshes but also a reality mesh and a design model. This would automate the process of detecting divergences from design which is essential for assuring that what is designed is built.

Classification of reality meshes (aka feature extraction)

Feature extraction is a methodology that has been used for many years to identify and classify objects in 2D images captured by remote sensing using cameras on satellites or fixed wing planes and helicopters. In the utility sector, classification is still a manual process with some exceptions. The advent of 3D scanning technologies including lidar and phodar (capturing many photos with a consumer digital camera) makes possible the really differentiating focus at Bentley on classifying 3D objects. 3D classification can provide more information than 2D feature extraction because the extra dimension allows you to determine, for example, the depth of cracks in concrete or the extent of deformation damage to a transformer. Bentley is applying deep learning and other AI technologies provided by the Microsoft Azure cloud platform to enable semi-automated and even completely automated classification of reality meshes. For example, Bentley is working with a partner SiteSEE to automatically classify scans of communications towers captured by UAVs. With a smart UAV and automatic classification, the process of inspecting communications towers could be fully automated, saving a lot of time and increasing reliability by enabling more frequent, less expensive inspections.

Reality modeling for substations

As an example where automated classification is feasible and could provide significant benefits is in creating digital models of existing (brownfield) substations. Currently the process is quite laborious. The substation is scanned using terrestrial lidar or phodar with the option to include lidar or digital cameras on UAVs. The reality mesh generated from the scans is then classified manually – which involves selecting recognizable objects and identifying them as transformers, holding tanks, pumps, buildings, or other types of object. The final step is to link each object to an entry in an asset database – such as Bentley's AssetWise. Bentley refers to this process as “operationeering”. Operationeering makes it possible to use the classified scan plus the linked database – possibly together with augmented reality - to improve operational efficiency. The link between an object's location and the information about the object stored in an asset database esnsures that maintenance staff have immediate access to everything – location and condition - they need to know when there is a problem.  Or it could be used to simply conduct an inspection.

There are tens of thousands of brownfield substations in the U.S. If the classification step could be automated, and if QR codes are pasted to objects in the substation before scanning, the entire process could be semi-automated saving a tremendous amount of tedious and time-consuming effort.

Interoperability

Large construction projects invariably involve CAD, BIM and other types of products from multiple vendors. For example, Revit, Bentley BIM, Tekla, Sketchup, and others may be used on the same project. At YII2017 Bentley announced iModel 2.0 and iModelHub which enable consistent interoperability between different vendors' products. For example, changes made in Revit can be synced as “change sets” through iModelHub to products from other vendors. Bentley developed an iModel for Revit two years ago and iModelHub is a generalization of the concept to other design products. iModelHub is a complementery product for ProjectWise, which is a file based system widely used in construction for sharing project designs and related information.

Connected data environment enabling full lifecycle management

Perhaps the most tantalizing technology concept advanced during Greg Bentley's session was the connected data environment (CDE) which conceptually is intended to enable data to be reused throughout the entire lifecycle, As Francois explained it to me, it is designed to enable roundtripping data between ProjectWise, which is used during design and construction, and AssetWise, which is intended for optimizing asset performance during operations and maintenance. Reusing data collected during design and construction for operations and maintenance remains extremely rare. If CDE provides a foundation for doing this, it could represent a first step in resolving a major challenge hindering full lifecycle data management.

During today's sessions there were many tantalizing hints of the things Bentley is doing to address major business challenges like enhancing project collaboration, automatically classifying 3D scans, and enabling information sharing across the full construction lifecycle. I am anticipating further elucidation about these very exciting developments in the next few days at YII2017.