Between the Poles

Ground-breaking initiative by Colorado DOT for capturing and sharing location of underground utilities

According to the Federal Highway Administration (FHWA) missing or inaccurate location information about underground utilities is a major source of highway construction project delays and budget overruns.  Some jurisdictions have taken advantage of their legal right to control what goes into public right-of-ways.  I have previously blogged about Calgary, Alberta and Las Vegas, Nevada as two examples in North America, which have attempted to control the accuracy of location information about underground utilities including telecom.  The chief challenges they faced have been finding a solution to address this issue cost-effectively all the while providing survey-grade accuracy and creating QA/QC processes for ensuring that this information is accurate, up-to-date and secure.  In general this requires a multi-pronged approach involving government legislation and regulation, modern technology and partnerships between different levels of government, utilities and telecoms, and the construction companies and contractors.

I had a chance to chat recently with Rob Martindale, Utility Program Manager at the Colorado Department of Transportation (CDOT) and Chairman of the GIS and Data Management Technical Committee for the American Association of State Highway and Transportation Officials (AASHTO) and Page Tucker, CEO and founder of Prostar Geocorp. Rob told me that in Colorado the legislative framework for managing and sharing utility information about underground infrastructure has been developed and implemented.  Using a Cloud and mobile solution developed by ProStar Geocorp, CDOT has implemented ProStar’s Precision Mapping Solution ™ to be used by CDOT as well as all of the utilities, telecoms, contractors and other stakeholders in Colorado to capture, display and share the location of utility infrastructure that lies below the earth’s surface. According to Page Tucker, ProStar’s Precision Mapping Solution leverages Open Geospatial Consortium (OGC) standards including Web Mapping Services (WMS) and Web Feature Services (WFS) and is also designed to support quality standards such as ASCE 38-02 in the US and CSA S250 in Canada. The ProStar Solution is natively built to be open source using GeoServer and seamlessly integrates with other modern technologies and legacy systems of record including Bentley, ESRI and GE Smallworld.  It also uses standard map services such as Google Maps, Bing, and OpenStreetMap for base mapping as well as LiDAR and drone imagery when higher levels of precision are required.  The Prostar Solution even integrates with One Call centers allowing the State One Call center to provide One Call tickets in real-time, so that it is possible to display the location of all open tickets and their status on a PC or mobile device such as a tablet or smart phone. The solution can even auto-populate digital locate forms based on the new One Call tickets.  The Prostar solution is also secure. Page Tucker told me that the ProStar Solution is SOC2 compliant and can use roles and privileges in the settings as well as geo-fencing to restrict views of operator networks, which is a key requirement for telecom operators.  Century Link is one of several key beta testers on the Colorado state implementation and many other utility owners including Zayo Communications have also expressed an interest in participating.

There are two components to the ProStar Solution. Transparent Earth® which is the Cloud platform and PointMan® which is the mobile application. PointMan runs on both Android and iOS and can use the device's internal GPS for mapping grade data collection as well as external GPS and RTK to identify the location of exposed or newly installed underground infrastructure in the field with precision levels that can range from a foot to centimeter accuracy without requiring a survey team.  The PointMan application according to Mr. Martindale is both intuitive and very easy for field crews to use.  Furthermore, construction contractors that have adopted the solution report that it does not add to their current workload and is already saving them time.

At this point the shared database does not include utility and telecom as-built data but Prostar incorporates support for bidirectional inclusion of this data with notifications to utilities when any of this data is changed so that it can be used for maintaining and improving utility as-builts.

This is a ground-breaking initiative involving legislation, regulation and new technology to address one of the chief causes of highway construction delays and cost overruns. CDOT's adoption of the ProStar Solution is now getting the attention from other State DOTs as well as several Federal agencies.  Utah DOT for example followed CDOT and has also implemented the ProStar Solution. 

New OpenGround product for sharing underground geotechnical data achieves over 500,000 boreholes

Unforeseen ground conditions are a major cause of delays in construction projects.  This year at the Year in Infrastructure event Bentley announced OpenGround a collection of applications for collecting, sharing, visualizing, analyzing, and accessing geotechnical information about the underground. It has already amassed a database of over 500 thousand boreholes. 

Reports from the National Economic Development Office (NEDO) found that 50% of commercial buildings and 37% of industrial buildings experienced delay due to unforeseen ground conditions.  There are two aspects to the problems associated with the subsurface.  One is geotechnics, soil, ground water, and geology conditions.  The other is the location of underground infrastructure, utilities, telecommunications and other cables and pipes.

Every construction project devotes considerable effort to discovering subsurface geotechnical conditions, but this data is used for the current project and rarely shared.  In the Netherlands an attempt is being made to address the problem by the Key Registry for the Subsurface (BRO) which came into force in January, 2018. The BRO registry ultimately will record 26 data types.  On 1 January 2018,it become mandatory to report the first three data types, geotechnical surveys (CPT), groundwater monitoring wells and soil drilling sample profiles. On 26 June 2018, this data became publicly available via the Dutch open data portal PDOK.

In the UK the Dig to Share project, supported by Atkins, British Geological Survey (BGS) and Morgan Sindall, is addressing this problem. Its aim is to develop a fully digital workflow, which is accessible to the whole industry, to upload and access data from the BGS web-based system. This will be developed on top of the existing system hosted and maintained by BGS. 

Last year Bentley announced the acquisition of Keynetix, a UK-based provider of cloud-based software for capturing, visualizing, modeling, and sharing geotechnical data. This added to Bentley's products for the subsurface including borehole report management (gINT and HoleBASE) and geotechnical analysis applications (PLAXIS 3D and SoilVision) with which Bentley plans to enable a digital twin of the underground.

This year at the Year in Infrastructure 2019 event in Singapore Roger Chandler, Director of Product Management, GeoTechnical Information Management, announced a new product OpenGround (based on Keynetix.cloud).  OpenGround is a cloud-based SaaS application with a web API that supports real-time logging of subsurface geotechnical data and enables sharing this data over the web.  In the very short time since OpenGround became available it has already amassed a database of over 500 thousand boreholes. 

Developing a roadmap for a reliable map of underground utilities in Singapore

In 2017, the Singapore Land Authority (SLA), Singapore-ETH Centre, and the Geomatics Department of the City of Zürich started the Digital Underground project as a first step toward achieving an accurate, current, and complete map of subsurface utilities in Singapore. I had a chance to chat with Rob van Son, of the  Singapore-ETH Centre and Project Lead of the DU project, and in this article we describe progress to date and the next steps toward developing a reliable map of underground utilities in Singapore.

A number of years ago Singapore recognized the urgency for a map of its underground utilities for a number of purposes; planning, design, construction, ownership and maintenance of underground infrastructure. Recently the Urban Redevelopment Authority (URA) of Singapore unveiled its Draft Master Plan 2019 to guide Singapore's development for the next 10 to 15 years which identified increased use of underground space as one of its strategies to create more space for Singapore's growing needs. In 2017 the Singapore Land Authority (SLA) started the Digital Underground project. The project aims to establish a reliable map of underground utilities in Singapore. Part of the first phase of the project was an experiment which applied a combined LiDAR and ground penetrating radar array to capture above and below-scans of various areas in Singapore. The experiment contributed to an assessment of the quality of Singapore's underground infrastructure information and the feasibility of large area utility mapping techniques. The first phase of the project resulted in a set of recommendations for moving forward toward a reliable digital twin of Singapore's underground utilities. These represent a comprehensive checklist of how at a national and city level we can begin to address the pressing problem of creating a reliable digital twin of underground utilities.

Singapore has recognized that insufficient quality of information about the subsurface can lead to poor decisions, unnecessarily lengthy planning processes, increased cost of construction, more litigation and missed business opportunities.

When designing and building new utilities and other facilities underground, a reliable map is needed to plan for the optimal use of scarce and often already congested space and avoid utility damage causing loss of service and creating delays in the construction process. Land ownership regulations require an accurate record of the alignment of subsurface utilities in order to minimize conflicts in the use of underground space.

Although professionals from land development agencies, utility owners, and engineers in Singapore are able to access various sources of utility information, this information cannot be considered reliable. It lacks accuracy, does not always represent the latest and current situation, and may not be complete, lacking information on legacy utilities. Furthermore, data quality itself is a largely unmanaged and unknown aspect, leading to uncertainty. Recently in Singapore, important steps have been taken to ensure that all newly built utilities are captured according to the same standard and exclusively by qualified persons. However, the lack of quality of existing information remains to be reconciled. Similar to the U.S. and the UK, in Singapore the contractor has full liability for damage to underground infrastructure, which means there is little motivation from the perspective of liabilities for network operators to improve the quality of their location information about underground utilities.

Since its release in 2017, the Standard and Specifications for Utility Survey in Singapore has progressively been mandated for all newly built utilities by utility regulatory authorities. The standard is for open trench cases only. It covers the techniques to be employed, the data to be captured, and the requirement for a qualified professional (a Registered Surveyor at the Land Surveyors Board of Singapore) to do the job.

In 2017, the Singapore Land Authority, Singapore-ETH Centre, and the Geomatics Department of the City of Zürich started the Digital Underground project with an initial objective of developing a road map for achieving a reliable map of subsurface utilities. One of the first activities was to assess the state of the art for reality data capture technologies and how these might be used to improve the accuracy of underground data for both new and existing buried utilities. The DU project is looking at the whole workflow relating to underground infrastructure from data capture to application which is a different approach from most of the rest of the world which takes a bury and find it approach.

For new infrastructure, non-conventional reality capture techniques were documented. For open trench utilities, this included the use of handheld laser scanners. For trenchless construction, a technique for determining location by pulling a gyroscopic mapping device through pipes was investigated.

For existing utilities the latest ground penetrating radar (GPR) technology was assessed to determine the effectiveness of GPR as a mapping technique with Singapore's soil conditions. Singapore soils are not particularly good for GPR, which is able to penetrate down to 2 meters. The DU project collaborated with Leica Geosystems and IDS GeoRadar to deploy the Pegasus:Stream, a combined LiDAR and ground penetrating radar array, for the first time in Southeast Asia. It was used to capture above and below-scans of nine distinct areas in Singapore. In the Toa Payoh estate, which has undergone continuous development since the 1960s, data was captured along a 1.8 kilometre four-lane road at a speed of 15 kilometres per hour. Scanning of the entire area took five hours. Post-processing of the data back in the office required three weeks. A total of 109 subsurface pipes and ducts were detected in the area. By comparing the detected utilities to available information on existing utilities, it was found that in older areas the location of a lot of underground infrastructure was not recorded or does not have an accurate location. Only around 30% of detected utilities could be identified using the existing information. For newer areas, this figure was higher, but many detected utilities linking to new developments were not mapped yet.

Data modeling was a second key objective of the first part of the project. The goal was to develop a data model that enables the captured subsurface infrastructure data to meet the requirements of end users use cases. With the data collected with LiDAR and GPR, the Digital Underground team explored how available map data and the subsurface utilities detected during the pilot could be combined to create a 3D model that could be fused with Singapore's cadastral maps. The results were showcased in a demonstrator application highlighting the value of reliable 3d information subsurface utilities for land administration purposes.

The invaluable results of the first part of the project resulted in a set of recommendations for moving forward toward a reliable digital twin of Singapore's underground utilities. These provide a set of high level requirements with which at a national level we can begin to address the pressing problem of accurately mapping underground utilities. The first main recommendation is to establish a single, consolidated national map of subsurface utilities that enables data quality management. This is underway in the UK and is part of a national strategy for a national infrastructure map that has been initiated in the U.S. Secondly a national mapping strategy for accurate mapping of subsurface utilities was proposed that is supported by two pillars. The first is to ensure that all newly built utilities are captured accurately. The second is to capitalize on data capture opportunities during the life cycle of existing utilities. For example, capital construction projects are expected to benefit directly from utility mapping in the design phase, and the captured information can be consolidated for use beyond their scope and timeline. Other events, such as visibility or accessibility of utilities during trial trench, maintenance and rehabilitation works may provide suitable data capture opportunities that can each be matched with the appropriate data capture technique. Another key requirement is accessibility, that this data is available to the people that need it, i.e. land administrators, planners, designers, utilities and telecoms, construction contractors, and emergency responders, but in a way that protects privacy, does not reveal information about networks that could be used competitively, and prevents misuse.

The first part of the Digital Underground project was just completed 2 months ago. Funding has been approved for Part 2 of the project. Over the next two years, SLA and its partners intend to further develop the foundations of an ecosystem that supports and enables a reliable map of subsurface utilities. The project will cover a wide array of themes such as governance, legislation, capacity development, data capture standards, and a prototype platform for quality control and consolidation of 3D utility information for planning and land administration applications. One of its early results is the launch of Digital Underground Connect: a community of practice for subsurface utility mapping together with Bentley Digital Advancement Academies.

 

Accelerating progress in open spatial data infrastructure at the municipal level in Canada

At the SDI Summit in Quebec City Gordon Plunkett offered an interesting perspective on the penetration of open spatial data infrastructure (SDI) at the municipal level in Canada.  Local governments are traditionally responsible for local services such as safety, infrastructure, environment, and recreation that directly impact citizens' daily life. But citizens are often not aware of available services. To increase citizen engagement local governments should make their services more easy to use, understandable, and relevant by providing them in visual form via web pages and web apps.

Background

Open data has been gathering momentum among governments initially driven by United States Federal Government data being in the public domain (since 1909.) The principle of free and open access to government geospatial data has been adopted by many governments including US Federal, Canada, Japan, South Africa, Brazil, California state and counties, and by the City of Vancouver and other cities.  In 2013 G8 leaders (Canada, France, Germany, Italy, Japan, Russia, United Kingdom, United States, and the European Union) signed a charter on open data.  The Ordnance Survey in the UK began releasing some of its data with open licensing as part of its OpenData initiative in 2010, but it wasn't until 2018 that the OS Mastermap was made openly available for the public and businesses to use. But being able to access prepared maps and other derived material is just a first step, governments need to provide access to raw geospatial data in commonly used Web-friendly formats. In the U.S. Data.gov was launched in late 2009.  In 2013, President Obama issued an Executive Order making open and machine readable the new default for federal government information in the United States.

In Canada the Government of Canada announced its Open Government Action Plan in 2012.  The fundamental principle was a government-wide directive making government information and data available to the public by default. Geospatial has led the development of open data in Canada.  Of the 80,000+ datasets that are available on the federal government's open data portal, 69,000+ are geospatial. Currently in addition to the federal government eleven provinces and territories and over 60 municipalities have open data portals. At the municipal level Vancouver has been in the forefront of municipal open data - Vancouver's open data portal was initiated in September 2009.  In November 2009  the City of Toronto launched Toronto.ca/open, the City of Toronto's official beta data set catalogue, which provided open access to city data.  In January 2010 the City of Edmonton opened Edmonton's open data catalogue, which made city data including geospatial data openly available.  In 2010 Ottawa created an Open Data Ottawa site to encourage digital innovation, improved service delivery, and stimulation of economic growth.

A major motivation for open data has been government transparency.  But there is also evidence that open geospatial data contributes to the economy.  In 2008  Australia and later New Zealand commissioned studies of the contribution of spatial data to the national economy that have concluded that with appropriate government policies the contribution to the GDP, estimated to be on the order of 1%, could be doubled.  In 2015 Natural Resources Canada released the Canadian Geomatics Environmental Scan and Economic Value Study that estimated the contribution of the geospatial sector to the Canadian economy to be on the order of $21 billion.

Current state of open SDIs in Canada

Interoperability is key for an effective SDI. The inability to easily share data is a serious constraint on the value that can be derived from it. Gordon presented a simple SDI interoperability maturity matrix. Most municipalities probably fit into what he called syntactic interoperability, the ability to share data using common file formats and APIs.  Some have ventured in sharing data at the semantic level, which means recognizing content through shared data models such as the Canadian Municipal Data Model (CMDM), National Road Network (NRN), and NENA standards for 9-1-1.  A few are venturing into addressing the issue of different names for the same objects in different jurisdictions or in different disciplines.

Patterns of use for SDI include mapping and visualization, monitoring, field mobility, data management, analytics, sharing and collaboration, decision support, citizen engagement, and design and planning. 

Perhaps the top challenge for local governments, who hold the data that is most directly relevant to citizens, is making them aware of what is available and how to use is to address their every day concerns. Gordon presented some examples across Canada of interesting open data portals: Kamloops, British Columbia; Saint John, New Brunswick; Halifax Regional Municipality, Nova Scotia, which have made a lot of geospatial and other data available on open portals; St Albert, Alberta which enables citizens to report broken infrastructure and track progress in resolving issues; Regina, Saskatchewan which also makes a lot of data available on an open portal; Red River Planning District, whose site is heavily used during the annual Red River flood season; Brampton, Ontario, a poster child for open data and citizen engagement; Repentigny, Quebec, which enables users to easily determine the availability of municipal services close to where they live or intend to live; Prince Edward Island Hydro, which shows current electricity outages; and Mount Pearl, Newfoundland, whose portal include a parcel map in addition to roads and other basic map layers.

I would suggest that another data layer that should be part of every comprehensive SDI and that is missing in virtually all local, provincial, and federal SDIs is accurate maps of underground infrastructure in public right-of-ways.  Not knowing the location of underground cables and pipes has important implications for citizen safety and the reliable delivery of essential services.

In summary, local governments are traditionally responsible for local services such as safety, infrastructure, environment, and recreation that directly impact citizens' daily life. But citizens are often not aware of available services.  Local government can improve citizen engagement by making their services to more easy to use, understandable, and relevant by providing them in visual form via web pages and web apps.

Reliable statistics provide a foundation for improving pipeline safety

Pipeline incidents (gas distribution, gas transmission, gas gathering and hazardous liquid) resulting from excavation damage 2005-2019

“If you can’t measure it, you can’t improve it.” (Peter Drucker) is a maxim that applies to airline accidents, gas and oil pipeline explosions and leaks and underground utility damage resulting from excavation.  Whereas reliable statistics have been available for the civil aviation industry since the early 20th Century, for gas and hazardous liquids pipelines reliable statistics have only been available since the beginning of the 21st Century.

Aviation safety: fatalities per trillion passenger kilometres 1970-2018

Reliable metrics provide a foundation for assessing the social and economic impact of incidents and the effectiveness of new technologies and policies in preventing and reducing the severity of these incidents. The commercial airline industry is a model of what reliable statistics and data for incidents and policies and technologies designed to reduce these incidents can achieve.  In the civil aviation industry there have been reliable statistics since the early 20th Century. The Air Commerce Act of 1926 created an Aeronautic Branch of the United States Department of Commerce. In 1929 there were 51 commercial airline incidents, which represents an accident rate of about 1 for every 1,000,000 passenger miles. If that rate is prorated to the current total number of revenue passenger miles this would be equivalent to 7,000 fatal incidents per year. Fatal accidents per million flights in 2018 have decreased 16 fold since 1970, from 6.35 to 0.39, and fatalities per trillion revenue passenger kilometers (RPK) decreased 54 fold from 3,218 to 59 in 2018. 

Since 2004 the total miles of pipeline in the US including crude oil lines, petroleum product lines, gas distribution mains, gas transmission pipelines, and gas gathering lines increased by about 12 % to 1,830,672 miles in 2017.  The Pipeline and Hazardous Materials Safety Administration (PHMSA) was created in 2004 by federal legislation.  Pipeline operators are required to submit performance measure reports for pipeline infrastructure covered by integrity management (IM) programs. This includes gas distribution, gas transmission, and hazardous liquids.  Operators are required to report all pipeline incidents including the amount of location, volume of product released, number of fatalities and injuries, costs and cause.  Since 2005, pipeline operators have reported excavation damage as the cause of 1052 incidents, resulting in 48 fatalities, 195 injuries requiring hospitalization, and $ 481,736,551 of property damage.

Gas distribution leaks by cause 2018 PHMSA

Possible causes include

• CORROSION - caused by galvanic, atmospheric, stray current, microbiological, or other corrosive action.
• EXCAVATION DAMAGE - incidents resulting directly from excavation damage by operator's personnel, by the operator's contractor, or by people or contractors not associated with the operator
• INCORRECT OPERATION - incidents caused by operating, maintenance, repair, or other errors by facility personnel
• MATERIAL/WELD/EQUIP FAILURE - incidents resulting from poor welds, poor construction, and stresses such as vibration, fatigue, and environmental cracking.
• NATURAL FORCE DAMAGE - incidents resulting from earth movement, lightning, floods and washouts, frost heave, and other natural causes.
• OTHER OUTSIDE FORCE DAMAGE - incidents caused by non-excavation-related outside forces including vandalism and terrorism.
• ALL OTHER CAUSES -  incidents whose cause is currently unknown

Gas distribution leaks by cause 2010 PHMSA

Since pipeline incident data began to be collected the proportion of incidents attributed to excavation has increased.  For example in 2018 34.4 % of gas distribution leaks (unintentional escapes of gas from the pipeline not reportable as Incidents) were caused by incorrect operation.  In 2010 only 16.4 % were attributed to incorrect operation.  Clearly improved training and other initiatives have reduced equipment operation failures but also being able to better identify causes has made it possible to reduce other types.

Serious pipeline incidents, fatalities, and injuries 1999-2018 PHMSA

Serious Incidents include a fatality or injury requiring in-patient hospitalization.  PHMSA statistics suggest there may have been some improvement in number of serious incidents and in fatalities, but prorating these statistics to pipeline mileage suggests otherwise.

Significant gas transmission pipeline incidents per 10 000 HCA miles 2004-2018

On June 10, 1999, a gasoline pipeline operated by Olympic Pipeline Company exploded in Bellingham, Washington. Three people died in the accident.  On August 19, 2000 a natural gas pipeline owned by the El Paso Corporation exploded near Carlsbad, New Mexico killing 12 people.  After these incidents the government moved quickly to improve the quality of geospatial and other data about underground pipeline assets. Operators were required to identify high consequence areas (HCAs) where there were 20 or more structures intended for human occupancy within a radius (potential impact radius or PIR) defined by the diameter and pressure of the pipe. 

One of the reported statistics is the number of significant incidents per 10,000 HCA miles.  Significant incidents are defined as incidents involving a fatality or injury requiring hospitalization, $50,000 or more in total costs measured in 1984 dollars, substantial liquid releases, or fires and explosions.  The data reveals no trend in improvement in this statistic.

By analogy with the civil aviation industry, reliable statistics resulting from government mandated reporting of pipeline incidents provides the foundation for the next phase of implementing safety management plans designed to produce continuous improvement in these statistics. Safety management systems tailored to specific industries and requirements have been developed often involving regulation for civil aviation, international maritime shipping, and the rail industry in Canada.  At the GITA Pipeline Forum in Houston, there was a lot of discussion about pipeline safety management systems (PSMS), specifically the RP 1173 framework developed by the American Petroleum Institute (API) in partnership with PHMSA, state pipeline regulators, and other interested stakeholders. RP 1173 provides best practices for pipeline operators including operational controls, risk management, incident investigation, evaluation, and lessons learned, and safety assurance.  This is intended for operators of hazardous liquids and gas pipelines under the jurisdiction of the US Department of Transportation.  It provides for the comprehensive and systematic management of safety-related activities for achieving its goal of zero incidents per year.

Tracy Thorleifson of G2 Integrated Solutions suggested that at this stage we should be really worrying about asset integrity risk assessment.  Pipelines are distinguished from airplanes in being in intimate contact with their environment which means many more variables to consider.  Pipeline operators routinely capture and store operational information about their networks including pressure, flow rates, valve status and so on, but they don't capture and store information about the environment in which the pipeline is operating, such as, earth movement including subsidence, soil types, landuse, vegetation, proximity to water bodies, etc.  Asset integrity assessment should include real-time dynamic segmentation using the thousands of data layers for the external environment available from NOAA, NASA, ESRI Living Atlas and other sources.  This is an area where machine learning could fruitfully be applied to calculate risks of external events affecting the integrity of pipelines.  For a pipeline in an area prone to flooding,  subsidence, land slide, or river bank collapse, using dynamic segmentation would permit using these factors to adjust the risk assessment algorithms. Furthermore risk models should include data quality, including location accuracy, as a potential threat factor.  For example, not knowing accurately the proximity of hazardous underground utilities can seriously compromise risk assessments.

In summary since 2005 the gas and oil pipeline industry and its regulator have achieved reliable statistics for pipeline incidents.  The focus is now on regulation, policies and technologies to improve these statistics to reduce the number and severity of pipeline incidents.

Open source geospatial technology enables cost-effective solutions in many domains

At the GITA Pipeline Forum in Houston, an enthralling presentation by Rich Henry of Pivvot, a recent startup that builds solutions for pipeline routing and other applications, highlighted how open source geospatial technology is transforming applications in a variety of sectors. Pivvot builds SaaS solutions using open-source components on Google Maps and Amazon Web Services to provide a web-based cloud computing platform with a curated database from over 900 public and private sources.  The advantages of this approach for siting pipeline routes are dramatic; applying infinite computing resources, completely avoiding the arcane art of managing ESRI ArcGIS licences, and integrating hundreds of data layers making it possible to run alternative pipeline right of way scenarios in hours rather than weeks or months.

Recent technical advances have made it possible to implement cost-effective solutions for utility, telecom and pipeline asset management, integrating BIM and geospatial, and geospatially-enabled machine learning.  Standards are a key underpinning in the development of geospatial applications for a number of vertical industries.  There are now many widely adopted geospatial standards; WMS, WFS, GML, KML, InfraGML, PipelineML, WaterML to name just a few from the Open Geospatial Consortium (OGC) that were not available two decades ago.  Furthermore, open source geospatial developers like open standards.  Many open source libraries and applications implement standards from the Open Geospatial Consortium (OGC), W3C and GeoJSON. Since 1999 there has been rapid development of open source geospatial projects including MapServer, GDAL, PostGIS, GeoServer, OpenLayers, Leaflet, QGIS and others.  The Open Source Geospatial Foundation (OSGEO) was founded in in 2006 and the open source code-base has enabled entrepreneurs to rapidly develop low cost geospatial applications because the cost of maintaining most of the code is shared across a broad developer community.  I'll describe a few of the solutions built on open source components that I have encountered in the last few years.

Records management for utilities

One of the challenges faced by large utilities with multiple GIS, outage management, customer information and other systems, which are often the result of mergers, is how to make spatial and associated data, in different, often incompatible systems, accessible to all users not just GIS professionals.  At Distributech 2018 I discovered how Duke Energy is running an off-the-shelf web-based product based on open source components that is being used by 8000 users, 2000 of them using it 24/7 every day, to access GIS and associated data in several GIS systems, both online and offline, using laptops, tablets and phones. A single, common easy-to-use web tool permits viewing geospatial asset data stored in GE Smallworld, ESRI ArcGIS, Intergraph GTechnology, spatialNET and Google Maps as well as non-GIS data in Maximo, SAP, three different outage management systems, four or five customer information systems, CAD drawings and photos and images.

At Distributech 2019 a new entrant IQGeo emerged into the utility and communications space and announced an alternative to traditional GIS.  myWorld Capture is a mobile solution that prioritizes direct data maintenance from the field.  Building on an open source geospatial platform that supports data versioning and interfaces to legacy utility/communications GISs such as ArcGIS and Smallworld, myWorld provides tools that make it possible to develop and maintain a high quality near real-time GIS with the information that is required for a complete network asset data model including equipment location, descriptions, connectivity, condition, and status.

Open access to municipal geospatial data

At GITA 2018 in Phoenix, Bob Basques, GIS Systems Developer at the City of St Paul, described a system called COMPASS he and his team have developed an easy to use tool that allows city employees and the public access to all of the City's spatial and associated data including, for example, scans of surveyors' notebooks, 2.2 million street level photos, and permitting and licensing information from 200 different applications. Based completely on open source components the system is compact and efficient enough to run on a Raspberry Pi.

Integrating BIM and 3D geospatial

At the GEO|Design+BIM conference I discovered mago3D a 3D web-based geo-platform developed by Gaia3D.  It is open source and integrates many well-known open source geospatial products and APis. It has an engine for handling massive, complex BIM objects that supports LOD (level of detail) control and efficient spatial queries and culling (the removal of objects that are not visible to the observer which is key to efficient visualization or large, complex structures.) It integrates with two of the best known open source 3D virtual globe tools, Cesium and NASA WorldWind, for visualization in a geospatial context.  Viewing BIM structures in mago3D has some similarities with a Google Earth experience. The source code is available under open source licenses.

Machine learning

Deep learning algorithms have been developed by academia and as a result the code for the most part is open source. For example, a deep neural network model developed originally for medical image segmentation called U-Net is open source and has been applied to identifying building footprints. Successful training of deep networks requires thousands of labeled training samples. Labeled data involves people on the ground manually ground-truthing land use types and other features so that the deep learning algorithms can learn what to recognize.  Machine learning is computationally intensive and requires a distributed engine.  Spark is a open source (Apache) project which enables distributed processing for global scale computation. Machine learning has been used successfully to identify building footprints and transportation networks and deforestation in satellite imagery.

RasterFrames is a free and open source toolkit allowing scientists, data scientists, and software developers to process and analyze geospatial-temporal raster data with the same flexibility and ease as any other data type.  Open source training data is beginning to become available, for example, the System for Terrrestrial Ecosystem Parameterization (STEP) has 2000 manually labeled sites covering 17 different land cover types including five forest types scattered across all continents.

Pipeline siting

To identify an optimal route for a pipeline on your own local hardware involves stitching together many DEMs, calculating slopes, finding, downloading and integrating hundreds of data layers, and avoiding certain areas (protected or uncooperative owners).  You have to worry about having enough disk space and the calculations can take a long time with no certainty of completion.  Running these calculations on several machines in parallel requires an ArcGIS licence for each of the machines.  Together this could result in months of data collection and processing.  In contrast even for long pipelines an open source solution from Pivvot running in the cloud means you don't have to worry about file management and disk capacity or getting enough machines and licenses to run it in parallel. Using Pivvot's curated datasets data collection and processing can be done in a few hours and it is scalable to many data layers.

Migrating to an open source platform is easier than you think

Samuel Song is responsible for the National Cemetery GIS implementation within Veterans Affairs and like many in government increased use of GIS means a greater sensitivity to cost.  Samuel outlined his experience in migrating to open source at a GIS in the Rockies conference in Denver.

 He had been using ArcGIS Desktop as his desktop GIS, SQL Server as a database, ArcGIS Enterprise as a web server, and ArcGIS API for Javascript for his web GIS. To look at ways to lower his costs Samuel experimented with open source geospatial software; QGIS for his desktop GIS, PostgreSQL/PostGIS for his database, GeoServer for his web server, and Leaflet or OpenLayers for his web GIS (all part of the OSGEO open source geospatial stack). His most important findings are that QGIS is a very stable, powerful alternative to ArcMap, PostgreSQL/PostGIS is a stable alternative to SQL Server and perhaps most importantly migrating data to an open source spatial database from an ESRI Geodatabase was much easier than he expected. His overall conclusion was that with the currently available open source software stack it is possible to move to a fully open source platform from an existing ArcGIS implementation and that the migration is much simpler than expected.

Records management for utilities

One of the challenges faced by large utilities with multiple GIS, outage management, customer information and other systems, which are often the result of mergers, is how to make spatial and associated data, in different, often incompatible systems, accessible to all users not just GIS professionals.  At Distributech 2018 I discovered how Duke Energy is running an off-the-shelf web-based product based on open source components that is being used by 8000 users, 2000 of them using it 24/7 every day, to access GIS and associated data in several GIS systems, both online and offline, using laptops, tablets and phones. A single, common easy-to-use web tool permits viewing geospatial asset data stored in GE Smallworld, ESRI ArcGIS, Intergraph GTechnology, spatialNET and Google Maps as well as non-GIS data in Maximo, SAP, three different outage management systems, four or five customer information systems, CAD drawings and photos and images.

Samuel Song is responsible for the National Cemetery GIS implementation within Veterans Affairs and like many in government increased use of GIS means a greater sensitivity to cost.  Samuel outlined his experience in migrating to open source at a GIS in the Rockies conference in Denver.

 He had been using ArcGIS Desktop as his desktop GIS, SQL Server as a database, ArcGIS Enterprise as a web server, and ArcGIS API for Javascript for his web GIS. To look at ways to lower his costs Samuel experimented with open source geospatial software; QGIS for his desktop GIS, PostgreSQL/PostGIS for his database, GeoServer for his web server, and Leaflet or OpenLayers for his web GIS (all part of the OSGEO open source geospatial stack). His most important findings are that QGIS is a very stable, powerful alternative to ArcMap, PostgreSQL/PostGIS is a stable alternative to SQL Server and perhaps most importantly migrating data to an open source spatial database from an ESRI Geodatabase was much easier than he expected. His overall conclusion was that with the currently available open source software stack it is possible to move to a fully open source platform from an existing ArcGIS implementation and that the migration is much simpler than expected.

New regulations for pipeline safety creates opportunities for geospatial professionals

New federal regulations in the U.S. will likely more than double the areas where higher levels of pipeline integrity maintenance standards are mandated by federal legislation.  Identifying these areas is more complex creating opportunities for geospatial professionals to apply new technologies including machine learning.

On June 10, 1999, a gasoline pipeline operated by Olympic Pipeline Company exploded in Bellingham, Washington. Three people died in the accident.  On August 19, 2000 a natural gas pipeline owned by the El Paso Corporation exploded near Carlsbad, New Mexico killing 12 people.  After these incidents the government moved quickly to improve the quality of geospatial and other data about underground pipeline assets. The result was within about two years federal regulation from the Pipeline and Hazardous Materials Safety Administration (PHMSA) which required companies operating pipelines carrying hazardous materials to identify high consequence areas (HCAs) where there were 20 or more structures intended for human occupancy within a radius (potential impact radius or PIR) defined by the diameter and pressure of the pipe. Within the HCAs utility companies were required to maintain facilities data at a higher level of integrity. The cost to identify HCAs for a relatively small utility in Pennsylvania that I was familiar with at the time cost $2 million over two years.  This represented a significant opportunity for experienced geospatial professionals.

The Pipeline Safety, Regulatory Certainty, and Job Creation Act of 2011 mandated telephonic or electronic reporting of accidents and incidents including the amount of product released, number of fatalities and injuries, but not an accurate location, within 48 hours of occurrence.  The act also required the Secretary of Transportation to maintain a map of high-consequence areas in which pipelines are required to meet integrity management program regulations.  At this time the locational accuracy requirement for pipelines was  ± 500 feet.

At the GITA Pipeline Forum in Houston, Darrel Donaho of G2 Integrated Solutions presented an overview of the latest regulations from PHMSA.  Referred to as the Gas Mega Rule the new regulation extends the higher standards of pipeline maintenance to moderate consequence areas (MCAs). Pipeline operators are now required to identify areas where 1) the number of occupied structures within the PIR is five or greater and less than 20 or 2) where any portion of the pipeline PIR intersects the paved area of interstate highways, freeways and expressways and other principal four-lane arterial roads.  They are also required to report longitude/latitude to five decimal places for all incidents.

Whereas in the past identifying HCAs involved hand drawing building footprints from imagery, machine learning can now be used to automate the process of identifying building footprints prior to using a buffer analysis to identify areas qualifying as MCAs.  The analysis is aided by the current accuracy requirement for locating pipelines of ± 50 feet.  Identifying MCAs where highways intersect pipeline PIRs is more complex because automating the identification of transportation corridors is more difficult.  There are alternative, but more expensive ways to find this information.

An analysis of a 14.91 mi transmission line suggests that the impact of the new regulation will be significant, more than doubling the areas where higher integrity maintenance applies.  This creates a significant opportunity for geospatial professionals to apply new technologies including machine learning.

HCA	MCA (>=5)	MCA roads	Total MCA
5.65 mi	2.63 mi	3.03 mi	5.66 mi