Between the Poles

Machine learning cost-efficiently creates high quality utility GIS data for smart grid applications

Many utilities' GIS contains only a small part of what is required for managing a smart grid.  Often even the data in the GIS is too unreliable to use for smart grid applications.  For example, utility GISs may not include which customers are connected to which transformer, or if they do include this information it is incomplete or incorrect.  Many utility GISs cannot identify the phase(s) each customer is using and they may not include network topology - what is connected to what.  Other missing data may includes street lights, distributed energy generation such as rooftop PV panels, the type and condition of poles, joint use of poles, and encroachment.  To correct these problems typically requires a foot survey which is expensive.  For example, a pole inventory costs at least $20 per pole.  At the Open Geospatial Consortium (OGC) Energy Summit at EPRI in Charlotte, John Simmins of EPRI described the research he is leading at EPRI to use modern technology including machine learning to reduce this cost to pennies per pole using available imagery such as Google Streetview.

I began working in the utility and telecom sector in 1993, specifically in developing software for records management (called network documentation outside of North America), and at that time the magnitude of the AEC+geospatial cultural mismatch was just beginning to be appreciated by utilities and telecoms. Planners tend to use GIS tools, engineers and designers CAD tools, construction folks in the field paper CAD drawings, and asset managers FM, GIS or integrated tools. Fashioning AEC and geospatial data into an efficient data flow from planning through design and construction to operations and maintenance represented a challenge that remains a problem for utilities.  For many utilities the backlog of as-builts and updates waiting to be entered into the GIS stretches into months, with the result that GIS data is permanently out of date inhibiting management and field staff from relying on this data operationally.  The Between The Poles blog is over ten years old and one of the persistent themes from its very beginning in 2006 was the challenge of integrating CAD and GIS data in a common workflow (some examples; 2006, 2007, 2008 ).

There are signs that modern technology is beginning to offer a solution to the problem of developing and maintaining an accurate digital model of electric power grids.

I wrote recently about a utility in western Pennsylvania that intends to implement an Advanced Distribution Systems ( ), but realized that to this requires a network data model. Duquesne Light is typical of many North American small to medium electric power utilities.  It does not have an outage management system or an electrical data model which means it is difficult to connect customers to transformers and to determine upstream devices when customers report power problems.  Currently Duquesne Light's circuit maps are AutoCAD drawings.  Duquesne Light is moving to modernize its grid and plans to deploy an Advanced Distribution Management Systems (ADMS) including outage management (OMS), DMS and DSCADA.  But the ADMS will require developing a complete distribution electrical data model and to that in the traditional way can be expensive.

To quickly, safely and cost efficiently develop a network data model, Duquesne Light looked at several alternatives including reality capture technology.  A pickup truck was equipped with with side mounted photo and IR cameras and LiDAR laser scanners. The reality capture survey could be completed rapidly and safely, with no boots on the pavement  Actual development of the data model was completed in the office without the necessity of going to the field for quality checks because the imagery provided all the data that was required to create and verify the data model.  Duquesne Light found several significant advantages of the reality capture approach.  The first was asset accuracy - both location and asset attributes.  In addition the captured imagery could also be used for other purposes.  For example, it could be used for a joint-use audit.

But this is still basically a manual process requiring a human being to scan images.  At the Energy Summit John Simmins described his research using neural network technology to automatically identify poles and attached equipment from available imagery such as Google Streetview.  Normally applying neural networks requires large quantities of training data.  To reduce this requirement, John has evaluated "biologically inspired", forward pass, neural networks.  In this approach the algorithm first looks (foveates) for vertical things that have the characteristics of a pole, and then for the characteristics of equipment attached to the pole, for example, cross members, street lights, transformers and large cables.  From this analysis is possible to determine pole location, height and width, cross member position on the pole and projected length, street light support strut position and length, position of transformer and large cable connection locations.  The ultimate goal of the research is in addition to automatically detecting each element of equipment to identify the type, size and manufacturer of each piece.

This is not a simple problem and there is no silver bullet.  Using a standard neural network approach requires large volumes of training data.  For poles it requires images of poles with different types of damage under different lighting conditions and in different types of weather.  An Australian company SiteSee claims to be able to do this for communications towers and transmission pylons - identifying the type and manufacturer of each antenna or insulator on a tower or pylon and comparing it to the manufacturer's specification to detect signs of corrosion and other types of damage.  When what you are looking for has a distinctive geometry that differentiates it from the surroundings, a biologically inspired approach may provide a way of identifying equipment without large volumes of training data.  Another area where several different approaches have been applied to automate feature extraction is electric power transmission lines for vegetation management.

Creating a digital twin of an electric grid is essential for developing and maintaining a smart grid.  Utilities have been wrestling with this challenge for decades and in many cases their GIS data is still not much closer to what is required.  If it would be possible to use the technology that John Simmins is investigating or other technology to reduce the cost of improving the quality of their GIS data, it would accelerate the adoption of smart grid technology at many utilities.  In his presentation at the Energy Summit Jared Green described his research in using smart meter and AMI network data to map secondaries from the transformer to the home, identify sources of unmetered load, and use impedance data from smart devices to correct utility GIS data.  Given the history of the challenge to develop and maintain accurate GIS data at many utilities, these approaches represent a beacon of hope which may enable utilities to cost effectively develop a digital twin of their networks that is an essential prerequisite for a  smart grid.

Mobile app uses consumer 360° cameras for visual construction documentation

At this year's BIMForum in Las Vegas, I came across an intriguing new company in the construction technology space called HoloBuilder.  I had an opportunity to follow up with Harry Handorf to gain a better understanding of how their solution, a combination of an app for iOS and Android with cloud processing, makes it possible to use a consumer 360° camera such as Ricoh Theta cameras to quickly document progress on a construction site.

The handheld app is called JobWalk, which makes it possible after uploading a floor plan to walk through a job site placing the camera and tripod in the middle of each room to take a series of 360° photos.  After the photos are uploaded to the HoloBuilder cloud, virtual tours of the construction site are available to anyone with a web browser on their platform.  In the virtual walkthroughs, dimensions can be measured with 99% accuracy.  The only input data that is required to make this possible is the distance between the camera lens and the floor.  Plug-ins are available for Revit and Navisworks which make it possible to view as-designed BIM model renderings side-by-side with as-constructed.  It is also possible to time travel over the lifetime of a construction site comparing images taken at different times to track construction progress.

HoloBuilder introduced its technology to the broad market in 2014. In 2015 it won the grand prize at the RICOH THETA Developers Contest.  Since then it has focused its solution entirely on the construction market.  It even offers a solution for real-time construction monitoring.  Harry told me that it has many customers in the construction market including the likes of Skanska, McCarthy, AECOM, Hensel Phelps, Mortenson and others.

As a practical example of how HoloBuilder is used, Skanska is currently building a 185,000-sq. ft. hotel with a construction contract value of $52.3 million.  Skanska is using HoloBuilder on the project to document existing site conditions, share progress with stakeholders and make decisions requiring the agreement of remote stakeholders scattered around North America. One of the use cases where Skanska found a lot of value with HoloBuilder is capturing images as part of pre-pour inspections - necessary inspections that occur before pouring concrete.  The challenge is to document all the in-slab deck conditions prior to pouring in a very limited window of time.  Cables, heating and ventilation, plumbing and electrical, rebar, and other equipment need to be captured.  Skanska’s Project Engineers use HoloBuilder to take 360° photos in conjunction with the bi-monthly concrete pour cycle.  Each of the floors scheduled for concrete pouring is captured and uploaded during a one-hour site walk using the JobWalk app and a standard 360° camera. HoloBuilder estimates that one 360° photo contains approximately the same visual information as 11 single lens photos. The resulting HoloBuilder project provides information about the contents and layout of materials in the slab including mechanical, electrical and plumbing as well as the rebar size, spacing and number. The images also contain additional context related to site conditions, weather, and location where the 360° photos were captured. Skanska is able to share the site conditions with the architect and structural engineer as a supplement to the inspection.  With HoloBuilder, Skanska’s project team reduced photo documentation time by 50%.  It also was able to offer a valuable deliverable that could be shared with the owner, architect, facilities management team and other project stakeholders.

This is another example of how consumer devices are changing the construction industry. The number of startups like HoloBuilder, Dot Product, Enscape, Revizto, Fuzor, Skur, Cupix and others in the construction industry is increasing and many are using consumer technologies such as gaming engines (Unity, Unreal Engine). As new technology gains traction in a $10 trillion industry that has traditionally been ranked at the bottom of R&D spending and digitization, construction productivity will rise and infrastructure will become increasingly attractive to private investors who are expected to become the source of most of the money required to develop, renew and maintain our infrastructure for the next two decades.

Documenting construction projects with a consumer 360° camera

Many see handheld scanners as revolutionizing the construction sector.  Portable and handheld laser scanners offer millimeter accuracy and can be used for engineering purposes.  Cameras with less precision are being used for rapidly documenting a construction site for a variety of purposes.   Most recently a couple of startups Cupix and HoloBuilder are lowering the bar by offering cloud software for the AEC industry to enable images of a construction site to be captured with a consumer grade 360° camera like the Ricoh Theta S or V. Walking through a construction site capturing 360° images along the way and uploading the images to the web is all that is required to create 3D walk-throughs of a site.  The 3D walk-through can be used to measure dimensions, it can be annotated, it can be compared with a BIM model, and 3D images captured at different times can be compared to monitor construction progress.

 

For engineering purposes, handheld laser scanners provide the requisite precision but are relatively expensive.  The scanning process requires time and produces large point cloud files and imagery that need to be managed and processed for visualization.

• The Leica Geosystems BLK360 weighs a mere 2.2 lbs (1 kg), is incredibly small 6.5 x 4 inches and lists for US$16,000.  It is a full laser scanner and imager, capturing 360,000 points/second with a range of 0.6 - 60 meters.  The scanner can be attached to a Microsoft Surface Pro tablet to visualize what you have scanned. Up to 16 of these can be used at the same time.
• Dot Product DPI-8 and DPI-8X (short range) are multi-sensor devices (IR laser and photo cameras).  Targeted at professionals, especially in the engineering and construction sector, the devices incorporate SLAM technology and works with Android handhelds.  All the processing is done locally on the handheld - no cloud processing is required.
• The Faro Freestyle can scan objects within a range of half a meter to 3 meters away with a resolution of  better than 1.5mm with a image point density of up to 45,000 points/m² at 0.5m and up to 10,500 points/m² at 1m.  The scanner is attached to a Microsoft Surface Pro tablet for visualizing the point cloud as you scan.

For rapid site documentation, Matterport allows you to scan a floor of a building in an hour or less, upload the captured data to the cloud (Amazon Web Services) and a couple of hours later have a fully-rendered 3D walk-through that can be accessed over the web.  It was originally targeted on the real estate sector but recently has been showing up at construction industry and other events including the recent BIMForum in Las Vegas.  It provides a dollhouse view to see a whole floor at once, an inside view for walk-throughs, and a floor plan view for a traditional top-down perspective. It relies on IR and photo cameras and multiple scans in each room - scanning every 1.5 to 2.5 meters.  It is capable of rough measurement, but not for engineering or construction purposes.  Its biggest advantages are the speed with which you can scan a site and the small size of the resulting file which has to be uploaded to Matterport's cloud site.  In addition to real estate and construction site documentation, it is used by FM professionals who need a 3D context for facilities management.  For example, it can provide a mechanical room walk through with tags on different pieces of equipment and a way of directing anyone to any location in the facility.  The camera is several thousand dollars and cloud processing requires a subscription.

 

The most recent additions to the technologies that can be used to document a construction site are Cupix and Holobuilder which lower the bar by only requiring a consumer grade 360° camera such as a Ricoh Theta.  In the case of Cupix cloud processing requires a subscription.  Similarly to the Matterport, you walk through the site capturing images with 360° camera as you go.  The resulting images are uploaded to the Cupix site and the result is 3d tours or walk-throughs accessible over the web.  You can compare 3D tours taken on different days to monitor construction progress, compare a 3D walk-through with a BIM model (Revit or IFC) you have uploaded, and measure dimensions with a precision of 1% or better at 20 meters.  To reach this level of precision in measurement requires including a marker when taking pictures.  Alternatively if a dimension is known it can be used to scale the entire walk-through.  Cupix can use images captured indoors and outdoors.  Cupix says that if GPS metadata is available from the photos, they can georeference the walk-throughs and generate a KML file. I used Cupix with a low cost 360Fly4K camera and a Samsung Note 5 phone to capture the interior and a bit outside of a cabin in a nearby park.  It took a couple of attempts to get the correct spacing between camera placements in capturing 360° images, but after that it was straightforward. I uploaded a rough picture of a floor plan and I was able to walk-through the resulting 3D tour using the Cupix web viewer.  Without doing anything additional I was able to get rough measurements accurate to about 5%.  Cupix only requires a consumer grade 360° camera.  A subscription is necessary for cloud processing. Cupix is targeting a broad market including travel and hospitality, real estate, and arts and culture in addition to the AEC industry

 

I just came across HoloBuilder at the recent BIMForum in Las Vegas.  Like Cupix it only requires a consumer 360° camera. At first glance it appears to be focused on the AEC industry and makes creating and managing 3D tours of construction sites very easy.  It offers 360° SiteStream which is a unique live streaming solution for real-time construction progress monitoring in 3D that runs as a plug-in app on the Ricoh Theta V.

Application of geospatial deep learning to real world problems accelerating

Deep learning is increasingly finding practical applications using geospatial data, specifically, satellite imagery. At HxGNLIve in Las Vegas, Wim Bozelie, Technology Director at Imagem, described some very practical cases in which machine learning, specifically, convolutional neural networks (CNN) were successfully applied to real world problems including detecting blocked canals and differentiating different types of roofs.

The original application of deep learning to photo imagery was by Geoffrey Hinton at Google who used it to distinguish dogs and other objects in photos that people uploaded. The basic process to create these models is to find or create some training data, some data to test the model, and then play with the parameters to optimize the model for the test data. There is a long-running competition Imagenet Large-scale Visual Recognition Challenge (ILSVRC) which provides training and test data for evaluating algorithms for object detection and image classification at large scale. A motivation for this public challenge is to allow researchers to compare progress in detection across a wider variety of objects by providing publicly available, relatively expensive data. The validation and test data consist of 150,000 photographs, collected from flickr and other sources of imagery, which have been hand labeled with the presence or absence of 1000 object categories. The object categories are very detailed for the types of objects found on facebook, flicker and othe social media. Some examples,

dial telephone, cell phone, airliner, warplane, airship, balloon, space shuttle,aircraft carrier, Tibetan mastiff, Doberman pinscher, warthog, Arabian camel, dromedary, golden retriever, border collie, boxer, english springer, malamute, malemute, whippet, weimaraner, husky, brittany spaniel, lion, beaver, irish setter...

This training data is not much use for geographic data where typical applications of satellite imagery are for landuse and agricultural identification. Wim talked specifically about a type of deep learning called convolutional neural networks (CNN). For example, remote sensed imagery from a satellite can be used to create a model that will differentiate between corn and potatoes, using factors that can be calculated from satellite imagery such as normalized difference vegetation index (NDVI) (min, max, mean), texture (min, max, mean), vegetation height, and geometric factors such as orientation.  I blogged about how open source code and publicly available training data has been applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil. Wim reported a study in the Netherlands using CNN to identify blocked waterways using overflight imagery. An approach that did not use machine learning was tried first, but it found many false positives and missed some real instances of blocked waterways and the success rate was about 50 %. When machine learning was applied with reliable training data, a success rate to 97 % was achieved.

An important application of satellite imagery is identifying built structures. I have blogged about a competition to use satellite imagery to identify buildings and transportation networks. The winner applied a deep neural network model developed originally for medical image segmentation called U-Net.  Reliable training data was provided by the competition organizers. The winning solution used OpenStreetMap layers and high resolution Worldview multispectral layers as the input of the deep neural network algorithm.  Using the same data NVIDIA demonstrated the ability to automate detection of many road networks using deep learning algorithms and multi-spectral high resolution imagery.  Wim described another application of CNN and imagery in the Netherlands to differentiate residential roofs with dormer windows from those without and discussed how different aspects of the model, such as convolution layers, can be manipulated to improve results.

The essential key to effective application of deep learning is good training and test data - ground-truthed data that involves, for example, someone on the ground identifying whether the fields seen in the imagery are corn or potatoes or something else. There are  publicly available training data sets that can be used to train satellite imagery CNN models. For example, I blogged about the System for Terrrestrial Ecosystem Parameterization (STEP) dataset which has 2000 manually labeled sites covering 17 different land cover types scattered across all continents. Wim described a large publicly available training dataset for Sentinel 2 imagery, which was developed by Eurosat. It contains 30,000 polygons of landuse training data for ten classes of landuse: annual crop, forest, herbaceous vegetation, highway, industrial, pastures, permanent crop, residential, river and sea or lake. In an experiment Wim took 500 training polygons from this dataset for three types of landuse: built-up, pastures and vegetation, and water, to build a CNN model. He applied the resulting model to 25 test areas. Of the 25 test areas the model correctly identified 23, but two were incorrect.

AI has often suffered from inflated expectations, complex code, heavy processing requirements and not very practical applications. There is a growing body of practical applications in the geospatial domain that shows that as a result of the immense processing power available in the cloud machine learning is able to generate fairly easily practical and useful results.  As Chris Holmes pointed out in his talk at FOSS4GNA, the challenge now is to develop reliable and publicly available training and test datasets to enable deep learning models to be created for a broader range of applications using the huge volumes of geospatial data that are now available.

A game changer for interior distance measurement introduced at HxGNLive

At HxGNLIve in Las Vegas Juergen Dold announced another very impressively designed device in the BLK line called the BLK3D. It is designed to be a game changer in the interior measurement space traditionally addressed by Leica's DISTO line because of its ease of use, size and paper-eliminating functionality.

The BLK3D follows in the footsteps of the BLK360, which is a game changer in the terrestrial laser scanner space because of its ease of use (one button), size (weighs 1kg / size 165 mm tall x 100 mm diameter), its capability (full 360 scan in less than 3 mins), and range (mm accuracy at 60 m). It it is intended the democratize laser scanning to other professionals beyond surveyors.

The BLK3D is small, somewhat larger than my Samsung Note 5.  It has two photo cameras and two laser range finders. I had a chance to chat with Johannes Hotz, Business Development at Leica Geosystems who explained that it is intended to replace the laborious paper-based measuring processes used in interior spaces. Leica Geosystems worked with architects, designers and contractors who work in interior spaces, and identified paper-based processes that could be replaced by a digital process.

The two cameras on the BLK3D allows you to capture a 3D record on-site that can be taken back to the office and shared as a JPG file. The device does not require anything to be uploaded to the cloud for post processing, it does all the processing itself. Once a 3D image has been created, you can tap on the screen on two locations, and it will measure and display the distance between the two points. if has edge and corner detection so that if you tap near a corner or an edge, it will snap to the exact corner or edge to ensure you get a precise answer (mm precision at up to two or three meters.)  It replaces the process of making a measurement  with a tape measure or laser range finder, writing the measured distance down on a piece of paper and then repeating the process over and over again for every measurement you need,

The BLK3D allows contractors to easily record images of their work at the time they do it so that afterwards if there are legal issues that have a permanent record of their work including what was there before they installed a cable or pipe and what the installation looked like after they completed their work.

This device is well designed as the BLK360 is and really lowers the bar on ease of use much as the BLK360 did for laser scanning.  I have used other Leica DISTO devices which are clearly professional devices.  I actually held and used the BLK3D for a few minutes and the basic functionality does appear to be usable by anyone - take a picture and then measure a distance by tapping on two points on the screen is a no-brainer. There is more advanced functionality which I didn't attempt to use. For example, you can create floor plans and import existing ones. You can also add measurable images. I expect that if I could sit down with the device for an hour, I would be able to use most of its capability.

Calculating forest loss using open source deep learning tools and publicly available training data

Deep learning algorithms have been developed by academia and as a result the code for the most part is open source but the successful training of deep networks requires thousands of labeled training samples and at the present time this training data is typically not open. In their presentation at this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis Jason Brown and Courtney Whalen, data scientists at astraea, showed how deep learning using publicly available labeled data for training was applied to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil.

Chris Holmes of Planet Labs, in his insightful talk at FOSS4GNA in St Louis about the application of deep learning to geospatial data, identified a challenge in making this technology open. The deep learning algorithms have been developed by academia and as a result the code for the most part is open source. For example, a deep neural network model developed originally for medical image segmentation called U-Net is open source and has been applied to identifying building footprints. Successful training of deep networks requires thousands of labeled training samples. Labeled data involves people on the ground manually ground-truthing land use types and other features so that the deep learning algorithms can learn what to recognize. At the present time this training data is typically not open source. In this presentation by Jason Brown and Courtney Whalen, both data scientists at astraea, deep learning using publicly available labeled data for training was used to track deforestation and reforestation in Mato Grosso, a state in the central amazon region of Brazil.

This is computationally intensive and a distributed engine was used. The computation engine used open source components. Spark is a top level Apache project which enables distributed processing for global scale computation. RasterFrames is a free and open source toolkit allowing scientists, data scientists, and software developers to process and analyze geo

Forest cover in Mato Grosso in 2002

patial-temporal raster data with the same flexibility and ease as any other data type in Spark DataFrames. This is a LocationTech raster project and is built on GeoTrellis. Using this software each year required 6 to 7 hours of computation using 48 cores.

The imagery that was used was captured by the MODIS satellite for the years 2001 through 2017. MODIS monitors the reflection back from ground cover for several bands including red, green, blue, short wave infrared and near infrared. Its cameras have a spatial resolution of 500 by 500 meters and a revisit rate of one to two days. From the bands that it captures the normalized difference vegetation index (NDVI) can be calculated. From the data monthly means and yearly aggregates can be calculated.

Forest cover in Mato Grosso in 2011

The training data used came from the System for Terrrestrial Ecosystem Parameterization (STEP) which has 2000 manually labeled sites covering 17 different land cover types including five forest types scattered across all continents. The model was trained on MODIS 2012 data. 80% of the data was used for training. After training was completed the remaining 20% was used to test the model.

Comparison of rate of forest loss with Global Forest Watch for 2001 to 2016

After training and testing the first application was to Mato Grosso in central Brazil, a large state that has seen a lot of deforestation. The rate of deforestation tracked the rate estimated independently by the Global Forest Watch for the years 2001 to 2017. The major feature, the slowing down of the rate of deforestation in 2011 probably as a result of increased enforcement by the state government, is very clearly discernible.

The successful application of the deep learning technology in Mato Grosso has encouraged astraea to aim at applying this approach globally. They also intend to use satellite data with higher resolution and to handle seasonal differences better.

About STEP

The System for Terrestrial Ecosystem Parameterization (STEP) is a model for deriving vegetation and land surface parameters from remote sensing data for use in remote sensing-based classification of land cover, ecosystems, and vegetation types. The model defines parameters that relate to important ecological and biogeophysical parameters and that can be reliably measured or inferred from remote sensing, collateral, and field plot data. STEP is maintained as a database of training polygons drawn on high spatial resolution imagery that can be extracted with GIS to produce a global land cover classification. STEP is periodically reviewed to filter out inconsistent sites and augmented to fill gaps in biogeographical coverage. The database was originally created to follow the International Geosphere-Biosphere Programme (IGBP) land cover legend but it has since evolved to support any number of additional classifications.

Accurately surveying utilities exposed during construction using a smartphone

The existing documentation about the location of underground utilities is poor in many countries and the costs associated with utility strikes, hitting a utility such as a fiber-optic cable, electric power cable, or a gas or water main during construction is high. Therefore, finding a way to improve our knowledge about the location of underground subsurface infrastructure is essential, but to be practical it has to be a way that does not add significantly to the cost of construction. At Geo Business 2018, Richard Bath, a surveyor at Costain, described an efficient way of capturing enough information with an ordinary smart phone to accurately survey, map and make available via the web the location of utilities encountered during a construction project.

About 4 million excavations are carried out on the UK road network each year to install or repair buried utility pipes and cables. Not knowing the location of buried assets causes practical problems that increase costs and delay projects, but more importantly, it increases the risk of injury for utility owners, contractors and road users. The problems associated with inaccurate location of buried pipes and cables are serious and are rapidly worsening due to the increasing density of underground infrastructure in major urban areas. The average direct cost of hitting an underground utility during construction ranges from £400 to £2800 depending on the type of utility. Research has found that the total cost of a utility strike in the UK is on average 29 times the direct cost.

Performing a survey of utilities exposed during a construction project using a laser scanner and total station can be cost prohibitive and as a consequence is rarely done. A research project involving Costain and Bentley has found that photogrammetry using a consumer grade smartphone results in a 3D model of comparable accuracy to a laser scan survey and is much more cost efficient. They have also found a way to make 3D models created from the pictures captured with a smartphone available over the web.

Richard described a simple workflow to capture enough information with a smart phone to accurately determine the location of underground utilities.

• Step 1 Mark ground control points (GCPs) around the area. They have to be visible in the pictures taken with the smartphone.
• Step 2 Take pictures from varying angles and heights around the exposed utilities with a smartphone.
• Step 3 Survey the GCPs with a total station, at least three are needed to accurately determine the location of the 3D model created from the pictures.
• Step 4 Upload the photos taken with the smartphone and process them and GCPs together to create a georeferenced 3D model of the exposed utilities.

This process has been found to result in a 3D model of comparable accuracy to a full laser scan survey and unlike a laser scan survey it is something that anybody can do. Typically it involves taking 40-60 pictures with an ordinary smartphone. After uploading the pictures, processing them with Bentley's ContextCapture software to create the georeferenced 3D model, the resulting 3D model can be made available to others on the construction project using an online web link that allows the 3D model to be displayed along with existing 2D utility as-builts.

Open developer-friendly standard for querying satellite imagery

One of the things that is required to make the vast quantity of satellite imagery easily searchable is a common way to query satellite data.  The SpatioTemporal Asset Catalog, known as STAC, is an open specification that came about when fourteen different organizations came together to increase the interoperability of searching for satellite imagery. At this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis, Matt Hanson, of developmentSEED, gave a technical overview of the STAC standard and described one of the first implementations.  The context for Matt's STAC presentation was provided by Chris Holmes' keynote in the morning.

Chris Holmes and others have been working on a standard for searching satellite imagery. Currently when a user wants to search for all the imagery in their area and time of interest they can’t make just one search — they have to use different tools and connect to API’s that are similar but all slightly different. The STAC specification aims to make that much easier, by providing common metadata and API mechanics to search and access geospatial data. This and other standards are essential for opening up satellite data to processing and visualization.  This is one of a new breed of standards that are designed to be developer-friendly to encourage the open source community and others to get involved with the Open Geospatial Consortium (OGC) to further interoperability.

The STAC standard is minimalist, but balances that with extensibility.  Basically it is comprised of metadata, catalog, and API.  The core of STAC metadata is very simple, with only three mandatory fields; ID, geometry (for example, a bounding box in a defined projection), and date and time.  The standard supports extensions, of which the earth observation standard "eo" has been defined at this time.  Future extensions could include point cloud, mosaic and video extensions.  The eo extension includes parameters such as camera viewing angle, resolution (dist between pixels), percent cloud cover, and band descriptions such as RGB, SWIR, band frequency and band accuracy.

The STAC standard and cloud optimized GeoTiff (COG) files makes it possible to search and stream imagery. COGs are GeoTiff files optimized for the cloud, for example, by streaming them from Amazon Web Services S3.  Planet, Google Earth, and QGIS already support COGs and there are other open source tools such as COG-Explorer that support them as well.

Sat-api is developmentSEED's partial implementation of STAC.  It currently supports Landsat-8 and Sentinel-2 imagery.  The API allows these imagery libraries to be queried by any of the metadata fields such as date and time, location (by defining a polygon), bands, viewing angle, and so on. It returns GeoJSON.  Matt provided some example queries.

 

Find all Landsat-8 scenes from 2017 with 0-20% cloud cover

https://sat-api.developmentseed.org/search/stac?datetime=2017&collection=landsat-8&eo:cloud_cover=0/20

Find all scenes from Dec 31, 2016 through Jan  1, 2018

https://sat-api.developmentseed.org/search/stac?datetime=2016-12-31/2018-01-01

The next developments that developmentSEED is planning for the STAC standard is to add more data CBERS and MODIS and to develop some applications based on the API.

Enabling the rest of us to find actionable information in huge volumes of satellite data

There is a huge volume of satellite imagery that is being captured and has been captured over the past couple of decades.  The challenge is to automatically find objects such as ships, trees, buildings, or containers in this vast amount if data and track changes in them over time.  This morning at at this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis, Chris Holmes of Planet Labs, gave an insightful overview of the application of deep learning to geospatial data, identifying opportunities for the open source community to lead in the application of this data and technology to addressing this challenge.

Outside of Google, Planet Labs, DigitalGlobe and some others deep learning has been applied primarily to recognizing people in selfies and dogs and cats from peoples photos on online repositories such as Flickr. I have blogged previously about competitions DigitalGlobe (which has 100 petabytes of imagery) and Nvidia have hosted into automating building footprint and road network recognition. Planet Labs has a computer vision and analytics team that has been applying the results of academic research to geospatial data, primarily imagery from Planet's constellation of 150+ satellites. For example, they have used deep learning to detect building footprints and road networks in Dar es Salaam. Chris showed examples of airplane, ship and container recognition achieved using deep learning. But these are research pilots. One company has applied deep learning to scale for a practical application. Google has been developing and applying the technology to building recognition - automatically identifying building footprints and features such as aerials. The deep learning algorithms have been developed by academia and as a result the code for the most part is open source.  For example, a  deep neural network model developed originally for medical image segmentation called U-Net  has been applied to identifying building footprints.  Successful training of deep networks requires many thousand labeled training samples and this training data is not open source. Labeled data involves people on the ground manually ground-truthing building footprints and other features so that the deep learning algorithms can learn what to recognize.

Chris believed that it is time to open these models to a broader group. There are some technical things that need to happen to make this possible. We need new formats for exchanging raster and vector data. Cloud optimized GeoTiff (COG) makes it possible to stream imagery from cloud sources such as Amazon S3. Planet, Google Earth, and QGIS already support COGs and there are other open source tools such as COG-Explorer that support them as well. Chris and others are working on WFS 3.0, a completely new developer-friendly version of an existing OGC (Open Geospatial Consortium) standard for exchanging vector data over the web.

Another technical advance that is required is a common way to query satellite data based on date, time, location, type of bands, and so on.  Chris and others have been working on a standard for searching satellite imagery for objects. The SpatioTemporal Asset Catalog, known as STAC, is an open specification that came about when fourteen different organizations came together to increase the interoperability of searching for satellite imagery. Currently when a user wants to search for all the imagery in their area and time of interest they can’t make just one search — they have to use different tools and connect to API’s that are similar but all slightly different. The STAC specification aims to make that much easier, by providing common metadata and API mechanics to search and access geospatial data.  Chris emphasized that standards are essential for opening up satellite data to processing and visualization and encouraged the open source community to get involved with the OGC to further developer-friendly standards.

Currently in order to use imagery data, a lot of data preparation is required of the end user such as lining up pixels, assigning a projection, removing clouds and mist, and other corrections. We need to move toward analysis ready data where data derived from different sources can be combined for analysis.

Because of the huge volumes of computational intensity we need cloud-native engines to perform the analyses. Google, DigitalGlobe (Maxar), Planet have these deep learning engines, but Chris encouraged the open source community to get more involved in developing these engines.

An essential piece that is required to open up the data and technology to a broader audience is open labeled data, Chris referred to it as a "open labeled data commons". There are analogies with OpenStreetMap initiative. Chris suggested that something like this is required for labeled data - it needs a repository of open data, tools to capture and manage the data, and a community around this to capture and curate that data.

The other key piece is converting this data into actionable information.  For example, dashboards that would alert municipal governments to un-permitted development, forestry enforcement to illegal forest cutting, or commodity price monitoring agencies to changes into the flow of ships into harbours or to large changes in containers at container ports.

Taken together Chris suggestions and work underway represent a roadmap for the open source geospatial community to lead in making the huge volumes of satellite imagery and other data queryable and able to generate actionable information.

Deep learning enables automated extraction of building footprints and road networks from satellite imagery

Automated feature extraction from satellite imagery has made major progress in the last year. Accurate building footprints extracted from high resolution satellite imagery are available from companies such as Ecopia (which has just announced a partnership with DigitalGlobe).  Also NVIDIA has demonstrated the ability to automate detection of many road networks using sophisticated algorithms and multi-spectral high resolution imagery.

Building footprints

SpaceNet is a repository commercial satellite imagery and labeled training data being made available through Amazon Web Services at no cost to the public to foster innovation in the development of computer vision algorithms to automatically extract information from remote sensing data.  It is the result of a collaboration between DigitalGlobe, CosmiQ Works, and NVIDIA.  Two competitions for automated building footprint extraction from high resolution satellite imagery and a third for road network extraction have been held based on SpaceNet data. 

The first SpaceNet Challenge was launched in November 2016.  Imagery of Rio de Janeiro from the WorldView 2 satellite at 50cm resolution using eight spectral bands was made publicly available on Amazon Web Services (AWS).  42 developers competed to create algorithms that extract building footprints from the imagery.  The source code of the winning algorithms was made available on the SpaceNetChallenge GitHub repository. 

The winning implementation was developed by Brazilian developer using an implementation based on random forests with brute force polygon search and did not leverage deep learning frameworks.  The organizers' conclusion based on the results of Round 1 was that automated building footprint extraction remained a challenge.

Round 2 winner. Most building footprints are precisely detected. Number on each building footprint represents the overlap between the prediction and the ground truth.

Round 2 of the SpaceNet challenge involved higher resolution 30 cm imagery from WorldView 3 for four cities across the globe. The competition challenged developers to improve performance from the first competition using the higher-resolution imagery and more geographically diverse training data samples. SpaceNet on AWS held imagery and building footprints for Las Vegas, Paris, Shanghai, and Khartoum with over 180,000 buildings181,619 footprints.  In addition additional bands were made available (panchromatic, 3-band pan-sharpened, and 8-band pan-sharpened).

Results from Round 2 winning solution: top left: Las Vegas, top right: Las Vegas, lower right: Khartoum, and lower left: Paris. Blue outlines: ground truth, green outlines: correct identifications, red outlines: false positives and yellow outlines: false negatives.

The winner of Round 2 applied a  deep neural network model developed originally for medical image segmentation called U-Net.  Successful training of deep networks requires many thousand annotated training samples.  Training was aided significantly by augmenting the imagery with OpenStreetMap data.  The winning solution used OpenStreetMap layers and Worldview-2 multispectral layers as the input of the deep neural network algorithm.  In Las Vegas, the winning solution was outstanding in outlining suburban homes, but also did well in identifying odd-shaped buildings. In Khartoum, the solution was very good in identifying the footprints of stand-alone apartment complexes, but had trouble with small buildings, with building footprints that are close together, and with horizontally large buildings.  One of the problems was inconsistent ground truth which affected training. The solutions offered by the developers in the competition were sufficiently successful that the organizers concluded that the winning algorithms achieved performance with potential for automated mapping tasks such as keeping maps up-to-date and to assist first responders during natural disasters.  The source code for the winning implementations can be found in the SpaceNetChallenge GitHub repository.

DigitalGlobe has just announced a partnership with Ecopia, which has established an automated process to create building footprints quickly and at scale by leveraging DigitalGlobe's Geospatial Big Data platform (GBDX) and advanced machine learning in combination with DigitalGlobe’s imagery library. The two companies plan to automatically extract accurate 2D building footprints globally, then refresh the datasets periodically to find and track changes over time.  These datasets would be valuable to municipal governments for permitting purposes and for first responders after disasters.   Ecopia is developing a database for all of Australia (7.6 million square kilometers) which will include every building with a roof area greater than 9 square meters.  The database will contain building footprints for about 15 million buildings. The database includes linkages to other datasets including geocoded address, property data and administrative boundaries. Ecopia intend to update the dataset regularly to be able to track changes, especially urban sprawl.

Road networks

In November 2017 CosmiQ Works, Radiant Solutions, and NVIDIA announced a third round competition, “Road Detection and Routing Challenge” to explore automated methods for extracting map-ready road networks and routing information from high-resolution satellite imagery.  

Detecting road networks in Paris. Photo image, ground truth, and the results achieved with three deep learning algorithms

NVIDIA has released the results of several deep learning algorithms that illustrate just how difficult identifying road networks is and the sophistication of the available tools.  These include using multi-band spectral imagery to identify the material properties of the road surface itself (asphalt, gravel, packed earth) and using more sophisticated processing including conditional random fields (CRF), percolation theory, and reinforcement learning. 

Detecting road networks in Las Vegas. Photo image, ground truth, and the results achieved with three deep learning algorithms

NVIDIA utilized high performance GPU compute resources provided by the NVIDIA GPU Cloud and Amazon Web Services.