Between the Poles

Low cost sensor enables citizen scientists to measure and share urban air pollution data over the web

The World Health Organization (WHO) estimates that more than seven million deaths every year are linked to air pollution exposure from household and ambient (outdoor) air pollution. But in spite of these statistics the average city has only one or two air quality monitoring devices which are very expensive costing about $1.5 million each. At this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis Steve Liang, a professor at the University of Calagary and CEO of SensorUp, described a way that citizen scientists can contribute to measuring air pollution using a low cost board and CPU and share via open source geospatial web software to map the resulting measured air pollution in real time.

Based on the latest results from WHO air pollution is now the world’s largest single environmental health risk, linked to 12% of all global deaths. Around 4.3 million deaths every year are attributed to exposure to household (indoor) air pollution, from heating, cooking and lighting using solid fuels. Around 3.7 million deaths every year are linked to outdoor air pollution , including exposure to fine particulate matter from fuel combustion from vehicles and from power plants, industry and biomass burning.

Steve described the sensors, which are typically built by interested people in workshops led by staff from SensorUp, a startup that Steve leads. The devices consist of a sensor that measures PM2.5 (particles less than 2.5 microns in size), temperature and humidity, a CPU and Wifi. Each device costs less than $100. After building the device, each participant in the workshop brings it home and installs it outside within wifi range of their internet router. The sensor reports ambient PM2.5, temperature and humidity every 5 minutes to a central server maintained by SensorUp. It uses the Open Geospatial Consortium (OGC) standard SensorThings API which provides an open and unified way to interconnect Internet of Things (IoT) devices, data, and applications over the Web. SensorUp SensorThings platform is the most advanced OGC SensorThings API implementation.

A web application built on the open source geospatial software Leaflet allows users to view the data on a map in real time, investigate air pollution historically and compare different cities. The data is open and accessible to anyone.  To date about 500 people across Canada have built their own devices and are sharing measured PM2.5. The real-time feed from these devices is mapped here.

Open developer-friendly standard for querying satellite imagery

One of the things that is required to make the vast quantity of satellite imagery easily searchable is a common way to query satellite data.  The SpatioTemporal Asset Catalog, known as STAC, is an open specification that came about when fourteen different organizations came together to increase the interoperability of searching for satellite imagery. At this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis, Matt Hanson, of developmentSEED, gave a technical overview of the STAC standard and described one of the first implementations.  The context for Matt's STAC presentation was provided by Chris Holmes' keynote in the morning.

Chris Holmes and others have been working on a standard for searching satellite imagery. Currently when a user wants to search for all the imagery in their area and time of interest they can’t make just one search — they have to use different tools and connect to API’s that are similar but all slightly different. The STAC specification aims to make that much easier, by providing common metadata and API mechanics to search and access geospatial data. This and other standards are essential for opening up satellite data to processing and visualization.  This is one of a new breed of standards that are designed to be developer-friendly to encourage the open source community and others to get involved with the Open Geospatial Consortium (OGC) to further interoperability.

The STAC standard is minimalist, but balances that with extensibility.  Basically it is comprised of metadata, catalog, and API.  The core of STAC metadata is very simple, with only three mandatory fields; ID, geometry (for example, a bounding box in a defined projection), and date and time.  The standard supports extensions, of which the earth observation standard "eo" has been defined at this time.  Future extensions could include point cloud, mosaic and video extensions.  The eo extension includes parameters such as camera viewing angle, resolution (dist between pixels), percent cloud cover, and band descriptions such as RGB, SWIR, band frequency and band accuracy.

The STAC standard and cloud optimized GeoTiff (COG) files makes it possible to search and stream imagery. COGs are GeoTiff files optimized for the cloud, for example, by streaming them from Amazon Web Services S3.  Planet, Google Earth, and QGIS already support COGs and there are other open source tools such as COG-Explorer that support them as well.

Sat-api is developmentSEED's partial implementation of STAC.  It currently supports Landsat-8 and Sentinel-2 imagery.  The API allows these imagery libraries to be queried by any of the metadata fields such as date and time, location (by defining a polygon), bands, viewing angle, and so on. It returns GeoJSON.  Matt provided some example queries.

 

Find all Landsat-8 scenes from 2017 with 0-20% cloud cover

https://sat-api.developmentseed.org/search/stac?datetime=2017&collection=landsat-8&eo:cloud_cover=0/20

Find all scenes from Dec 31, 2016 through Jan  1, 2018

https://sat-api.developmentseed.org/search/stac?datetime=2016-12-31/2018-01-01

The next developments that developmentSEED is planning for the STAC standard is to add more data CBERS and MODIS and to develop some applications based on the API.

Enabling the rest of us to find actionable information in huge volumes of satellite data

There is a huge volume of satellite imagery that is being captured and has been captured over the past couple of decades.  The challenge is to automatically find objects such as ships, trees, buildings, or containers in this vast amount if data and track changes in them over time.  This morning at at this year's FOSS4GNA (Free and Open Source Software for Geospatial North America) get together in St Louis, Chris Holmes of Planet Labs, gave an insightful overview of the application of deep learning to geospatial data, identifying opportunities for the open source community to lead in the application of this data and technology to addressing this challenge.

Outside of Google, Planet Labs, DigitalGlobe and some others deep learning has been applied primarily to recognizing people in selfies and dogs and cats from peoples photos on online repositories such as Flickr. I have blogged previously about competitions DigitalGlobe (which has 100 petabytes of imagery) and Nvidia have hosted into automating building footprint and road network recognition. Planet Labs has a computer vision and analytics team that has been applying the results of academic research to geospatial data, primarily imagery from Planet's constellation of 150+ satellites. For example, they have used deep learning to detect building footprints and road networks in Dar es Salaam. Chris showed examples of airplane, ship and container recognition achieved using deep learning. But these are research pilots. One company has applied deep learning to scale for a practical application. Google has been developing and applying the technology to building recognition - automatically identifying building footprints and features such as aerials. The deep learning algorithms have been developed by academia and as a result the code for the most part is open source.  For example, a  deep neural network model developed originally for medical image segmentation called U-Net  has been applied to identifying building footprints.  Successful training of deep networks requires many thousand labeled training samples and this training data is not open source. Labeled data involves people on the ground manually ground-truthing building footprints and other features so that the deep learning algorithms can learn what to recognize.

Chris believed that it is time to open these models to a broader group. There are some technical things that need to happen to make this possible. We need new formats for exchanging raster and vector data. Cloud optimized GeoTiff (COG) makes it possible to stream imagery from cloud sources such as Amazon S3. Planet, Google Earth, and QGIS already support COGs and there are other open source tools such as COG-Explorer that support them as well. Chris and others are working on WFS 3.0, a completely new developer-friendly version of an existing OGC (Open Geospatial Consortium) standard for exchanging vector data over the web.

Another technical advance that is required is a common way to query satellite data based on date, time, location, type of bands, and so on.  Chris and others have been working on a standard for searching satellite imagery for objects. The SpatioTemporal Asset Catalog, known as STAC, is an open specification that came about when fourteen different organizations came together to increase the interoperability of searching for satellite imagery. Currently when a user wants to search for all the imagery in their area and time of interest they can’t make just one search — they have to use different tools and connect to API’s that are similar but all slightly different. The STAC specification aims to make that much easier, by providing common metadata and API mechanics to search and access geospatial data.  Chris emphasized that standards are essential for opening up satellite data to processing and visualization and encouraged the open source community to get involved with the OGC to further developer-friendly standards.

Currently in order to use imagery data, a lot of data preparation is required of the end user such as lining up pixels, assigning a projection, removing clouds and mist, and other corrections. We need to move toward analysis ready data where data derived from different sources can be combined for analysis.

Because of the huge volumes of computational intensity we need cloud-native engines to perform the analyses. Google, DigitalGlobe (Maxar), Planet have these deep learning engines, but Chris encouraged the open source community to get more involved in developing these engines.

An essential piece that is required to open up the data and technology to a broader audience is open labeled data, Chris referred to it as a "open labeled data commons". There are analogies with OpenStreetMap initiative. Chris suggested that something like this is required for labeled data - it needs a repository of open data, tools to capture and manage the data, and a community around this to capture and curate that data.

The other key piece is converting this data into actionable information.  For example, dashboards that would alert municipal governments to un-permitted development, forestry enforcement to illegal forest cutting, or commodity price monitoring agencies to changes into the flow of ships into harbours or to large changes in containers at container ports.

Taken together Chris suggestions and work underway represent a roadmap for the open source geospatial community to lead in making the huge volumes of satellite imagery and other data queryable and able to generate actionable information.

How much does it cost to run a FOSS4G conference ?

Since the formation of the Open Source Geospatial Foundation (OSGEO) in 2006, there has been an annual FOSS4G event that brings together open source geospatial developers, users and supporters from around the world. The first event in Lausanne, Switzerland attracted 560 participants and last year's event in Boston, USA brought together over 1,100. The first FOSS4G event I attended was in 2007 in Victoria, British Columbia and what I remember most was the energy and collaborative spirit among all the participants. These are all volunteer initiated and run events and usually generate a positive financial return.

There are also regional FOSS4G events around the world. At this year's FOSS4GNA (FOSS4G North America) get together in St Louis, Michael Terner, who helped organize the Boston FOSS4G event, gave an overview of what is involved in organizing a FOSS4G event including a financial breakdown of the revenue and expenses for the Boston meeting.

This year's FOSS4G meetup will be held in Dar es Salaam, Tanzania, the second time the event has been held in Africa. The venue for the 2019 meeting has been decided - Bucharest, Romania. If you are interested in staging this event, Michael's breakdown of the revenue and costs will be very helpful in budgeting for such an event.

The Boston event generated about $1 million in revenue primarily from registrations, but 25% of the total was provided by sponsors. The space for the event was provided by Harvard University. It employed a professional conference organizer, but a lot of the work in making the event happen including finding sponsors came from volunteers. OSGEO provided about $70,000 as an advance to help get things started. The single biggest cost, nearly half a million dollars, was the venue and food. The next biggest costs were the social events and the audio-visual plus recording, each of which cost over a hundred thousand dollars. The Boston event generated about a $140,000 surplus for OSGEO.

Extensive use of geospatial open source software by the National Geospatial Intelligence Agency

The National Geospatial-Intelligence Agency (NGA) is a U.S. Government intelligence agency and Department of Defence combat support agency. But it also provides support for disaster relief and humanitarian aid. At FOSS4GNA today in St. Louis I found out that NGA is the largest consumer of geospatial data in the U.S. and the first intelligence agency to contribute to Github.

Mark Munsell of NGA gave a very interesting keynote at FOSS4GNA this afternoon. Until a decade ago NGA did not use open source software. But NGA found that relying on proprietary software left it lagging behind. About 10 years go NGA decided to begin using open source software. It acts as a user of and contributor to open source code and as a supporter of the geospatial open source community. Boundless, Postgres, OSSIM, MapStory, OpenSensorHub, geomesa, GeoTrellis and others have all been supported by NGA. Internally NGA provides geospatial services to other intel services as well as the Department of Defence. NGA has implemented a Data Corps consisting of data scientists and specialists that can work on specific problems in any intel agency.

In the mid 1990s the U.S. government decided to outsource most development work and laid off most the the coders working in the intelligence community. They subsequently realized that they had lost an incredible wealth of expertise and knowledge as a result of that policy decision. Under the name of Dev Corps NGA now wants to hire developers, especially from the geospatial open source development community Mark emphasized.  Mark finished off with an impassioned plea for geospatial open source developers to continue contributing as they have been, but if they are interested in working for an agency that does do humanitarian and disaster relief in addition to intel and defence, he wants to talk to them.

Open source viewer sharing all city's geospatial data can run on Raspberry Pi

Locations of Cross Section surveyors' notebooks

At GITA 2018 in Phoenix, Bob Basques, GIS Systems Developer at the City of St Paul, described a system called COMPASS he and his team have developed that provides a shared, easy to use tool that allows city employees and the public access to all of the City's spatial and associated data including, for example, scans of surveyors' notebooks, 2.2 million street level photos, and permitting and licensing information from 200 different applications. Based completely on open source components the system is compact and efficient enough to run on a Raspberry Pi.

One of the problems that hampers efficient operations at municipalities is accessing geospatial data originating from multiple sources such as infrastructure maintenance, planning and zoning, property ownership, engineering, permitting, licensing and code enforcement. Spatial data is generated by CAD drafters, GIS users, surveyors and even users with smart phone apps. Imagery data can come from a variety of sources including earth observation satellites, aerial overflights, street photography, and drones. This data is constantly changing which means that any process that involves making copies such as converting it to a common format creates a bigger problem than it solves.

Location and maintenance data for each street light

The City of Saint Paul is responsible for infrastructure maintenance for all of the infrastructure within the public right-of-way. It is also responsible for maintaining parcel information linking and addressing assignment. St Paul is only 56 square miles, but it is densely developed and there is a lot of infrastructure and the associated regulations to manage. In addition St Paul is part of the metro area of the Twin Cities and users of St Paul data often need access to data outside of St Paul.

Organizations participating in GeoMoose project

The application is intended to improve the efficiency of common tasks including infrastructure maintenance management, planning and zoning, property ownership, engineering, permitting. licensing and code enforcement. To do this the system accesses multiple disparate data sources from different data custodians including Public Works, Safety and Inspections, Planning and Economic Development, Parks & Recreation, Office of Technology, Fire, Police, Real Estate, and District Councils. The public has access to 73 layers of data through OGC web standard interfaces. The tool, called GeoMoose, is an open source project (MIT license) on OSGeo and uses only open source components. There are quite a few governmental agencies around the U.S participating in the GeoMoose project.

Creating a project polygon and buffer to identify affected parcels and owners

The application is extremely compact and light weight - Bob demonstrated it running an a Raspberry Pi with a 64 GB drive (just about any size MicroSD card will work) - which means that users working in areas with restricted wireless can still access important data. GeoMoose runs on Linux and Windoes and can be accessed via Wifi from Windows, Linux, Android, iOS and Mac OS - actually any device with Wifi and a browser including a USB stick. The core of GeoMoose is comprised of HTML and Javascript code. It can access data stored in the file system as well as from OGC WMS/WFS sites over the internet. GeoMoose included an editor that allows users to create geometric objects such as polygons. A typical application would be to define a polygon representing a new project, put a buffer around it and find all the parcels that would be impacted by the project.

The system handles many different data types including 305 layers ( a new layer is added every week or two), 32 live database connections, 39 imagery layers, and 31 infrastructure layers. Besides Public Works users, users from more than 40 other organizations access the system. The system is an OSGeo project and is built on open source components including OpenLayers and MapServer, and can be integrated with PostgreSQL/PostGIS, Webdav, svn and Apache.

Locations of surveyors' notebooks

COMPASS exposes data to the world via OGC standard tools. Most of the data in the COMPASS system of tools is available as a WMS service. Selected datasets are also available as WFS layers. Some layers are protected for security or contractual reasons. GeoMoose 3.0, which was released in August of last year is mobile friendly and supports modern Javascript frameworks such as React.

Digital images of surveyors' notebooks can be accessed

The system has become widely used and trusted. It gets about 50,000 hits per week not only within Public Works but also by other users inside and outside of the city government. It has eased many previously labour intensive processes. Because more users are accessing their data, custodians of the data are taking greater responsibility for data which has led to an overall improvement in data quality and timeliness. Open source has allowed customization for specific users that would not have been possible without access to the underlying code. For example, it has made it possible to create specialized visualizations and reports, connecting to various proprietary databases, mixing data sources on output and building custom lookup processes.

Open source-based mobile viewer creates value for electric power utilities with multiple GISs

One of the challenges faced by large utilities with multiple GIS, outage management, customer information and other systems, which are often the result of mergers, is how to make spatial and associated data, in different, often incompatible systems, accessible to all users not just GIS professionals.  At Distributech 2018 Duke Energy described an off-the-shelf web-based product based on open source components that is being used by 8000 users, 2000 of them using it 24/7 every day, to access GIS and associated data in several GIS systems, both online and offline, using laptops, tablets and phones.  An important advantage of the the selection process Duke used in selecting the product was that it made it possible for a relative newcomer to compete successfully with the traditional GIS incumbents.

Background

Large investor-owned utilities are often the result of mergers. For example, in 2012 Duke Energy and Progress Energy merged to create the largest U.S. utility with 7.4 million U.S. customers and approximately 52,700 megawatts of electric generating capacity. The challenge for the IT folks is how to manage the many disparate IT systems that such mergers bring together. Historically Duke Energy and Progress Energy have employed all the major GIS systems including Synercom, Intergraph FRAMME, GE Smallworld, ESRI, Intergraph GTechnology, and IBM GFIS. Accessing the valuable operational data in multiple systems is a challenge because each system has its own user interface and command structure. Over the years I have seen a number of solutions that have attempted to address the challenge of providing a common viewer for visualizing data from multiple data sources. But each has had limitations, for example, all data had to be synced to a central Oracle database, or it could only run online making it difficult to deploy to the field, or it had limited support for the many GISs and user files used by utilities, or it supported iOS on handheld devices but not Android, or did not support open standards such as OGC WMS and WFS, or did not support Google Maps, or was so complex users required extensive training and it was like pulling teeth to geth them to use iit.

At Distributech 2018 Ron York, Lead Consultant/ GIS, and Francisco Sarmento, Lead Consultant, both at Duke Energy described how Duke has resolved the problem of a common viewer for several different GIS systems which was brought to a head by the merger of Duke and Progress. They also described the somewhat unusual process by which Duke selected the vendor whose product provided the successful solution.

Open source geospatial components

A key differentiator of the product Duke chose is that it is leverages open source components; PostGIS, GeoServer and Leaflet being the main ones. The product builds on the collective wisdom, code and support provided by the worldwide open source geospatial community.  Leaflet is the leading open-source JavaScript library for mobile-friendly interactive maps. PostGIS adds spatial capabilities to the PostgreSQL relational database. It extends PostgreSQL so it can store, query, and manipulate spatial data. GeoServer is a web server that allows maps and data in a variety of formats to be served to standard clients such as web browsers and desktop GIS programs. Data is published via Open Geospatial Consortium (OGC) standard interfaces, such as WMS and WFS.  These are all Open Source Geospatial Foundation (OSGEO) projects.

Multiple GIS support

The product Duke selected supports ESRI ArcGIS, GE Smallworld, Intergraph GTechnology, spatialNET and Google Maps. At Duke it is used to access data in a couple of GISs as well as non-GIS data in Maximo, SAP, three different outage management systems, four or five customer information systems, and can link to CAD drawings and photos and images.  Importantly for engineering users it preserves the original symbology and background maps of the different GIS systems.

Online and offline

Another key advantage of Duke's solution is that it can run equally well offline and online and is able to seamlessly combine and switch between online and offline data.

Mobile and office devices

It supports mobile as well as office devices, running on laptops, tablets and phones running iOS, Android, and Windows. The user interface adapts automatically for each device, displaying map layers, forms and other data in a way that is appropriate for each device.

Synced or direct access to databases

It provides two ways of accessing data; one is to periodically import or sync data into the system's spatial database, which uses PostGIS, and the other is to directly access the data via a web service. It supports ESRI REST Services, which are used by ArcGIS Online and ArcGIS Enterprise, and the industry standard OGC WMS and WFS web services, which are supported by all major geospatial vendors.

It does not support direct edit of spatial data, but does allow a GIS data correction requests to be routed through Maximo following Duke standard business procedure for making corrections.

It is flexible enough to support different business processes for collecting and organizing data. For example, pole inspections are conducted by circuit in what was Progress service territories and by square area in what was Duke's service territory.

Simplicity of use

Another of the key differentiators of Duke's web-based solution is that it is not targeted on GIS professionals, but on the 95% of people within Duke who can get a lot of value from a really simple Google Maps style interface onto their enterprise data.

Leading up to and during deployment Duke put on road shows, made web demos available online, utilized computer-based-training, and provided job-aid documents to make people aware of the new system.  The tool caught on surprisingly rapidly. Within five months of it being deployed there were 5000 users. Currently there are 8000 users with over 2000 users accessing it 24/7 every day.

Anticipated and unanticipated benefits

The major expected benefit that was realized by the system is that eases the integration of Duke and Progess. All GIS asset data is exposed through a single tool to designers and asset engineers. But there were some unanticipated benefits as well.  It has had a direct impact on tax calculation - it shortened the cycle for internal auditors to complete tax assessments.  It is also widely used during storms and other natural disasters. It enables crews from different area to use a familiar tool to analyze unfamiliar data in areas where they haven't been before.  It is used to provide driving directions for field crews.  Storm managers overlay outages, assets, flood plains, wind, rain, traffic  and other maps for situational assessment.  It allows manager to share staging sites during storms. It has also  proved extremely useful in locating "unmatched outages" - outages called in by first responders with just an address.

Process for choosing a vendor levels playing field for new entrants

Duke used an interesting process to select the vendor. Preliminary vendor interviews were conducted to exclude vendors that clearly didn't have a solution that could meet Duke's needs. Remote web demos provided a cheap and easy way to narrow down the list of vendors. A request for proposal (RFP) process was used to solicit technical-only proposals from product vendors. (Duke was only interested in off-the-shelf products not bespoke solutions.)  Duke picked the top five and asked them to come to Duke for a scripted demonstration. Instead of asking the vendors for a benchmark demonstrating how their product would address Duke's requirements using Duke data, they asked the vendors to demonstrate their product using their own data. They picked the top two vendors and began contract negotiations to decide on one vendor which turned out to be Ubisense MyWorld, a relatively new entrant in the geospatial world.  An important advantage of the selection process Duke used is that it made it possible for a relative newcomer to compete successfully with traditional GIS incumbents.

Avoiding a repeat of the CAD/GIS quaqmire of the past quarter century

Yesterday at Distributech 2018 in San Antonio I listened to a talk from a consulting company contracted by an electric power utility to clean up GIS data originally sourced from CAD drawings.  This reminded me once again that is a challenge that telecommunications and cable companies and electric and water utilities have been attempting to deal with at least since the early 1990s and for some even before that.  Telecommunications and cable and leading electric power utilities resolved this problem at least a decade ago, but for some electric power and even more water utilities it remains a problem that is going to hold these utilities back when they should be accelerating their adoption of digital technologies.

I began working in the utility and telecom sector in 1993, specifically in developing software for records management (called network documentation outside of North America), and at that time the magnitude of the AEC+geospatial cultural mismatch problem was just beginning to be appreciated by utilities and telecoms. In utilities and telecoms planners tend to use GIS tools, engineers and designers CAD tools, construction folks in the field paper CAD drawings, and asset managers GIS integrated with their FM tools. Fashioning AEC and geospatial data into an efficient data flow from planning through design and construction to operations and maintenance represented a challenge that remains a problem for utilities. The Between The Poles blog is over ten years old and one of the persistent themes from the very beginning in 2006 was the challenge of integrating CAD and GIS data in a common workflow (some examples; 2006, 2007, 2008 ).

A workflow that is found at every utility and telecom involves the design-build-maintain information flow. The workflow begins with an engineer and drafter producing a design, typically a CAD drawing, which is printed for the construction contractor to use in building a facility. The contractor uses the drawing in the field and then returns it as an "as-built" to the utility's records management group, who enter it, traditionally by manually redigitizing it from the paper drawing, into a GIS that is intended to contain the digital twin of the network. The data in the GIS is used for network maintenance, outage management and other functions. The reason for the manual redigitization step is the "errors" that CAD drawings inevitably contain unless drafters are forced to adhere to "geospatial rules". This entire process is slow and error-prone and backlogs of as-builts waiting to be entered into the GIS stretch into months and even years. The result is that utility GIS data is more often than not incomplete, out-of-date, and unreliable. For example, at the beginning of the broadband revolution, telecoms needed to be able to tell customers whether they were close enough to a terminal and fiber to be eligible for ADSL. In one of the largest telecoms in the U.S. the GIS records were so incomplete and unreliable that survey teams had to be sent out at considerable cost to resurvey the location of their terminals and fiber cables.

Because there was no standards-based solution to the CAD+GIS problem, every utility and telecom has had to try to find its own solution to the problem. This problem has persisted from the 1990s and remains a problem for utilities - as I said I heard a presentation yesterday that wasn't that different from presentations on the same topic I and others gave a quarter of a century ago.

Over the years the poor CAD/GIS interoperability has cost the utility and telecom sectors considerable amounts of money. Large telecoms have spent hundreds of millions of dollars trying to cleanup their GIS data and streamline their CAD+GIS workflows. A rough back-of-the-envelope calculation suggests this problem has cost directly and indirectly trillions of dollars over the past 20 years. In addition unreliable and out-of-date network infrastructure data that results from this problem has safety implications.  

The juncture between distinct sectors each with its own technologies and business models, AEC (Autodesk, Bentley, and others) and geospatial (ESRI, Hexagon, and others), has analogies with the clash of tectonic plates and the associated incessant seismic events. In the mid-1990s pressure from customers motivated Autodesk and ESRI, 800 lb gorillas in their respective areas of CAD and GIS, to attempt to collaborate to address the CAD+GIS problem. The result of the agreement between Jack Dangermond and Carol Bartz at the time resulted in an unsuccessful product called ArcCAD, which satisfied neither CAD nor GIS users.

Averting a tectonic clash repeat: BIM+geospatial interoperability

The prospect that, by analogy with CAD+GIS problem, we could still be individually finding solutions to BIM/geospatial integration 20 years from now is discouraging. While there are parallels between CAD/GIS and BIM/geospatial, there are grounds for optimism that this time around we may find ways the address the problem in a general way. Firstly, in the mid 1990s there were virtually no geospatial or 3D AEC data standards. The Open Geospatial Consortium (OGC) and builsingSmart were both only founded in 1994. It required quite a while for the OGC geospatial and buildingSmart's IFC standards to broadly penetrate industry and government.  Nowadays, there are many widely adopted geospatial standards; Simple Features for SQL, WMS, WFS, GML, KML, InfraGML, CityGML to name just a few from the OGC. buildingSmart's IFC standard is supported by all major BIM design tools including the market leader Revit.

Perhaps most significantly there has been significant progress in developing common conceptual models for geospatial and AEC views of city infrastructure. The major breakthrough in bringing the architectural and geospatial views onto a common footing is the OGC LandInfra Conceptual Model developed by the OGC in cooperation with buildingSMART International and approved as an OGC standard in August, 2016. The LandInfra conceptual model was developed by Bentley Systems, Leica Geosystems, Trimble, Australian Government Department of Communications, Autodesk, Vianova Systems AS, and buildingSMART International and provides a unifying basis for land and civil engineering standards including the OGC's InfraGML and buildingSmart International's IFC for infrastructure standards.  InfraGML is the OGC's application schema supporting land development and civil engineering infrastructure facilities. InfraGML supports a subset of the existing LandXML standard. buildingSmart International's IFC-Alignment project uses this common conceptual model of alignments for roads, railways, tunnels and bridges. The objectives of the IFC-Alignment project is to enable the exchange of alignment information through the full infrastructure lifecycle from planning through design and construction to asset management.

Secondly, in the 1990s and the early 2000s there were few open source geospatial projects. The Open Source Geospatial Foundation (OSGEO) was only founded in 2006. Since then there has been rapid development of open source geospatial projects. There is enough open source geospatial code now that at the GEO|Design+BIM conference in Amsterdam, Sanghee Shin described a web-based geo-platform integrating massive, complex BIM models using open source geospatial libraries and tools including PostGIS, GeoServer, and Cesium and NASA WorldWind virtual globes. The importance of this project is that it shows that it is possible using the available open source geospatial APIs and libraries - together with some genuine innovative development - to create an open, non-proprietary, 3D geospatial platform for integrating geospatial and BIM. Given the critical importance of addressing the cultural and technical divide between the AEC and geospatial worlds, clearly many believe that a viable open source alternative will be essential for developing the innovative solutions to the challenge of interfacing the two worlds. 

Finally, major players in the AEC and geospatial industries, Autodesk and ESRI, have again agreed to collaborate. This time around they are probably more aware of the technical issues hindering BIM+geospatial integration. Both have adopted open standards when it made business sense - often under pressure from government clients - and both have also adopted open source technology where it made business sense. Together the combination of major players supporting the effort, standards organization like the OGC and buildingSmart already working to address the problem, and a broad range of open source code and a vibrant developer community may be enough to address the technical problems. If building and infrastructure owners and facilities managers including government as well as regulators become aware of the risk of not solving the problem, a solution may be found so that we will not be having the same BIM+geospatial integration discussion a quarter of a century from now.

An open source web-based platform for integrating BIM and 3D geospatial

BIM+geospatial integration is being driven by a trend toward infrastructure construction projects that require maintenance and/or operations during an extended period post-construction.  At the GEO|Design+BIM conference in Amsterdam, Sanghee Shin, CEO of Gaia3D Inc., gave a riveting presentation about a web-based geo-platform integrating massive, complex BIM models using open source geospatial libraries and tools including PostGIS, GeoServer, and Cesium and NASA WorldWind virtual globes.

Drivers of BIM+geospatial integration

Since the world became aware of the importance of human impact on the environment, it has become increasingly difficult to design buildings without reference to geographic context.  Many cities have environmental zoning and other types of by-laws that require designers to take into account neighboring buildings, infrastructure and vegetation.  States and national governments are developing building codes that require designers to assess the energy, water, emissions, and other environmental impacts of buildings.  This means that when designing and constructing a building or renovating existing buildings, the traditional approach ( often associated with a 0,0 coordinate system ) of design in isolation is no longer adequate for sustainable design.

Furthermore integrated geospatial+BIM is essential during the operations and maintenance phase of a building.  Utilities, transportation, emergency planning, first-responder access, energy performance, and evacuation management all require information about internal and external infrastructure.  A common geospatial coordinate system enables a comprehensive operational view of all infrastructure including internal and external structures. 

BIM+geospatial integration provide greater value to projects that involve not just design and construction but also operations and maintenance.  A leader in this space, Rijkswaterstaat, the Dutch transportation authority, began offering design-build-finance-maintain (DBFM) projects a number of years ago which has motivated private Dutch engineering and construction companies to adopt an integrated geospatial+BIM approach to construction.  For example, the firm Royal BAM Group nv/BAM Infraconsult adopted integrated BIM+geospatial because of market developments including more complex construction assignments and an increasing demand from customers for service provision throughout the entire life cycle of a project.

It is indicative of the importance of the integration of these technologies that Autodesk and ESRI, 800 lb gorillas in their respective areas of BIM and GIS, have announced an agreement to collaborate to help bridge the gap between BIM and 3D GIS.  This is their seond attempt.  The first attempt, an agreement between Jack Dangermond and Carol Bartz in the mid 90s, resulted in a product called ArcCAD, intended for civil engineers and planners.

An open source geo-platform for integrating BIM+geospatial

mago3D is a new 3D geo-platform developed by Gaia3D.  It is open source and integrates many well-known open source geospatial products and APis. It is web-based so it is not necessary to install anything to run it. It has an engine for handling massive, complex BIM objects based on an internal format (F4D) that was created to reduce network data traffic, enable LOD (level of detail) control, support octrees, a 3D spatial structure that enables efficient spatial queries and (frustrum and occultation) culling.  Culling refers to the removal of objects that are not visible to the observer and is key to efficient visualization or large, complex structures. It integrates with two of the best known open source 3D virtual globe tools, Cesium (supported by AGI and Bentley) and NASA WorldWind, for visualization in a geospatial context.  Viewing BIM structures in mago3D has some similarities with a Google Earth experience. The source code is available under Apache and AGPL (Affero GPL) open source licenses.  Cesium is an open source project supported by AGI and Bentley.

mago3D includes a F4D Converter that converts BIM models in IFC (buildingSmart Industry Foundation Classes) format as well as 3D objects in .3ds (Autodesk 3ds Max), .obj (Alias/Wavefront) and .dae (Collada) formats into its internal F4D format.  Industry Foundation Classes (IFC) is a vendor neutral and open model that most industry software BIM tools including Revit and Tekla are able to export, although there is typically some information loss. The bimserver.org open source project provides an open repository of IFC models and offers a Java API to extract partial building information models from a repository.

The main components of mago3D are mago3D.js, a javascript plug-in for WebGL virtual globes and open source web server (NGiNX), map server (GeoServer), geospatially-enabled database (PostgreSQL+PostGIS), 2D mapping (OpenLayers, Leaflet) and 3D visualization (Cesium or NASA WorldWind). The mago3d.js javascript plugin is designed to run in a WebGL virtual globe environment in a browser so that moving/rotating objects such as BIM models and heading adjustment happen in-browser and are very fast. 

I went to the mago3D site and viewed a BIM model of a chemical manufacturing facility in South Korea.  I was impressed with the ability to zoom ala Google Earth from the entire virtual globe down to viewing the chemical facility in its geospatial context.  I could continue down to a very detailed, almost nuts and bolts, level showing that the LoD control and the octree-based culling algorithms make visualization very efficient, even for large complex objects.

The importance of this technology is that it shows that it is possible using the available open source, geospatial APis and tools - together with some genuine innovative development - to create an open, non-proprietary 3D geospatial platform for integrating geospatial and BIM.  Given the critical importance of addressing the cultural and technical divide between the AEC and geospatial worlds, a viable open source alternative is essential for developing the innovative solutions to the challenge of interfacing the two worlds. 

The other part of the solution is common, open standards. There has been encouraging progress on that front.  The major breakthrough in bringing the architectural and geospatial views onto a common footing is the OGC LandInfra Conceptual Model developed by the OGC in cooperation with buildingSMART International and approved as an OGC standard in August, 2016.  LandInfra  was developed by Bentley Systems, Leica Geosystems, Trimble, Australian Government Department of Communications, Autodesk, Vianova Systems AS, and buildingSMART International and provides a unifying basis for land and civil engineering standards including the OGC's InfraGML and buildingSmart International's IFC for infrastructure standards.  

Open source geospatial application reduces copper cable theft by 70%

A highlight of the SA Geotech conference in Johannesburg was a presentation by Ryno Goosen of Suritec Geospatial.  Ryno, who was awarded the Best Paper prize at SA Geotec, gave a fascinating presentation about how he developed an open source GIS solution to fight copper cable theft.

In Canada copper cable theft costs the Canadian electricity sector an estimated $40 million annually.  In South Africa in 2014 there were 72,533 reported incidents of copper cable theft resulting in 10,736 arrests.  Copper theft incidents in the U.S. have tripled in the last 5 years.

Goosen Goosen's solution architecture was based on open source geospatial software.  He chose open source for several reasons.  First and foremost, it enabled massive scalability by providing an elastic IT infrastructure.  Secondly it provided open interoperability with other open source software as well as proprietary systems. He also emphasized the importance of using software that was "battle-tested" in both military and commercial environments. 

His solution architecture was built on Cesium, GeoServer, Boundless Desktop/QGIS, Protobuf, Cassandra, WildFly, Solr and GeoTrellis.  Using this solution he was able to combine data from different sources including ground patrol and remote sensors.  By analyzing this data geographically and temporally he was able to determine spatial and temporal patterns of copper cable theft.  This enabled optimum deployment of resources.  The results were impressive.  Over a 30-month period cable theft was reduced by 69%.

The success of Goosen's shows that open source GIS software has become a viable alternative for building complex and highly scalable solution-oriented architectures.  He also suggested that it levels the playing field for small to medium size enterprises by providing them with similar intelligence capabilities to their larger counterparts.