Between the Poles

Spatial Data Infrastructure- A Practical Example

In my last blog I outlined some alternative architectures that a large organization could implement to share spatial information.  In this blog I would like to present a practical example of the spatial warehouse architecture that adds an interesting wrinkle that may make this more universally applicable. 

The workflow for this spatial warehouse architecture involves periodically uploading information from the data sources (creators), and then providing a web interface to allow users (consumers) of this data to select and extract for download subsets of the data.  I've included a diagram showing this workflow (again thanks to Bruce Argue for this.)  This architecture corresponds to the standard IT architecture of a data warehouse.  As I mentioned previously, the advantage of this approach is that it has minimal impact on the operational subsystems that own (create and manage) the data.   

However, the most common issue with this approach is that of data format.  Frequently, the data creators are using an application from a geospatial vendor like ESRI, Autodesk, Bentley, MapInfo, Microsoft, or intergraph and likewise the consumers are using an application from a different vendor.  Often WMS/WFS may not be supported by the creator or by the consumer.  One way to get around this problem and provide universal access to everyone in the organization is to integrate Safe Software's FME on the server and on the client.  With FME the data can be converted on the upload and loaded into the spatial RDBMS (e.g., Oracle Locator/Spatial) and on the download to whatever format is appropriate for the consumer.  This way access to the data is not restricted to WMS/WFS servers and clients, which are not yet universally supported.

Spatial Data Infrastructure for Government

One of the scenarios that is becoming increasingly common especially in government at the national level is an IT solution for sharing data, often referred to as spatial data infrastructure or SDI.  In most organizations this is what I would call a Web 1.0 application  where spatial data that is created somewhere within the government, for example in the national mapping agency, and is made available through a single portal to consumers of the data.  Initially, the consumers are typically other government agencies, but over time the portal frequently evolves into a public portal.  Often in addition to providing a description of the data and access to the data, the portal will also include metadata about the data.  I've included a logical architecture (thanks to Bruce Argue for this), that illustrates what a typical SDI looks like logically.  There are creators, who create data, a consolidator or portal which provides a gateway to and metadata about the data, and consumers, who use the data.

Challenge

Typically the creators or data providers have used applications from multiple vendors to develop their datasets, so that the consolidator needs to be able to consume data in Autodesk, Oracle Spatial, ESRI, MapInfo, Intergraph,and Bentley formats and then publish this data through the consolidator portal in a common standards-based way so that everyone in the organization is able to access the data.

Solutions

From an IT perspective the solution to this type of problem rests on two basic components, standards-based interoperability and a scalable architecture.

For spatial data, fortunately the Open Geospatial Consortium (OGC) has defined standards which are becoming widely adopted in government.  The OGC's open web standards WMS and WFS are the key to interoperability in a web environment.  In the context of SDI this means that the consolidator portal needs to be able to publish WMS/WFS compliant data and consumers need to be able to consume WMS and WFS datasets.  As I mentioned in a previous blog, most vendors of geospatial products support at least WMS, and increasingly WFS.

From an architectural perspective there are several solutions to problem of publishing datasets stored in different formats through a WMS/WFS portal.

Clearinghouse

A common architecture is a clearinghouse where the portal simply includes descriptions, metadata, and an URL for each dataset.  In the past this type of approach was common, but the URL would redirect you to a web site where you could download the data in whatever format the data creator supported.  The advantage of a standards-based architecture as illustrated here is that the data is accessible through a common standards-based web services API.

An advantage of this approach is that the portal is relatively easy to simple to implement because it does not publish data directly.  In addition it puts the onus on the data creator, rather than the consolidator, to ensure that the data is accessible.

A disadvantage is that cross-dataset queries are limited to the contents of the metadata, since to query the actual data you need to go to the website of each dataset.  Another potential disadvantage is that the data creators may not  all support the same versions of the WMS and WFS standards so you may not find the same level of data access for all datasets.

Spatial Warehouse

This is probably the most typical approach at the present time. Periodically data creators upload a copy of their current dataset to the consolidator who converts it and loads it into spatial data warehouse consisting of a spatially-enabled relational database management system (RDBMS).  This process can automated so that the datasets are refreshed periodically.  The frequency with which this occurs depends on the volatility and persishability of the data. 

The primary advantage of this approach is that it has minimal impact on the operational processes of the data creators.   Another advantage is that since the data is stored in a common spatially-enabled RDBMS, spatial SQL queries on each of the datasets is supported.  Potentially cross-dataset queries can also be defined.  In addition uploads can be timed to occur at off-peak times so that during normal business hours operational systems are not impacted by large uploads occurring at times of restricted network availability.  For remote sites with poor network connections datasets can be provided to the central office on CDs.

This approach requires that RDBMS schemas for all of the datasets be defined. In general each dataset will require its own schema. You also have to identify applications such as FME for data conversion and write scripts for converting each dataset and loading it into the RDBMS.

Distributed Databases

Another approach is based on an intelligent portal that provides a common data access API to all the datesets.  This approach requires data providers for the different types of spatial datasets such as ESRI shape, Autodesk DWG, and Oracle Spatial. 

The advantages of this approach is that each dataset is current and accessible through a common programmable interface, though you should bear in mind that in general each will have its own data model and the portal needs to reflect this.  In this approach each dataset is individually queryable using the common data access API.  Another potential advantage is that this approach also allows the portal to provide cross-dataset queries.   A potential disadvantage is that in situations of limited network capacity, for example in the case of a low bandwidth WAN, there may simply not be enough available capacity, so that performance levels become unacceptable.

But with the future in mind one of the most exciting things about this approach is that it supports Web 2.0 interactivity, in other words, users can edit the data.

Integrating MapGuide Applications with Google Earth

One of the neat things about MapGuide Open Source is that it is designed to make it very easy for you to build mashups.

You've probably heard the term mashups, which refers to web applications built by integrating an application you've developed together with Google Earth/Maps to create an integrated application.  The first mashup that I am aware of was built by Paul Rademacher and integrated a real estate property list called Craig's List with Google Maps.

At the Location Intelligence 2006 (Li2006) conference Google announced Maps 2.0 Beta, which is the latest release of the Google's web services API.  One of the important things about this version is that it includes full documentation, which means that  Google will support it.  Google also announced some other things that were related and interesting; the relaxation of the 50,000 pageview limit, performance optimizations, and support for custom map types.

One of the big questions that had been on everyone's mind who had built a mashup with Google Maps is whether they would be required to carry ads.  After all Google's primary source of revenue is advertising.  At LI2006 Google announced two things, first that there would be a 90 day advance notice of a change in advertising policy, which currently does not require you to carry Google ads, and secondly that developers of mashups could opt to carry Google ads (via Google's Adsense), which would allow them to monetize their mashups.

Google and MapGuide

If you are familiar with Google Earth, you know that there are several components.  You have to download and install a free desktop application called Google Earth.  Google Maps is the web services API that allows you to integrate your MapGuide application with Google.  Google Maps also provides satellite imagery and maps from TeleAtlas and others.

By combining MapGuide functionality with the Google Earth desktop application and Google Maps web services, you can deliver your content to anyone with Google Earth.  This means that any organization can use Google Earth to share its data. For example, the results of an analysis of mobile phone coverage can be published through MapGuide to all types of audiences—technical and nontechnical—through the Google interface. MapGuide is the core analysis application, and Google Earth is the results delivery interface.  MapGuide can use Google Earth as a client by taking advantage of Google Earth's Network Links feature and the Autodesk MapGuide web application programming interfaces.

Licensing

There is no license fee to use Google Earth Basic. However, Google does charge a fee for additional functionality. Terms of use regarding the new API are found on the Google website at

www.google.com/apis/maps/terms.html.

Sample Code

To access sample code that connects MapGuide Open Source with Google:

http://data.mapguide.com/mapguide/DynamicKml/source/GoogleEarth.zip.

Sample Sites

To view the following links, you need to have Google Earth installed on your PC or Mac.

Jason Birch:

http://earth.nanaimo.ca/files/nanaimoMaster.kml

Andy Morsell, Spatial Integrators, Inc.:

http://mapguide.spatialgis.com/mapguide/GoogleEarthDemo

Autodesk Sample Demonstration:

http://data.mapguide.com/mapguide/DynamicKml/GoogleEarth.php

Interoperability using Tightly and Loosely Coupled (SOA) Architectures

One of the key trends of recent years is the integration of geospatial technology into mainstream IT. What this means practically is that standard IT architectures, tools, and standards are now being applied to geospatial solutions.

One of the major problems that IT in general faced since the beginning of IT is interoperability. Different applications, applications from different vendors, and applications running on different operating systems need to share data.  The most widely used approaches are paper and file exchange.  For example, many utilities and telecoms still rely on paper construction drawings as the solution for interoperability between engineering design (CAD) and records (GIS).

For data sharing and application integration two approaches have evolved in mainstream IT and are being applied to problems involving spatial data.  I think the simplest way to refer to them is as loosely and tightly coupled architectures.

Tightly coupled

A tightly coupled architecture integrates enterprise applications around a single point of truth, which is often a single spatially-enabled RDBMS. The types of applications that are linked include engineering design (CAD), facility records management (GIS), asset management, workflow, ERP, CRM, outage management, and other enterprise applications.  An EAI tool is sometimes used to link enterprise applications from specific vendors such as SAP.  A major advantage of a tightly coupled architecture is that it enables the rapid and efficient processing of large volumes of data, provides a single point of truth instead of several, often redundant, data sources, and enables open access to data throughout the organization.  This architecture has been widely adopted within large organizations.

Tightly coupled architectures rely on standards such as  SQL, ODBC, JDBC, and OLEDB, SQL/MM, and the Simple Feature Specification for SQL from the OGC, to provide open and secure access to data, including geospatial data, throughout the organization.  Most geospatial vendors provide the basic level of support for geospatially-enabled RDBMSs, which means practically that they support basic spatial data types, points, polylines, and closed polygons.  Many RDBMS vendors support the OGC's Sinple Feature Specfication for SQL.  The OGC with the support of geospatial and RDBMS vendors is working to extend this support to more complex data structures including oriented points and cartographic text in the short term and, I hope,  long-transactions, topology, linear referencing, and georeferenced rasters in the longer term.  The objective is to enable comprehensive geospatial interoperability, so that spatial data can be shared by different applications, applications from different vendors, and applications running on different operating systems.  At the recent Oracle Spatial User Conference, which took place right after the GITA 2006 Annual Conference, two muncipalities shared their experiences in implementing tighly coupled architectures centred around shared data in Oracle Locator/Spatial and involving applications from Autodesk, ESRI, Intergraph, GE Smallword, and MapInfo.

Loosely coupled

The loosely coupled architecture most people are familiar with currently is a service-oriented architecture (SOA) based on web services. Web services enables internet-based access to distributed data sources and applications using open standards from the W3C (http, XML, SOAP, WSDL, UDDI) and from the OGC (WMS, WFS, GML).  All geospatial vendors including support the basic geospatial web services (WMS, WFS, and GML). Autodesk and almost all of the major vendors demonstrated WMS/WFS/GML interoperability recently at GITA 2006 in Tampa.

Different Problems Means Different Solutions

I think it is fair to say that most IT professionals would agree that loosely and tightly coupled architectures address different problems and that both are necessary and they are not incompatible.  For example, a tightly coupled solution can expose web services.  I think that it is necessary for geospatial vendors to support both of these approaches. Specifically, this implies supporting web services available through mass market vendors such as Google and others, the OGC’s open web services initiatives, as well as working with the RDBMS vendors such as Oracle, the OGC, and with other geospatial vendors to move standards forward for tightly coupled architectures involving spatial data.