Between the Poles

MEST 2009: Spatial Data, Convergence, Intelligent Models, Collaboration, Open Source Geospatial, GeoWeb, and Fusion

One of the distinguishing characteristics of the MEST conferences, is the daily wrap up where one of the organizers briefly reviews and summarizes the day's presentations.  I find the exercise very worthwhile because it gives you a chance to review and digest what you've heard during the day's proceedings.  There is also a session at the end of the conference, where the entire conference is reviewed by several of the speakers and organizers. 

By way of a little background, MEST is organized by the Bahrain Society of Engineers, so I find it more akin to the infrastructure conferences I attend than a traditional GIS conference.   Although it has commercial sponsors, it is non-denominational and this is reflected in most of the presentations.  It also tends to attract government people from many of the Gulf states, Bahrain, Yemen, Oman, Abu Dhabi, Dubai, and Saudi Arabia.

This year I was one of the panelists for the final wrap up, and I mentally did my own review which I presented as a personal precis of the conference.

Spatial Data

One of the themes that came up repeatedly was the importance of data, and especially of accurate, real-time data. Spatial data now includes not just traditional geospatial data sources but other sources such as architectural and engineering design, laser scanning, high resolution photogrammetry with multiple cameras, and ground penetrating radar.  Some of the newer technologies for acquiring spatial date were the subject of several talks including laser-scanning and LiDAR, high resolution photogrammetry, and earth observation satellites.  Dominic McPolin of the Central Planning Unit (CPU) in Bahrain and others emphasized the importance of reliable data in infrastructure planning.

Intelligent Models

Increasingly people want to do more than simply prepare maps.  They want to analyze and and simulate which requires more intelligent data.  For years utilities and telecommunications companies have maintained information about their outside facilities including location, properties such as maintenance records, and connectivity, because outage management, asset management and other utility applications require this information.  For example, outage management (OMS) is the primary user of connectivity because OMS typically needs to perform traces to identify controlling devices which may be the source of network traces and to identify customers affected by the failure of a device such as a transformer or pump. The increasing adoption of model-based design for structures such as buildings (building information modeling or BIM), roads and highways, electric power substations, nuclear power stations, hydroelectric plants, and renewable energy generation facilities, means that intelligent models are becoming available for many more types of infrastructure. Government permitting organizations are seeing the advantage of model-based design and are mandating (US GSA) or recommending (Singapore BCA) BIM to support building permit submissions.

Convergence

Another common theme is the convergence of architectural and engineering design, traditional GIS, and 3D technologies which enable what Dominic McPolin called a more holistic view of the built world.  World trends such as population growth, global climate change, and increasing complexity of our world are seen as major drivers motivating the broader interest in convergence.  Dominic McPolin emphasized that the complexity of modern economies requires a new fusion that will allow us to respond effectively to global climate change and other world challenges and predicted that because of its small size next year Bahrain will be a world leader in developing a new fusion of information and systems or conceptual business architecture that will provide a more holistic view of the national economy.

Collaboration

An important theme throughout the conference was the importance of collaboration between different disciplines such as engineering and business, different levels of government, different government ministries and agencies, and government and the private sector.  IT in many organizations is characterized by stovepipes, or islands of information, with limited communications between the different IT systems such as ERP, CRM, GIS, engineering design and others.  For example, a large utility or telecommunications firm may have thousands of applications, each running on one of ten or more operating systems, with its own proprietary database, which may be one of nine or more independent databases containing the same information.  This problem is exacerbated in government by multiple levels of government, national, state or provincial, and municipal.  In this respect small, centralized nations such as Singapore and Bahrain often have a significant advantage, because the problems associated with multiple levels of government are minimized.

Dr Al-Tamimi outlined a vendor-neutral architecture for the integration of different spatial systems within government including infrastructure, engineering, building permitting, mapping, and environmental, that is designed to be implemented in Bahrain and Saudi Arabia.  A key feature of the architecture Dr Al-Tamimi is developing is that it includes the full range of spatial information from the very detailed, engineering precision, large scale to very small scale.  He said that integrating model-based design or BIM is a priority and planned as the next step. 

Ron Lake, who has developed and has been proselytizing the concept of the GeoWeb or worldwide, web-based  integration of spatial information for several years, made what I think is a key point that the GeoWeb needs to be treated as a web of systems, rather than a web of documents.  Ron, who is the founder of the GeoWeb conferences, has been a proponent for the integration of model-based design and BIM, which was a central theme of the last GeoWeb in Vancouver, as a key component of the GeoWeb.

New Software Business Models

Several of the presentations at the conference either focussed on open source geospatial software or described projects that used open source software.  Daniel Ames, from Idaho State University, gave what I suspect was an eye-opening presentation for many in the audience, outlining some of the advantages of open source software.  He referred several times to the Open Source Geospatial Foundation (OSGEO) and very generously mentioned Autodesk's initial and continued support for the OSGEO.  Dan is the founder of  the MapWindow open source project, which has about 250 participants worldwide including 20 committers (developers with permission to contribute and modify code.) 

Innovation

Bahrain has announced that it intends to reduce CO2 emissions per capita by 60% by 2014. This will require the biggest infrastructure program (road, water, electricity) program the country has ever seen.  Dominic McPolin was unequivocal in identifying the key to achieving this aggressive goal as innovation and he made an impassioned plea to the private sector to provide the technology to enable this goal to be realized.

MEST 2009: Locating Underground Infrastructure in Bahrain

Today at MEST 2009, I had the opportunity to see a system for maintaining a single database containing all underground infrastructure in Bahrain.  A single repository for all underground facilities is something that is only possible in a very few places in the world such as Tokyo and other Japanese cities, Sarajevo, Calgary, and Edmonton, which are the ones I am aware of.  The system I saw is the Intelligent Decision Support System (iDSS) of the Ministry of Works and it was shown to me by Mr. Abbas Ally, Head of Central Planning Engineering in the Central Planning Office (CPO).

The Bahrain underground infrastructure system is unique for several reasons.

One Database of All Underground Infrastructure

According to Mr. Abbas the location of all underground infrastructure in Bahrain is stored in a single Oracle Spatial RDBMS.

• electricity including transmission, distribution, and street lighting,
• water including transmission and distribution,
• wastewater including storm, road, sanitary, and combined
• telecommunications

Stewardship

The source databases are maintained by the respective owners, water and electricity by the Water and Electricity Authority, telecommunications by Batelco, and wastewater by the Ministry of Works.  iDSS has several layers of security, using OS and Oracle security, that determines who can see what, and who can update what.

Frequent Updates

The intention is for each operational database to be replicated to the iDSS database, which means that the iDSS database will be as up-to-date as the source databases are, though this capability does not appear to be functional at the present time.  In the short time that I have been here I haven't been able to determine how reliable the individual source databases, electricity, water, wastewater, and telecommunications, are.

Excavation Requests

Anyone proposing to add to or make a change to undergound infrastructure is required to complete a Proposal Request, essentially a building permit.  The request is forwarded electronically to all of the participating utilities, who are required to review and respond to the request within three days.  Utilities who don't respond within three days are assumed to have approved the request.  In 2008 over 7,000 requests were processed.

Voluntary

Accordng to Mr. Abbas, participation in iDSS is voluntary, but all utilities and Batelco are participating.  Mr. Abbas explained that the primary reason that all the utility and telecommunications companies have agreed to participate is the business benefits they see resulting from participation.  Given the amount of money utilities and telcos in North America spend on locating underground facilities in response to Call-Before-You-Dig and One-Call centers requests, I expect the business benefits to the individual utilities are considerable.

Latest Release of the FDO Toolbox v0.8.8 Available

The latest release of the FDO Toolbox is now available.

FOSS4G: Open Source GeoREST Project Announced

I have blogged on previous occasions about the RESTful API for accessing raw geospatial data which Haris Kurtagic developed and which Jason Birch has used to provide open access to geospatial data at the City of Nanaimo, BC.  At FOSS4G I gave a presentation on behalf of Haris and Jason, who couldn't make it to Sydney, which announced the creation of the open source GeoREST project (www.geoREST.org).

Open Access to Raw Geospatial Data

The principle of free and open access to government geospatial data has been adopted by many governments including US Federal, Canada, Japan, South Africa, Brazil, California, and most recently by the City of Vancouver.  The benefits of free and open access to government geospatial data as identified by Senator Kate Lundy are

• Citizen engagement
• Transparency and accountability
• Economic development

But being able to access prepared maps and other derived material is just a first step, governments need to provide access to raw geospatial data in commonly used Web-friendly formats. Tim Berners-Lee, in Government Data Design Issues, is a strong advocate of making raw data accessible over the web including supporting standard Web methods, most critically, searching.  The thesis on which GeoREST is based is that standard Web protocols, HTTP and MIME, are the simplest way of providing searchable raw geospatial data on the web.

REST

Representational State Transfer (REST) is an architecture style of networked systems which relies on the Web protocols HTTP and MIME. The best example of a RESTful implementation is the Web itself.  REST uses URIs, HTTP methods (GET, POST, PUT, DELETE), HTTP response codes, and MIME.

The single major advantage of REST is that because it relies only on standard Web protocols, standard services available on the Web are available "out of the box" including

• Searching
• Passing through firewalls
• Caching
• Scaling/load balancing

GeoREST

GeoREST is a framework for accessing, distributing, and editing raw geospatial data over the web. Because it is based on REST, it supports searching using standard Web search engines such as Google and Bing. 

In its current form it provides feature-based access to FDO spatial data sources including editing. Examples of FDO data sources include PostGIS, SHP, Oracle Spatial, Microsoft SQL Server Spatial, MySQL, SDF, GML, and others.  GeoREST returns geospatial data over the Web in a number of Web friendly formats including KML, GeoRSS, HTML, GeoJSON, XML, PNG, HTML, CSV, and other text-based formats.  

GeoREST relies on open source libraries including CTemplate, POCO, FDO, MapGuide Open Source, and libkml.  GeoREST source code is available at www.geoREST.org.

Example Application Using GeoREST

You can find an example of an application built on GeoREST at the City of Nanaimo Property Search Site. More information about the implementation can be found on Jason Birch's Blog.

If you want to verify that Nanaimo's raw geospatial data is searchable, enter "2323 Rosstown Road, Nanaimo, BC" into Google, Bing or any other search engine, and then click on the first item that gets returned by the search engine.

FOSS4G: Jackie Ng on the FDO Toolbox

Wednesday at FOSS4G Jackie Ng gave a presentation on the FDO Toolbox.  It was presented so well that I was flabbergasted to discover subsequently from one of Jackie's blog that this was his first public presentation.  

There are a couple of important reasons why it is worthwhile talking a look at FDO Toolbox.  First of all, it is built on the .Net platform which differentiates it from the alternative C++ and Java tools.  If you are used to working in a Windows .Net environment, you will feel at home here. Secondly, Jackie has put a lot of effort into the UI so the FDO Toolbox is simple to use. Thirdly, it is a comprehensive schema and data management tool that can handle geospatial and non-geospatial data with equal facility.  I was very impressed with how many things you can do with FDO Toolbox including 

Data Management (ETL)

• Create and edit data models (schemas)
• Read and write geospatial data for up to 150 data sources supported by FDO

Data Preview (maps for geospatial data)

• FDO query
• SQL query

Batch automation

• Batch scripting to automate common geo-processing tasks

Translation (FDO2FDO)

• Bulk copy selected geospatial data from one FDO data source to another
• Fast copy one FDO data source to another, not quite one-click but pretty close

Extensibility

• FDO Toolbox can be extended through extension modules. With extension modules you can add new commands in C# for custom functionality and include new menu entries to expose them in the user interface.

Jackie demonstrated the latest release of the FDO Toolbox 0.8.7 which includes a dramatically improved bulk copy with a completely revamped user interface and XML file definition.  Also FDO Toolbox now supports Python scripting.

Web 2.0, Participation, Google Maps, Quality of Asset Data in Utilities and Smart Grid

If you remember, Eric Raymond in the Cathedral and the Bazaar floated the "many eyeballs" theory that all bugs are shallow if you have a large enough community involved in finding solutions. I also remember that Tim O'Reilly in his seminal paper What is Web 2.0 identified some of the important differences between Web 1.0 and Web2.0, among which a key one in my view was participation. In the early days of the web, we were passive consumers of data.  Web 2.0 brought Wikipedia, where users contributed as well as consumed information.  Like Wikipedia the new paradigm has become so successful that wikis are now ubiquitous and the advantage of  involving users in creating and maintaining data has been demonstrated over and over.  A now classic example is OpenStreetMap. Most recently Google has realized that users are "remarkable data sources themselves", and Google Maps and Google Earth have a new tool that lets you the user report errors in Google's maps. There is a new menu item "Report a Problem" that allows you to suggest edits, like a new highway on-ramp, new names of parks or buildings, and so on. Google promises to vet the edit within a month and if you submit your email address, they'll keep you posted on their progress.

I have blogged about the poor quality of network facility data in utilities and telecom and about how critical up to date reliable network facility data is going to be for the smart grid.  Many years ago I was struck by the simple low tech way Brad Lawrence of ENMAX Power Corp addressed the data quality problem.  He guaranteed field staff a 24 hour turnaround on all updates from the field by instructing records staff to give updates from the field highest priority.  A subsequent audit by an external auditor, in which a field survey sampled a subset of ENMAX's database and compared it to what was actually in the field, reported 99.6% reliability for ENMAX's outside plant database.  For an industry in which 40-70% data accuracy is typical, this struck me as a remarkable achievement. What Brad did was to empower field staff by enabling them to be participants in maintaining asset data. 

Google Maps and Google Earth have provided a simple online tool that can be used by the user community (typically field staff in the case of utilities) to report errors and changes in network facility data.  In the case of utilities that provide open access to assset data like North Shore City in New Zealand, not only field staff, but citizens could become active participants in maintaining utility asset data.  Given the long history of poor quality asset data in utilities, it seems to me that a radically new approach along the lines of what Google Maps has done is a promising way to achieve and maintain the level of data quality that we need to make the the promise of the smart grid a reality.

Protected Ecological Areas Map of Canada

The Atlas of Canada is an interesting site supported by Natural Resources Canada that contains a number of maps of Canada including a map of the protected ecological areas of Canada.

Other maps include typical sea ice conditions at the end of the summer melt season, as well as how often sea ice has been present at any location on September 10 over the last 30 years.

DATA.GOV Launches

Data.gov is the Executive Branch data portal.  It provides access to machine readable raw data sets and to applications including spatial. There are 47 raw datasets and 27 applications available now, but you're encouraged to suggest additional datasets and site enhancements to provide "seamless access and use of your Federal data."

Raw datasets

• 1 Global Trade - World Copper Smelters
• 1 National Operational Hydrologic Remote Sensing Center (NOHRSC) 
• 1 Patent Grant Bibliographic Data (2009)  
• 1 Residential Energy Consumption Survey (RECS) Files, All Data, 2005  
• ACRES - Brownfields Properties
• Active Mines and Mineral Plants in the US
• Ages from the National Geochronological Database
• Benefits Data from the Benefits and Earnings Public Use File, 2004  
• Chemical Analyses of Soils and other Surficial Materials of the Coterminous US
• Clean Air Status and Trends Network (CASTNET): Ozone  
• Clean Air Status and Trends Network (CASTNET): Visibility  
• Community Collaborative Rain, Hail and Snow Observations (CoCoRaHS)   
• Current Watches, Warnings, or Advisories for the United States
• Database of Tornado, Large Hail, and Damaging Wind Reports, 1950-2006  
• Earnings Data from the Benefits and Earnings Public-Use File, 2004   
• EPA Geospatial Data Download: Facility and Site Information
• Geochemistry of Sediments from the PLUTO Database 
• Geochemistry of Sediments from the Rock Analysis Storage System (RASS) Database 
• Geochemistry of Sediments in the US from the (NURE-HSSR) Database 
• Geochemistry of Soils from the PLUTO Database  
• Geochemistry of Soils from the Rock Analysis Storage System (RASS) Database 
• Geochemistry of Water Samples in the US from the NURE-HSSR Database   
• Geographical Information System Graphical Database of Tornados 1950-2006   
• Geology of the Coterminous United States
• Interactive Access To National Income and Product Accounts Tables   
• Lower Colorado River Daily Average Water Elevations and Releases   
• Migratory Bird Flyways - Continental United States  
• Mineral Operations of Africa and the Middle East
• Mineral Operations of Latin America and Canada
• Mineral Resource Data System: Conterminous US 
• National Geochemical Survey Database
• Next Generation Radar (NEXRAD) Locations  
• Office of Advocacy's News Update File    
• Past Atlantic Storm Tracks  
• Past East Pacific Storm Tracks  
• Patent Application Bibliographic Data (2009)
• Residential Energy Consumption Survey (RECS) Files, Energy Consumption, 2005  
• Resource Data File - Alaska (ARDF)
• Severe Weather Data (SVRGIS) GIS Data - County Warning Area
• Storm Prediction Report  
• Tornado Tracks and Icons, 1950-2006
• Worldwide M1+ Earthquakes, Past 7 Days  
• Worldwide M1+ Earthquakes, Past Day  
• Worldwide M1+ Earthquakes, Past Hour  
• Worldwide M2.5+ Earthquakes, Past 7 Days   
• Worldwide M2.5+ Earthquakes, Past Day
• Worldwide M5+ Earthquakes, Past 7 Days

  Tools

• 1 Federal Bureau of Investigation Widget
• 1 HHS 2009 Flu Info
• 1 Medical Expenditure Panel Survey MEPSnet Insurance Component (MEPSnet/IC)
• 1 USA Spending Contracts and Purchases
• 1 USA Spending Grants and Loans
• Airline On-Time Performance and Causes of Flight Delays
• American FactFinder
• Cancer Incidence - Surveillance, Epidemiology, and End Results (SEER) Registries Limited-Use
• CDC Emergency Text Messages
• DataFerret
• Emergency Preparedness and Response Widgets
• FDA Peanut-Containing Product Recall
• Federal Hurricane Response Widget
• Federal Register
• FluView National Flu Activity Map
• Food Availability (Per Capita) Data System
• Geospatial One-Stop
• National Assessment of Educational Progress, NAEP 
• National Postsecondary Student Aid Study (NPSAS) 
• NOAA Watch Web Widgets
• North American TransBorder Freight Monthly and Annual Summary Data
• SESTAT
• The National Atlas
• The National Map
• VitalStats
• Web-based Injury Statistics Query and Reporting System (WISQARS)
• Wide-ranging Online Data for Epidemiologic Research (WONDER)

Oracle and Open Source

I've been at the first day of Oracle Open World in San Francisco.  This conference is huge, I have heard estimates of 47,000 visitors.  A major street in the middle of San Francisco has been closed to accommodate this volume of people.

Conferences like this and Autodesk University (Nov 28 - Dec 1) are valuable because they bring together a lot of people, many of whom are technical (at Autodesk University most of the participants are technical), and you can pick up some very interesting information and meet people you might not have run into otherwise.

There were two very interesting sessions I found some time to attend, the first was  a panel of Oracle folks (Ken Jacobs, Chris Jones, Mike Olson, and Omar Tazi) discussing Oracle and open source, the second a presentation by Noel Yuhanna of Forrester on the future of databases.

Oracle and Open Source

This was a very interesting discussion.  First of all I was surprised to see Ken Jacobs on this panel.  I have been acquainted with Ken for many years, starting back in the days when Oracle began looking at long transactions and versioning (now encapsulated in Workspace Manager).  He has been at Oracle for a long time and has always been on the forefront of technology.  When you see Ken involved with something you know immediately it is new and it will become a big thing at Oracle.  Ken's new role involves open source and in particular he is responsible for the Innodb team in Helsinki, which Oracle acquired in the past year.  For those of you not familiar with Innodb, it is open source and was the default data store for MySQL for several years.  Also on the panel was Mike Olson, who used to be the CEO of Sleepycat which was responsible for Berkeley DB, prior to being acquired by Oracle.   Mike is now responsible for embedded databases at Oracle.  Berkeley DB was and is open source and is estimated to be running at 200 million sites around the world (considerably increasing Oracle's installed base I would add.)  The other panel members were Chris Jones, who is responsible for Oracle's PHP support, and Omar Tazi, who is Oracle's chief open source evangelist.

To me, and I think to the audience, the message conveyed by this panel is that Oracle has supported open source for many years starting with Apache and Linux, that Oracle is not "religious" about open source, that its primary business remains closed source, but that Oracle sees a market need and opportunity in the open source arena.  It sees advantages in having open source offerings for both market expansion and for revenue generation.  Based on this panel I would say that Oracle is serious about open source technology, the open source market, and the open source business model and doesn't intend, for example, to take BerekelyDB proprietary.  Oracle seems to be continuing with the same dual license open source business model that Sleepycat used for 10 years.  Also Oracle has released two new versions of Berkeley DB since acquiring Sleepycat.   One can only speculate about Oracle, MySQL and PostgreSQL, but I suspect that Oracle sees MySQL has the major competitor in the open source market and I expect something interesting will evolve with InnoDB.

An interesting factoid that indicates the business opportunity that Oracle sees in the open source market is that the total revenue from Linux-based RDBMs in 2005 was $1.5 billion, of which Oracle's market share was 80.6%.  Oracle RDBMS is a proprietary product which it is selling successfully into the  Linux market.  Clearly Oracle sees significant revenue opportuntities in the open source marke. 

Oracle is also investing in open source.  In addition to acquiring InnoDB and Sleepycat, Oracle contributed its cluster file system to the Linux effort, and has contributed to Apache, open source testing, and other initiatives.

Managing DWG spatial data using Oracle Locator

While I was in Brazil I had the opportunity to meet with Carlos Diniz and Fabio Gomes from DigiCADD. DigiCADD has been working with a large telephone company in Brazil, who have been using AutoCAD for ten years, have a large number of DWG files,  and who want to make this large volume of data available to a wide audience.  To do this they plan to improve the manageability of their data, streamline data maintenance, and make their data available to their field staff by making this data accessible in real-time using a spatially-enabled relational database management system (RDBMS).  Using Autodesk Map, DigiCADD has loaded DWG files into Oracle Locator 9.2 and at this point have about 0.7M polygonal, 10M linear, and 21M point objects (total of about 32M spatial objects) stored in Oracle Locator. 

I saw DigiCADD's MapGuide application for publishng this data to about 1000 web users and was impressed both with what DigiCADD has built and the scalability that a spatially-enabled RDBMS like Oracle Locator together with MapGuide provides.  DigiCADD intends to completely eliminate the need for maintaining and managing DWG files by providing real-time update capability for all spatial data in Oracle Locator using Autodesk Map3D and MapGuide Enterprise.  They also plan to add imagery support, integrate their spatial data with other enterprise systems, and provide access to the data to field staff using mobile devices.