Between the Poles

INSPIRE and Free Data

Last week, the European parliament and council of ministers worked out a compromise to the INSPIRE directive whose purpose is to harmonise spatial information in Europe. The directive had become the focus of an intense discussion about making public sector data freely available. It has been reported that the European Parliament supported mandating free data and the Council of Ministers was opposed.  The compromise that was worked out appears to allow governments to charge for data, unlike the US and South Africa, for example, but with some restrictions, for example, arbitrary pricing.

This implies that the Ordnance Survey is probably not going to go out of business, as a result of INSPIRE Ed Parsons CTO of OS .

MapGuide Open Source and Community Participation

Most of this week I am at Autodesk University in Las Vegas.  I don't know the numbers, but last year there were about 5,000 participants, if I remember correctly, and this year there appears to me to be more.  Participants are primarily technical, so most of the sessions are technical.

Yesterday was focussed primarily on user groups, and among these was the MapGuide User Reception, which I attended.  The most remarkable thing in my mind was that this meeting was unlike any other event of a similar type that I have attended at AU in the past, and I think the reason was that the focus was on open source, specifically MapGuide Open Source.

Bob Bray, who is the architect of MapGuide, gave a presentation on MapGuide Open Source.   First of all he outlined some pretty remarkable current statistics.  To date there have been 23,000 downloads, one third of them Linux interestingly.  Surprisingly to most of us I think the monthly rate of downloads continues to climb.  The Feature Data Object (FDO) API, which is a separate open source project, has had 3,000 downloads.  Most interesting to me is the number of new FDO Data Providers, being developed by the community, including PostGIS, Oracle Locator/Spatial, GDAL, OGR, and FME2007.

But the most interesting thing to me and to others who have been at AU in the past is when Bob oulined the process for defining the feature set for future versions of MapGuide Open Source.  This is unlike anything that people with closed source experience with product management would be familiar with.  I've included Bob's slide, which outlines the Request for Comment (RFC) process, which I expect that anyone with Apache experience would recognize,  but that closed source product managers would definitely not.  The voting uses the Apache system where +1 means in favour, 0 means abstention, and -1 is a veto. 

Why the final sentence why am I telling you this ?  First of all you need to understand what the Project Steering Committee (PSC) is a group of community participants, currently seven people I believe, most of them not Autodesk and including only developers and users, no product managers.  Secondly, this is an open process conducted via email.  Thirdly, because it is open anyone in the community can submit the features that he or she thinks is critical.  This effectively gives power to add or modify features to the community.  It also means that new features can be implemented much more rapidly.  The RFC process in my mind is one of the most important advantages of open source software development, because if someone in the community needs a new feature this process empowers them to make it happen.  To me this is something that has enormous implications that it will take people with solely closed source experience some time to get their head around. 

Andy Morsell, an independent software developer, rounded off the presentations appropriately by describing his experience in progressing from a pure closed source background to his current perspective where he uses both models in his business activities.  Exciting times !

Map Africa and Open Source Geospatial

I've just spent a few days at Map Africa and this has been a very interesting conference.  First of all it has not been a traditional GIS conference.  There has been a strong focus on capacity building, specifically in Africa as you would expect.   I found two particularly interesting things about this conference.

Open Source Geospatial

At the end of the first day there was a panel discussion on Interoperability and Open Source.  The panelists were Andrew Terhorst from the Meraka Institute in South Africa, Professor Dasgupta from India, and myself.  Andrew provided a uniquely African perspective on open source.   He was vocifereous in arguing that open source is good for South Africa becaue open source creates an opportunity for the development of human capital in South Africa.  He also reiterated what I think is generally recognized that security is not a reason to not use OSS, but he did allow that OSS needs to improve its usabality and that this is the primary barrier to its adoption. He encouraged people to be more adventurous and to use open source because it's a way to help Africa to develop its human capital.  As he pointed out himself, he has an academic perspective and he needs to be able to see code to undertand the algorithms used.  For example, he likes GRASS because he can see how things work. In this context I found it interesting that he sees open source as not only providing a way for people to collaborate but as a new form of peer review.

My perspective is that both open source and closed source have necessary roles in the software industry.  For example, I believe that the state of web mapping servers is analogous to web (HTTP) servers in the late 90's, and that this area of the geospatial industry is ripe for commoditization and open source.  Desktop geospatial applications is another matter - there is still lots of room for closed source vendors to differentiate themselves in this software segment.

I detected a serious interest in open source geospatial among the South African participants at this conference.  What was most interesting to me from what I heard from a number of people with whom I chatted at the conference is that in South Africa cost is one of the most important reasons motivating this interest.  There is what appears to be a widespread perception in South Africa that traditonal GIS is too expensive.  Part of this is no doubt due to the fact that in the developing world, if you express the cost of software in terms of the local per capita income, the cost of software is much higher than in North America.   For example, I remember seeing some time ago that in Bangladesh Windows98 cost the equivalent of three years' income for the average Bangladeshi.  But South Africa is quite different from Bangladesh.  For example, Gauteng province, which includes Johannesburg and Pretoria, represents 37% of the African economy.  So I think this is only part of the reason.

Free Digital Spatial Data

One of the remarkable things I discovered about South Africa is that data collected by the national government, mainly by the Department of Land Affairs: Surveys and mapping, is free in digital form.  This is remarkable.  The country where spatial data was first made available at no cost was the US where the Federal Government's guiding principle is that if taxpayers paid to collect data, the government can't charge them again.  Many people believe that one of the reasons that the US is such a major force in the geospatial industry, both in the areas of software applications and data providers, is because of the availablility of low cost data.  I think this means that great things are in store for the South African geospatial industry.

Improving the Information Flow at the St Johns River Water Management District

At Oracle Open World I had a chance to chat with Peggy White, CIO of the St Johns River Water Management District, and Jim Ren, who I've known for many years and who also is in IT at the SJRWMD, in northern Florida.  They described a very simple but effective way of improving the flow of information between land developers and the SJRWMD. 

Drainage Engineering Submittals: Paper-based Business Process

A land developer who plans to develop a plot of land larger than a certain size, 9000 sq ft I think Peggy said, are required to submit engineering drawings showing drainage plans.  Most of these engineering drawings are created using AutoCAD.  In the past the process was paper-based.  The developer printed or plotted the drawings and delivered them to the SJRWMD office.   An SJRWMD engineer  reviewed the engineering drawings, prepared a formal response outlining any changes in the design that may be necessary in the form of a letter which was sent to the developer.  The developer then made the changes and the process was repeated.  As you can imagine, this was a time consuming process.

Electronic Engineering Submittals

Peggy and her team have made a very simple change which is having a significant impact on the time it takes for the submittal and review process and which has efficiency benefits for both developers and the SJRWMD.  Applicants can now submit drawings electronically in the form of DWF files.  DWF is a light weight web format that is 1/10 or less the size of a DWG or DGN drawing file, can be sent across the web on a dial-up line, and can be generated by AutoCAD out of the box or by any Windows-based application with a free download.  The SJRWMD maintains a portal SJRWMD E-permitting where the DWF files can be uploaded.  SJRWMD engineers retrieve the DWF files from the portal server, and can review and redline the DWF sheets electronically using a desktop tool (Autodesk Design Review.)  The engineer prepares a letter response for the developer, and if modifications are required, the developer can make changes to the engineering design and resubmits a new DWF file.

Business Benefits

What impressed me about this it is a very simple and effective solution to an information flow problem.  It didn't require an expensive investment in new technology by either the SJRWMD or the developers, but it significantly improves the flow of information, is more efficient, and speeds up the business process of getting drainage submittals approved.

Engineering GIS and APWA

Last week I attended a fascinating conference in Los Angeles, the annual APWA (American Public Works Association) Southern California Conference.  Most of the people attending were water and waste water management professionals, primarily from Los Angeles and Southern California. 

Aging Field Force

I gave a presentation on open source geospatial and its impact on infrastructure management with a particular focus on the impact on field force automation.  My take was that Southern California public works departments are experiencing the same fundamental problem that utilities and telcos across North America are, an aging work force and the lack of an effective business process for capturing the knowledge about the network infrastructure that is currently in the heads of experienced (and soon to retire) field staff.   For example, I chatted with an employee of an Arizona utility recently, who told me that in his organization 50% of the work force is eligible for retirement this year.  To me the problem can be addressed by enfranchising the field force, making it possible for them to feedback information from the field to a central data repository.  The process most organizations feel most comfortable with is redline, which is a two step process, where field staff annotate drawings and Records staff are responsible for updating the central repository.  But increasingly, with the advent of ubiquitous WiFi, 2.5G/3G telephony and handhelds, organizations are seriously contemplating providing direct update access to the central data repository to their field staff.

CAD/GIS Integration

One of the talks I attended was by Diane Ray of RBF Consulting on "The interoperability of GIS and AutoCAD Map."  Diane talked about another common problem, making the same data available to engineers and GIS professionals, and how RBF addressed this problem for the Beamont Cherry Valley Water District.  Beaumont Cherry is interesting in that it requires that contractors provide submittals documents in both CAD and GIS formats.  For contractors who are unable to do the conversion themselves, BCVWD charges about $275 per lot to convert a CAD drawing to GIS format.  The conversion process is not a simple format conversion process, but involves exploding blocks, ensuring topological consistency such as ensuring nodes and line endpoints are snapped, converting lines to polylines, and making attribute and other property data available in MS Access files.  None of this is really that complex and in fact CAD operators can use many of the features already available in modern CAD tools, such as classification in AutoCAD and Autodesk Map, to address these issues during the engineering design phase.  Some of the problems require in addition fairly simple application development to enforce.  But in my opinion the best way to deal with most of these issues are through using a rule-based constraint engine such is found in Autodesk Topobase to enforce attribute, relationship and other constraints.

Open Source and GIS

In the last month or so there has been some interesting developments with respect to open source and GIS.  I'm including a link to James Fee's blog James Fee on GIS and Open Source where he discusses what he sees as a developing trend in the GIS market.

Also interesting is that another major GIS vendor has decided to support open source 52north .

Spatial Data and Earth Observation Satellites

I've been at the Space Technology and Geo-Informatics 2006 conference and there have been a number of presentations about new earth observation satellites.  I've seen presentations from Japan, Thailand, Canada and MDA, and Digital Earth.  Japan is planning a new satellite ALOS. Thailand is planning THEOS, Canada and MDA are planning Radarsat II, and Digital Earth is planning Worldview I.  All of these satellites appear to be scheduled for launch in 2007.  ALOS, THEOS, and Worldview I appear to be equipped with panchromatic telescopes and multi-spectral cameras and are targeting  a highest resolution of 1-3m.  Radarsat II is different technology, radar, which has a tremendous advantage in that it is not affected by cloud cover.   Radarsat II is targeting 1-3 m as their best resolution.  The last estimate I heard at a previous conference is that we were generating a terabyte of spatial data a day, not all of which is available to civilians, of course, but with these new satellites which seem for the most part to be gathering data for civilian use, beginning in 2007 it appears that there will be an increasing volume of spatial data available.  In fact, 2007 is looking very competitive for spatial data, which should bode well for the availability of data at reasonable prices for users spatial data including on-line services such as Google Earth and Virtual Earth.

Open Source and Microsoft

A headline on CNET this AM begins "When hell freezes over...".  What the headline was referring to was the agreement announced by Novell and Microsoft to collaborate to help Microsoft's Windows to work with Novell's Suse Linux, which is open-source.  This is a tremendous milestone in the acceptance and legitimization of open source as a software development methodology.

Microsoft and Novell are reported to have reached an agreement on patents that will provide each other's customers with patent coverage.  This is interesting, but even more interesting is Microsoft's agreeing to not enforce its patents against "individual, noncommercial Linux developers."  At this point Novell claims to be the only company that provide its customers with a patent agreement from Microsoft.

And it doesn't stop here, Novell and Microsoft also agreed to create a joint research lab at which they will build focus on virtualization, Web services for server management, and Microsoft Office-OpenOffice.org compatibility.

Together with Oracle's announcement about offering support for Red Hat Linux, to me this agreement means that the world is evolving very rapidly and open source seems to be in the midst of it.  I can't help but believe this agreement is good for customers, because it means that open source and proprietary operating systems and applications can be expected to work together better as a result.

Future of Database

One of the most interesting talks I went to at Oracle Open World last week was a talk by Noel Yuhanna of Forrester on The Future of Database Technology: An Analyst's View.

Features and Technologies

First of all the features and technologies that he expects to become the focus of database vendors in the immediate future are open source, consolidation, in-memory databases (caches), very large databases (apparently the largest one at the present is 50 TB), grid architecture, unstructured data, higher performance, increased availability, database security, and archiving and retention.

DBAs

An interesting fact is that apparently there are 265,000 DBA's around the world.  An even more interesting statistic is that each DBA manages on average a terabyte of data.  I validated this statistic myself chatting to a couple of folks who turned out to be DBAs.  For example, I chatted with a very interesting person from the Liquor Licensing Board (LLB) of British Columbia, and he said that the LLB managed about 2.5 TB of data, and employed two senior DBAs and a junior consultant DBA.  This is pretty close to the 2.5 DBAs you would expect for 2.5 TBs of data.

Over the next few years Noel expects that DBA's will spend a decreasing amount of time on tuning and performance, availability and disaster recovery, because these areas are being increasingly automated.  RDBMSs are becoming increasingly adaptive, so they they can diagnose and solve their own problems and tune themselves.   This means that instead of spending their time on administration and tuning they will spend an increasing proportion of their time on security, performance, and functionality. 

By 2012 DBAs will be spending a lot of their time on architecture and implementation to support database virtualization.

Database Virtualization

Noel outlined a very general architecture for database virtualization that includes a cache layer (for performance), a database management layer (for backup/recovery, transaction management, and other administration tasks), and the data layer itself.  To me this is a very general data management architecture that has been proven over and over again for OLTP  (online transaction processing) applications and it appears to be evolving into the de facto architecture for database virtualization.