Between the Poles

Canada's open geospatial data initiative

At the Geospatial World Forum in Geneva, as part of a ministerial panel discussion on government's role in evolving geospatial policy for national and regional development, Prashant Shukle, Director General of the Canada Centre for Mapping and Earth Observation gave an overview of the Canadian Government's plan for open access to federal government geospatial data.

Open data

In July, 2012 Tony Clement, Treasury Board President said that "Data is Canada’s new natural resource…”  It is well-known that spatial information contributes significantly to the Canadian economy, in the range of billions of dollars.   The Canadian Economic Value Study and Sector Scan, which is currently underway and will be completed this Fall, will put a more precise dollar figure on the value of spatial data and technology to the Canadian economy.

But the government recognizes that the true value of geospatial data to the Canadian economy will only be recognized by its liberation and use. Canada has taken a number of steps as part of the Open Government Action Plan 2.0.  It has developed an open government license which removes restrictions on the reuse of all types of data published by the Government of Canada and aligns Canadian policy with international best practices to promote the re-use of Federal data as widely as possible.  It has created an open data portal (data.gc.ca) which houses 191,608 datasets of which 96.5% are geospatial datasets.  In addition it is working toward a pan-Canadian Open Data Strategy.

In June 2013 Prime Minister Steven Harper signed  the G8 Open Data Charter, a set of norms and standards for the proactive release of high-quality data that is usable by all and is unrestricted in its use and re-use.  All of this reflects a data policy where government data is published openly by default, not as the exception.

Sharing data within government 

The new open data policy is helping the Canadian Government to transform itself.  The Government's Blueprint 2020 aims at encouraging open and networked, engaged citizens through policies and standards including standards for geospatial data, a standard for metadata, and shared services.

Within the federal govenment there is a growing need for geospatial data.  In a FCGEO survey of federal departments in May 2012 94% of respondents reported increasing demand for and use of geospatially-related data to meet business requirements.  21 federal departments rely on geospatial data related to weather, resource development, endangered species, national parks, national defense, census data, public health and oceans management, and other topics.  The challenge is to enable easy discovery and access to all federal government data holdings. 

In 2012 the Government of Canada created the Federal Committee on Geomatics and Earth Observation (FCGEO) to work toward a whole of government, enterprise solution for geospatial and Earth observation data.

One of the FCGEO's important initiatives is the Federal Geospatial Platform (FGP), which is intended to be a common “platform” of technical infrastructure, policies, standards and governance that will open up  the government’s massive storehouse of geodata and imagery for new and innovative uses. This integrated national system is intended to reduce unnecessary duplication, fragmentation and overlap of geospatial services.  Currently it can take weeks to months to find the data owner, acquire and manipulate the data, conduct the analysis and generate reports.   The FGP is intended to enable users to “search once and find everything” within the federal government’s geospatial data and information holdings and reduce the time required to collect data and generate reports to hours.  In addition the FGP’s visualization tools and analytics are intended to enable the public, private and academic sectors and individual Canadians to create new value from government data sources.

Korea's BIM on GIS project is developing a platform for spatial interoperability

At the Geospatial World Forum in Geneva, several geospatial-related projects from South Korea are being highlighted.  One of the most important is a five-year project conceived, sponsored and funded by the Korean Institute of Construction Technology (KICT).  The objective of the project, called BIM on GIS Interoperability Open-Platform, is to develop a software platform that enables GIS and BIM interoperability.  The project is being carried out by Korean software company Gaia3D.

Gaia3D has been prototyping a software architecture comprised of application plug-ins, desktop and web clients, 3D BIM/GIS object server, spatial RDBMS, and spatial data including BIM and GIS models.  The main functions that the software platform supports are BIM format conversion (rvt to internal) for improved performance, simultaneous spatial query and analysis for geospatial and BIM data, application APIs for facility management (FM) and energy management, interactive navigation inside and outside of buildings, control over transparency (making walls transparent, for example), and 3D visualization.  It supports large data volumes including BIM, digital terrain models, orthoimagery, photorealistic 3D visualization, textures, augmented reality, both BIM and GIS models and international standards IFC and CityGML.  The application is intended to be vendor neutral with respect to both BIM and GIS vendors.

The current prototype relies on open source geospatial software including GeoServer, World Wind SDK for Java, OpenLayers, and PostGIS.

An example that was demonstrated to me at Geospatial World Forum by Heegu Park of Gaia3D was KOPRI (Korea Polar Research Institute), who maintain a research station called King Sejong in Antarctica.  BIM models were developed for the 35 buildings of the research station and integrated with geospatial data including a digital terrain model and satellite imagery.  Navigation inside and outside of buildings was demonstrated, looking outide through windows, as well as removal of walls and rooves to visualize inside infrastructure.

Crowdsourcing geospatial data for authoritative government databases and NSDI

Since Google Earth's release in June 2005, national mapping agencies (NMCAs) and other government organizations involved with geospatial data and technology have been reassessing their role.  At the Geospatial World Forum (GWF) in Amsterdam two years ago, Paul Cheung of the United Nations Initiative on Global Geospatial Information Management (GGIM) voiced the concern about their role that many national mapping agencies and other government organizations with responsibility for geospatial information have had since the advent of Google Earth, private data companies like Digital Globe, crowd-sourced geospatial data like OpenStreetMap, and open data policies adopted by many governments around the world.

The challenge for NMCAs is that in parts of the world, Google. Bing, OpenStreetMaps, or a local crowd-sourced alternative like Malsingmaps provides more comprehensive, accurate, current and accessible data with less restrictive licensing and at a lower cost than the NMCA.  A report from the GGIM argues that "governments have a key role to play in bringing all actors together to ensure that our future society is a sustainable, location-enabled one, underpinned by the sustainable provision and effective management of reliable and trusted geospatial information."  In other words governments and NMCAs have a unique critical role in developing and maintaining authoritative national databases.   To improve the quality and timeliness of these authoritative databases many have argued that NMCAs need to find ways to incorporate volunteered or crowdsourced geospatial data in these databases.

In the United States the National Map is a significant contribution to the National Spatial Data Infrastructure (NSDI).  It is a collaborative effort among the US Geological Survey (USGS) and other Federal, State, and local partners to deliver topographic information for purposes that range from recreation to scientific analysis to emergency response. The geographic information available from The National Map includes orthoimagery, elevation, geographic names, hydrography, boundaries, transportation, structures, and land cover.

In addition to being an important contribution to the NSDI, the National Map is a foundation for the Department of the Interior (DOI) Geospatial Modernization Blueprint and its mission to protect America's treasures for future generations, provide access to America's natural and cultural heritage, offer recreation opportunities, honor trust responsibilities, conduct scientific research, provide wise stewardship of energy and mineral resources, foster sound use of land and water resources, and conserve and protect fish and wildlife.

The USGS's National Map Corps (TNMCorps) was initated a year ago as a nationwide program with the goal of using volunteered or crowdsourced data for the National Map.   TNMCorps' Volunteered Geographic Information (VGI) project engages citizen scientists to collect manmade structures data including: schools, hospitals, post offices, police stations and other important public buildings. In the past year, civilian volunteers in 50 U.S. states have begun providing accurate geospatial data to the The National Map,

• 1,422 – volunteers
• 50,696 - total number of edits
• 50  - U.S. states

According to the Director of the National Geospatial Technical Operations Center in the U.S."This project has proven that we can count on volunteers to provide quality information to be included in authoritative government databases".

If you are interested in becoming a volunteer for TNMCorps, go to the National Map Corps project site to sign up.

Thanks to Dave Smith for pointing me to this.

India Geospatial Forum: Implementing GIS at a privatized Indian power utility

At the India Geospatial Forum in Hyderabad, I moderated a session on electric power.  It turned out ot be an absolutely fascinating conversation with a wide range of speakers representing different aspects of the Indian power industry.

Arup Ghosh, Chief Technology Officer at Tata Power Delhi Distribution Ltd presented an insightful view into implemnenting GIS from the prespective of a private utility.  Only 5% of India's power industry is private, but the private sector seems to be leading a transformation of the Indian power industry in a number of areas.

Mr Ghosh pointed out that a major challenge for the electric power industry in India is that distribution is chaotic and this is primarily because of financing.  The numbers that Mr Ghosh intimated convey just how big a problem this is.  Distrubution companies owe financial institutions Rs 200,000 crores ($33 billlion).  Consumers owe the power distribution companies Rs 60,000 crores ($10 billion).   According to official statistics aggregate technical and commercial (AT&C) losses (non-revenue generating) amounts to 30% of total power generated, but for some parts of the country the rate of AT&C losses is much higher.  For example, when TPDDL was privatized, its AT&C rate was 52%. oVer half of its power was non-revenue generating.

GIS at TPDDL                                                       

Mr. Ghosh offered some very interesting statistics and lessons learned from TPDDL's implementation of GIS. In 2006 TPDDL invested Rs 12 crores ($2 million) in implementing a GIS.  Annual maintenance and operations for the GIS cost Rs 1.5 crores ($250,000).  The estimated annual rate of return on the investment is Rs 4 crores ($700,000) per year.  It is estimated that the investment in GIS paid for itself in 3.5 to 4 years.

Mr. Ghosh offered some valuable insights based on TPDDL's experience in implementing GIS.  

1. When implementing GIS there need to be a clear roadmap where the business processes that will use GIS are identified.  Mr Gosh was able to provide very specific statistics for TPDDL.  It has 360 business processes, of which 40 leverage GIS. 
2. To optimize the value derived from GIS, it has to be integrated with other enterprise systems.  Mr Ghosh specifically mentioned ERP, CRM, DMS, outage management (OMS), and electric network modeling (SYME).
3. For the long term success of GIS, it important to make sure that a robust schema is developed for  the database in which land base, customer, and facility information is stored. In Mr Ghosh's experience It is very difficult to rework a poorly structured database.

The major implementation challenges that TPDDL experienced were

• Developing the land base - In India utiltiies have to put this together themselves from high resolution satellite imagery,  field surveys,  and a variety of source sof geographic and demographic data sources.  Private utilities experience a challenge in accessing satellite data (from Indian or international sources) because access to satellite data is controlled by the National Remote Sensing Centre (NRSC) and private companies are not permitted direct access to satellite imagery.
• Consumer Data –  One of the major data challenges in India as in other jurisdictions is relating consumers to the power network, linking consumers to a specific phase, transformer, feeder and substation.
• Data is perishable - if a field survey requires 6 months, the data collected can quickly become obsolete.  Business processes need to implemented that are designed to optimize and maintain data quality.
• Finding and recruiting skilled GIS professionals.  The GIS group at TPDDL has about 60 field personnel and 18 analysts and support staff.  None of these has an educational background in GIS.  My understanding was that 12 are electrical engineers and the rest are people with electric power experience.  All have learned GIS "on the fly".  Mr Ghosh said the major problem is that engineering facilities do not include GIS in their curriculum.

Mr Ghosh reiterated several times that this has to be treated as a business process reengineering activity, not just a matter of installing new software and initiating data cleanup and field survey projects.  Maintaining data quality required changing business processes.

• Keeping the land base uptodate – new developments such as subdivisions (estates) and renovations to existing sites are occurring continually and the utility has to implement processes to ensure these changes are captured.
• Ensuring the facilities database was uptodate – new subdivisions and renovations to existing sites means new electric power circuits and equipment.  In addition the power network requires replacement and upgrading (reinforcement).  Business processes that ensure these changes are captured ina  timely fashion in the facilities databae are required.  Many utilities have as-built backlogs that extend to months and even years.
• Updating consumer data – consumers move, they get disconnected and reconnected, and increasingly they may be requesting new serrvices.  a business process that ensures these changes are captures in a timely fashion is required.
• Dynamic revision of consumer indexation – ensuring that each consumer is linked to the corrrect phase, transformer, feeder, and substation is a challenge the world over.

TPDDL realized a number of important benefits as a result of implementing a GIS program.

Asset Management:

Streamlined by integrating GIS and DMS and GIS and SAP.
Reduced redundant data by implementing a single source of truth for asset data.
The asset lifecycle was streamlined by integrating GIS, CYME, SAP and financial asset management.
This helped TPDDL optimize their capital expenditure and investment planning.

Operation Management:

Integrating GIS and DMS  for their 11KV network, and integrating GIS and OMS (outage management) helped them improve outage management.  The network diagram extracted from the GIS was used for operations and streamlining the permit to work (PTW) process.  The asset attributes data stored in the GIS improved maintenance planning and scheduling.
Perhaps the biggest operational benefit is that GIS helped TPDDL understand the causes of their technical losses and plan for mitigation.

Commercial Management:

Being able to exactly geolocate their customers, and link each customer to phase, transformer, feeder, and substation has helped provide better services and their collections efficiency which is now 99.7%.
Their pole inventory has also been implemented in their GIS as are street lights.
GIS has also helped them automate customer connection/disconnection and speed uo revenue generation.  it has also helped with being able to respond more rapidly to power availability requests from high demand consumers.

Based on their GIS and other initiatives, TPDDL has been able to reduce their AT&C loss percentage to the low double digits with the intention to get this down to single digits.  Based on TPDDL's experience implementing their GIS, Mr Ghosh offered some recommendations for regulators and government policy makers.

• Availability of satellite images – Must be made available to private sector.
• Regulators should encourage investment in GIS – Investment linked with clearly defined benefits and transparently calculated IRR.
• Regulators should encourage GIS-based asset management – Planning, creating, maintaining, moving and retiring electricity network assets should be implemented through a GIS.
• Preferential funding for GIS-supported capex schemes – Banks and other funding agencies should recognise GIS-supported capex scheme as superior with better assured returns.
• Training and skill development – this is a major challenge.  Engineering faculties do not include geospatial in theri curriculum and there soes not appear to be a link between the colleges and universities that do include geospatial in their curriculum and utilities.

Spatial interoperability is a model for the benefits of interoperability

In August 2004, the NIST released a study that attempted to identify and estimate the efficiency losses in the U.S. capital facilities industry resulting from inadequate interoperability among computer-aided design, engineering, and software systems.  This study includes design, engineering, facilities management and business processes software systems, and redundant paper records management across all facility life-cycle phases.  For 2002 based on interviews and survey responses $15.8 billion in annual interoperability costs were quantified for the capital facilities industry in 2002.  In addition to the quantified costs, respondents indicated that there are additional significant inefficiency and lost opportunity costs associated with interoperability problems that were beyond the scope of the analysis. Based on this the NIST team concluded that the estimated $15.8 billion cost found by the study was likely to be a conservative estimate.

Arguably the geospatial industry has done more to address the issue of interoperability than any other major sector.  One of reasons is that spatial data has always come from diverse sources.  Even when a dominant player such as an ESRI or Google emerges, it has not been possible for that player to dictate a unified way of achieving interoperability.  Safe Software is a major player in the interoperability industry and since the 1990s has maintained a graph showing the number of file formats that their FME translation engine has supported.  The graph shows that over the years the number of new formats added each year has remained roughly constant with if anything a tendency to increase.  In the future we may be talking more about "formats and systems" because SalesForce / Google Maps Engine / ArcGIS Online are really startpoint and endpoints, but there is no reason not to expect a similar growth pattern for systems.  The rule of thumb is that new devices and new technologies require new file formats and APIs and there seems to be more new things every year.

The way the geospatial sector has addressed interoperability is first of all through standards.  Standards from the Open Geospatial Consortium (OGC) have been widely adopted by governments.  Web services for exchanging spatial data such as WMS (web mapping service), WFS (web feature service), GML (geography markup language), and others are almost universally supported by government mapping agencies. 

The 2005 NASA Geospatial Interoperability Return on Investment Study concluded that projects that adopted and implemented geospatial interoperability standards saved 26.2% compared to projects that relied upon a proprietary standard. In other words for every $4.00 spent on projects based on proprietary platforms, the same value could be achieved with $3.00 if the project were based on open standards.

LandXML, Multispeak, and GeoJSON are examples of standards involving geospatial data that are not associated with a major standards body like the OGC, W3C or IETF, but are non-proprietary open standards that have been widely adopted.

Another vehicle for interoperability are geospatial translation engines like FME that not only provide a tool for converting one file format to another, but also provide a server for publishing data stored in 320-odd formats and systems.  This is a major step forward, because it means that making copies (in another format) of data can be avoided.  As a rule of thumb, once you make a copy of a spatial dataset, you've created a problem. 

And finally there are de facto standards like ESRI shape, Autodesk DXF, and others.  Google's KML (originally Keyhole Markup Language) is a unique example of a de factor standard that became an official OGC standard - through the foresight of Google and the OGC. 

LocationTech attracts open source geospatial projects for managing "big spatial data"

I blogged earlier about Google joining LocationTech as a strategic member (together with IBM, Oracle, Boundless( was OpenGeo), and Actuate). 

After lining up key strategic members, the next step was to attract open source geospatial projects which wanted to take advantage of the governance and intellectual property services including commercially friendly licensing that LocationTech/Eclipse provides.  LocationTech has recently received a number of project proposals that will put LocationTech on the leading edge of open source geospatial development.

SpatialHadoop

Hadoop provides scalable processing for huge datasets by using MapReduce, a programming paradigm for distributed processing, to build an efficient framework for processing large-scale data.  SpatialHadoop is an extension to Hadoop that allows efficient processing of "big spatial data".

GeoMesa

Accumulo is a robust, scalable, high performance data storage and retrieval system based on Google's BigTable design and built on Hadoop.  GeoMesa provides a foundation for storing, querying, and transforming spatio-temporal data in Accumulo.  It implements Geotools interfaces that enable Geoserver and other Geotools related projects to use Accumulo as a data store.

GeoTrellis

GeoTrellis is a general framework for low-latency geospatial data processing developed using Scala and Akka.    GeoTrellis was designed to solve three core problems, initially for raster processing, scalable, high performance geoprocessing web services; distributed geoprocessing services for large data sets; and arallelizing geoprocessing operations to take full advantage of multi-core architectures.

GeoGit

I blogged about this recently because it is a project to develop distributed long transactions for geospatial data. 

GeoGit is a distributed version control system specially designed to handle geospatial data efficiently. It takes inspiration from the source code versioning system Git, but has an approach suited to spatial data (currently Shapefiles, PostGIS, MS SQLServer, or SpatiaLite.) GeoGit efficiently handles very large binary data and is optimized for spatial data using a spatial index.

JTS Topology Suite

The JTS Topology Suite (JTS) is an open source Java software library that provides an object model for planar geometry together with a set of fundamental geometric functions. JTS conforms to the Simple Features Specification for SQL published by the Open GIS Consortium.  JTS provides the core component for vector-based geomatics software or can be used as a general-purpose computational geometry library.

Geoff (Geo Fast Forward)

Geoff is intended to enable existing Eclipse Rich Client Platform (RCP) applications to integrate into the geospatial domain using lightweight components to visualizing geospatial information on a geographical map embedded in the RCP applications.

GeoScript

GeoScript adds spatial capabilities to dynamic scripting languages. GeoScript provides a collection of modules based on the Geotools library for geometry handling, spatial data access, and vector feature rendering.  GeoScript is currently implemented in Groovy, JavaScript, Python, and Scala.

Mixed proprietary and open source as a successful business model for enterprise infrastructure software

A very insightful article about open source business models was posted by Mike Olson, Chief Strategy Officer at Cloudera.  By way of background Mike Olson was involved in Michael Stonebraker's Illustra data base startup, which I remember as the first database that extended DBMS support to spatial data types.  Michael Stonebraker called this type of DBMS, object-relational. I still have Stonebraker's book called Object-relational DBMS, The Next Great Wave published in 1996.  The basis for the project was Postgres developed as an open source project at UC Berkeley under the BSD license.

According to Mike Olson, for Illustra Stonebraker relied on the approach that he had made successful with Ingres in the 1980s, which I also remember.  He created a proprietary fork of Postgres (now known as PostgreSQL) and made Illustra a traditional, closed-source software company to sell it.

In 1998, Mike Olson joined Sleepycat, who made and open source embedded database called Berkeley DB, now owned by Oracle.  Sleepycat made money by "dual licensing" BerkeleyDB.  Dual licensing was quite widely used as I remember.  You could either use software under a GPL licence, which meant that if you added any of your own code, you had to make it available under the GPL.  But there was a way that you could keep your own code proprietary, by buying a license from Sleepycat.  Apparently this was pretty lucrative for Sleepycat.  I think MySQL (now part of Oracle) and others had a similar type of dual licensing.

Another open source business model that a number of folks used was selling services and support - basically programmers' time to modify or enhance an open source project.  Organizations would pay open source developers to add a new file type or new functionality to an open source project.

Mike Olson makes the case that there were very few successful large stand-alone open source vendors.  He counts Red Hat as the only one that has not been acquired or has not failed.  But he says that Red Hat did not become successul purely through a support and services model, but through its closed-source Red Hat Network product, which it made open source after eliminating the serious competition.

The problem Mike Olson sees with a services and support model for open source is that as soon as a vendor becomes successful, meaning getting large enough to be noticed, it gets the attention of good, but low cost programmers in China, India, and Eastern Europe, who have access to the source code.  Ultimately this drives down the price yu can charge for support and services.

The other trend that Mike Olson points to is ubiquitous open source software infrastructure including operating systems (Linux), web servers (Apache), scripting language (Python, Perl, PHP), databases (MySQL), enterprise programming language (Java), virtualization (Xen), and so on.  Closed source is a diminishing part of this business.

So the quandary for folks who want to be in the enterprise infrastructure software business is, how do you create a large successful, stand-alone software company when the platform has to be open source, but pure play open source software companies historically haven't survived when they get large.

According to Mike Olson, the model that Cloudera has found to be successful is to build value for custmers as closed-source, proprietary IP that differentiates Cloudera on top of an open source platform Hadoop.  Clearly this requires commercially oriented, non-GPL open source licensing for the platform, such as the Apache or Eclipse licenses.

Thanks to Andrew Ross for pointing me to this.

James Fee on remaining relevant as a GIS professional

At this year's GIS in the Rockies, James Fee gave an enthralling keynote targeted at GIS professionals on the topic of how to remain relevant in the age of change (or What GIS Pros Can Do to Keep Their Skills in Demand).

He went through his own discovery of maps and technology beginning with atlases and continuing with a lot of the key technologies of the past decades most of which I can remember such as personal computers (Atari 800), the Logo programming language, Turbo Pascal, Hypercard, Freehand, Hamster dance (which I don't remember), the first releases of Arc/Info, Arc Macro Language (AML), spatial databases, the Internet, Sun, Sparcstations and Solaris, ArcView and the Avenue scripting language, mobile devices, big data, real-time, Perl, Python, spatial analytics, UAVs and 3D cities.  His main point is that geospatial folks have always been at the forefront in adopting new technologies and that GIS has a history of pushing the envelope.  His recommendation for GIS professionals is

• A always
• B be
• L learning

He also concluded that a GIS professional is a programmer, not in the restricted sense of a software developer, but in the more general sense of someone who makes computers do things.  James is clearly a fan of scripting languages including AML, Javascript and Perl.  He sees Python becoming the world's primary scripting language ("Python is the bee's knees").  James is a major baseball fan and he asserted that If you like GIS, you like statistics and furthermore if you like statistics, you like baseball. He gave a very simple example of using Python and several Python mathematical libraries to generate an interesting perspective on current baseball standings.

To answer the title question, what should a GIS professional do to stay relevant ?  James asserted that if you

• Put points on a map and throw up a scale bar
• Perform geoprocessing without Python or Model Builder
• Have a job description of “Plotter Operator”
• Have no idea what “fuzzy tolerance” is

you will soon be out of a job.  On the other hand if you

• Embrace Python as your GIS tool of choice.
• Use Model Builder to automate your work flows.
• Learn new tools such as TileMill/Mapnik/PostGIS

(all open source geospatial I would add) your future is bright.

Distributed spatial data management with long transactions

One of the key enabling technologies introduced in the early days of AM/FM/GIS systems was long transactions, also known as data versioning.  Long transactions make it possible to store spatial design data in the same database as as-built data, enabling design engineers to work on new designs at the same time as operations staff are relying on as-built information to operate and maintain active utility networks. 

The first commercial systems implementing this technology in the early 1990's were IBM's GFIS which ran on DB2, Intergraph FRAMME, and VISION* which managed versioned spatial data in an Oracle RDBMS (this was long before Oracle Spatial and Workspace Manager).  As I remember, San Diego Gas & Electric and BC Hydro were GFIS sites.  The largest FRAMME site was Ameritech, now part of AT&T.  The largest VISION^ sites, which are still running and supporting about a thousand concurrent designers,  are US West, now CenturyLink, and Telesp, now Telefonica Sao Paulo.

Smallworld implemented long transactions in 1996.  Since then both Oracle (Workspace Manager) and ArcSDE (now ArcGIS) have implemented support for long transactions. 

Each of these early implementations implemented long transactions in a quite different way.  For example, in some the number of versions at any given time is assumed to be small, but the number of objects in each version is large.  In others the number of versions can be large, and the the number of objects per version is small.

Recently there has been a renewed interest in spatial long transactions with a focus on managing them in a distributed data environment.  It is being proposed that the underlying version management platform be based on the same principles as Git which has become the de facto standard in the open source community.  Git was originally developed by Linus Torvalds for source code management for Linux.  Google, Facebook, Microsoft, twitter, Linked In, Qt, Gnome, and Android are some of the folks using Git. Millions of open source projects are hosted on GitHub.  The GeoGit project is an attempt to apply a Git inspired version management system to spatial data in a distributed environment. Currently, users can import raw geospatial data as Shapefiles, PostGIS or SpatiaLite into a repository where any changes to the data are tracked. These changes can be viewed in a history, reverted to older versions, branched in to sand boxed areas, merged back in, and pushed to remote repositories. GeoGit, still in early development or alpha mode, is written in Java, and is available under the BSD License.

One of the use cases GeoGit is intended to support is a hybrid data management process involving an authoritative database that can be updated by crowdsourced data.  It is being proposed that with distributed versioning, it is possible to have the  best of the crowdsourced (e,g,, OpenStreetMap) and authoritative (e,g,, TIGER) worlds.  "Data stewards could maintain their own data repositories for their particular areas of interest. They could also apply the same quality assurance checks as the central authority, which would enable their data to be easily pulled in to the central database. However, the central authority need not be involved in order for partnering stewards to exchange updates with one another. These inter-steward exchanges would also facilitate quicker updating to the central authority, because there would be fewer conflicts during the merging of the updates the various stewards submit."

Oracle Spatial and Graph to support NURBS

At the Oracle Spatial and Graph User Meeting, the folks from Oracle outlined some of the things to expect in the next version of Oracle 12c.  I was very impressed.  All of the below is my interpretation of what will be in Oracle 12c.

Parametric curve support

The really big news for the architecture, engineering and construction community (AEC) who use CAD and BIM applications is that Oracle 12c is expected to support NURBS (non-uniform rational basis spline) a type of parametric curve widely used in the design space.  According the Siva Ravada the support will be very general and will be able to handle different ways of representing NURBS. Up to now the only parametric curve supported by Oracle was circular arcs.

To me it looks like Oracle is taking a major step toward being able to store natively and manipulate (in some cases by stroking) the parametrized curves used by CAD and BIM applications.  There is a lot of CAD and BIM data out there and being able to store it and manipulate it in Oracle Spatial and Graph is going to be a game changer.

Orthorectification

Oracle is providing support for orthorectification embedded directly in the database.  This will have important implications for the industry.   Up to now, you had to pull the image out of where ever it was stored, transfer it over the network to someone's application often from an image processing company to orthorectify it, and then transfer it back to store the image or stream it to the end user.  With 12c you will be able to orthorectify images in the database.  This is in line with Oracle's objective of bringing processing to the data, rather than the data to processing.

Raster algebra

Oracle 12c will also have support for raster algebra, again in the database.  For example, you will be able to average in the database12 months of national temperature data stored in monthly raster files.

GDAL and PDAL support

Oracle supports the open source GDAL raster libraries, the industry standard developed by Frank Warmerdam (now with Google) and used by just about everyone for accessing raster images including ESRI.  PDAL is a similar type of open source library for point clouds developed by Howard Butler and Michael Gerlek.

3D

Oracle supports three types of 3D data.  3D vectors, point clouds and terrain (surfaces).  Oracle 12c will provide full support for 3D geodetic data, lon lat and elevation in meters or feet.

Performance

Oracle has developed a new engine for handling massive point clouds.  Dan Geringer showed impressive results for a 2.8 billion point point cloud (286 gigabyte file). 

Processing vector geospatial data is reported to be 50 to 100 times faster in 12c than in 11g. 

Raster operations have been parallelized which can dramatically reduce the time it takes for these operations.

Engineered Database

Since acquiring Sun, Oracle has become a hardware and software company and is offering integrated hardware and software solutions that offer extremely good performance characteristics.  For example, processing LiDAR data benefits from hardware acceleration on engineered machines such as Exadata boxes.

Bundled data

Oracle is planning to bundle vector data from Nokia (Navteq), TomTom and others.

Oracle Spatial and Graph

The name of the product is changing to Oracle Spatial and Graph partly because it has had graph capabilities (network and RDF semantic) already for several years.  But probably more importantly because the demand for graph capabilities is accelerating, especially RDF semantic graphs for linked data, text mining, and for social media analytics, according to Xavier Lopez.  Even Oracle's NOSQL database (aka BerkeleyDB) is apparently getting graph capabilities.