Between the Poles

GeoWave aims at linking popular geospatial tools to "big data" technology

A very exciting project has been proposed at LocationTech (it's in the Project Proposal Phase as defined in the Eclipse Development Process).  Simply put, GeoWave intends to do for "big data" databases (initially Apache Accumulo) what PostGIS does for SQL databases (PostgreSQL).  GeoWave is open source software (licensed under Apache 2.0) that adds support for geographic objects, multi-dimensional indexing and geospatial operators to Apache Accumulo.  

NoSQL databases

To deal with data volumes that are too large for traditional SQL databases, beginning in 2004 Google developed "BigTable" which is a compressed, high performance, and proprietary data storage system built on the Google File System that is used by a number of Google applications including Google Maps.  Apache Accumulo is a distributed database that is based on Google's BigTable design and is built on top of Apache Hadoop and other Apache projects.  For putting geospatial data into a key/value store like Accumulo the key concept is that of the "geospatial hash" which converts a 2D, 3D or 4D coordinate such as a lon and lat, lon, lat and elevation or a lon, lat, elevation, and time to an integer index, such as a quadtree or R-Tree index, that can be used to order and rapidly retrieve spatial data.  GeoWave means that you can manage massive amounts of geoinformation in key/value databases such as Accumulo and take advantage of programs such as MapReduce which Accumulo uses for distributed processing.

Connecting Accumulo to GeoServer, GeoTools, and PDAL

In addition GeoWave includes a GeoServer plugin to enable geospatial data in Accumulo to be shared and visualized via GeoServer OGC standard web services. It provides plugins to connect the popular geospatial toolset GeoTools and the point cloud library PDAL to an Accumulo based data store. The PDAL plugin makes it possible to interact with point cloud data in Accumulo through the PDAL library.
 
The GeoWave project Work plans to extend the same geospatial capabilities to other distributed key-value stores in addtition to Accumulo.  The next data store will be HBase.  It also will support other geospatial frameworks in addition to GeoTools/GeoServer.  Mapnik is the next geospatial framework targeted for GeoWave support.  GeoWave says it is very interested in GeoGig and support for this geospatial data versioning library is currently on their backlog.  GeoGig takes the concepts used in distributed version control such as Git and applies them to versioned spatial data.

Background
 
GeoWave was developed at the National Geospatial-Intelligence Agency (NGA) in collaboration with RadiantBlue Technologies and Booz Allen Hamilton.  The NGA released GeoWave under an open source license in June, 2014. The primary goal of GeoWave is to bridge the gap between well-known geospatial projects such as GeoTools and distributed databases.

I blogged previously about GeoMesa, the first LocationTech project that aims at providing a foundation for storing, querying, and transforming spatio-temporal data in Accumulo.  It implements interfaces that enable Geoserver and other Geotools projects to use Accumulo as a data store.

Open source spatial data versioning coming for utilities

The Research and Markets report Global GIS Market in the Utility Industry 2014-2018 identifies an important new market trend in the increasing penetration of open-source geospatial software into the utility market. According to the report many small and medium-sized organizations that cannot afford to invest in expensive proprietary GIS solutions are now choosing open-source software. In the U.S. most of the 3000 electric power utilities are small to medium-sized organizations, primarily municipal and rural electric coops.

Spatial data versioning (long transactions) for utility records management

Two essential capabilities for utility and telecom records management are storing spatial data in an SQL database and long transactions, more commonly referred to as data versioning these days  For a utility data versioning makes it possible to store spatial design data in the same database as as-built data, enabling design engineers to work on new designs at the same time as operations staff are relying on as-built information to operate and maintain active utility networks.  Many of the largest telecoms and utilties have outside plant solutions based on these technologies, in some cases since the 1990s.

Open source spatial data versioning

In the most recent Eclipse LocationTech Newsletter, Paul Ramsey describes the latest development on a major open source initiative to support spatial data versioning.  Paul points out that software developers have long had extremely versatile tools, such as Git, for managing software versions which tracks the details of all changes made to source code including dependencies in a distributed environment.  He and others have asked why don’t we have this capability for spatial data management ? 

The answer is GeoGig which takes the concepts used in distributed version control such as Git and applies them to versioned spatial data.  GeoGig is written in Java and is available under an open source  BSD License.

At the Location Intelligence Summit earlier this year in Washington, Juan Marin, CTO at Boundless,  gave an overview and demonstration of what was then called GeoGit, a distributed system for geospatial data versioning using the same philosophy as Git.

To further leverage the abilities of GeoGig, Boundless has announced Versio for organizations to manage their spatial data.  According to Paul Versio is a "hosted service built on GeoGig that provides easier toolks for data authors to collaborate on spatial data while streamlining the creation, maintenance, and distribution of spatial data."

Git

By way of background Git is a distributed revision control and source code management (SCM) system.  Git was initially designed and developed by Linus Torvalds for Linux kernel development in 2005.  It is free open source software distributed under the GNU General Public License (GPL)  V 2.

Every Git working directory is a full-fledged repository with complete history and full version tracking capabilities. Each local repository is standalone and is not dependent on network access to a central server. Git is known for very fast performance and for scalability.

History of long transactions for telecom/utility records management

Two capabilities that are essential for utility and telecom records management, storing spatial data in an SQL database and long transactions, were introduced in the early days of Automated Mapping/Facilities Mapping/Geographic Information Systems (AM/FM/GIS).  The first commercial systems implementing this technology in the early 1990's were IBM's GFIS which ran on DB2 and VISION* which managed versioned spatial data in an Oracle RDBMS (this was long before Oracle Spatial and Workspace Manager).  VISION* systems are still running at First Energy, supporting hundreds of concurrent utility network designers, and at CenturyTel (at the time US West then Qwest) and Telefonica Sao Paulo (at the time Telebras where the far-sighted visionary was the late Geovane Magalhaes) where each VISION* system supports about a thousand concurrent telecom outside plant designers.  VISION* was acquired by Autodesk in 1999.

Oracle Database Cloud Service supports Spatial

At Oracle Open World in San Francisco 2014, September 28–October 2 the Oracle Spatial and Graph SIG held a Meetup.  By way of background Oracle Spatial and Graph support geospatial data and analytics for land management and GIS, mobile location services, sales territory management, transportation, LiDAR analysis and location-enabled Business Intelligence. The graph features include RDF graphs for applications ranging from semantic data integration to social network analysis to linked open data and network graphs used in transportation, utilities, energy and telcos and drive-time analysis for sales and marketing applications.  

There were several very interesting discussions at the Meetup that revealed some interesting aspects of Oracle Spatial itselef and its interoperability with other products such as ESRI ArcGIS, Autodesk Map3D, and Bentley.  These are taken from a summary recorded by Nick Salem moderator of the Meetup.

The Meetup was attended by about 20 users came from industries such as rail, local city government, national land management agencies, and defense. The SIG Board and members of the Oracle Spatial product development team joined up with them in the Oracle Technology Network Lounge.

Some of the most interesting discussion topics include new features in Oracle Spatial 12c and real world experience interoperating Oracle Spatial with CAD and GIS products from other vendors.

• Oracle’s new Database Cloud Service offering now includes Spatial. 
• A new Oracle Spatial 12c SDO_POINTINPOLYGON function can be used to ingest large point datasets (point clouds) without the overhead of creating a spatial index.
• Bechtel has developed a way to use ESRI ArcGIS versioning with Oracle Workspace Manager (OWM).  They have developed in-house custom tools to usefeature-level metadata to be able to retrieve spatial content from their project and public spatial databases, as well as to compile the revision history of layers in the database to retrieve maps impacted by changes to individual features.
• Dan Geringer, Oracle Spatial Solutions Architect, advised that other large enterprises have also successfully implemented OWM with ESRI ArcGIS.
• Bulk processing and interoperability are two of the important advantages of Oracle Spatial for enterprises. Bechtel has large project-based deployments of Autodesk and Bentley CAD systems, and has worked with Autodesk in recent years to successfully write spatial data to their Enterprise GIS Oracle Spatial database from Autodesk Map 3D. They also have developed a GIS Desktop Procedure that details the required Autodesk client software configuration for Oracle Spatial and ESRI ArcSDE database connections. They will be updating it for use with Autodesk 2014 and 2015 clients.
• Latest version of MapViewer (11.1.1.7) supports HTML5 and JSon and the ability to work in disconnected mode. Standalone, web-based vector editing is also available. Wish Li and Jayant Sharma demonstrated  MapViewer on mobile devices, an Android phone and an iPad.

 

GeoGit - Distributed geospatial data versioning based on Git

At the Location Intelligence Summit 2014 (LI2014) in Washington, Juan Marin, CTO at Boundless,  gave an overview and demonstration of GeoGit, a distributed system for versioning geospatial data using the same philosophy as Git.  Data versioning is also known as long transactions and is a fundamamental data management technology in a number of industries.  For example, long transactions are  used by utilities to protect their production as-built information (what is actually in the ground) with a process for managing updates and new network design work.  Another example is OpenStreetMap (OSM) where there is a production OSM database that is continually being updated with crowd-sourced data.  Data versioning is an ideal strategy for managing the updates and protecting the authoritative database.

Currently GeoGit users are able to import raw geospatial data from shapefiles, PostGIS or SpatiaLite into a repository where every change to the data is tracked. These changes can be viewed in a history, reverted to older versions, branched in to sandboxed areas, merged back in, and pushed to remote repositories.  GeoGit is written in Java, and available under the BSD License.  GeoGit is a proposed LocationTech project.  LocationTech is part of the Eclipse Foundation.

Git

By way of background Git is a distributed revision control and source code management (SCM) system.  Git was initially designed and developed by Linus Torvalds for Linux kernel development in 2005.  It is free open source software distributed under the GNU General Public License (GPL)  V 2.

Every Git working directory is a full-fledged repository with complete history and full version tracking capabilities. Each local repository is standalone and is not dependent on network access to a central server. Git is known for very fast performance and for scalability. Git allows and encourages you to have multiple local branches that can be entirely independent of each other. One of the things this branching model allows you to do is to keep several concurrent branches, for example,

• Production - one branch that contains only what goes to production
• Working - another branch that you merge new work into
• Several smaller ones for new features you're working on so you can switch back and forth between them, then delete each branch when that feature gets merged into your main line

Pushing to remote repositories - When you push to a remote repository, you can choose to share one of your branches, several of them, or all of them. For example, you may only want to share your production branch.  When you push your production repository to a remote repository (called a merge) which may have been updated by several other folks while you were working, Git will detect all conflicts and has a well-defined model of an incomplete merge.  Git has multiple algorithms for completing the merge automatically. If it isn't able to complete the merge automatically, manual editing will be required.  The other important thing from a data flow perspective is that only the incremental changes are updated in the remote repository.

GeoGit

GeoGit is currently available as version 0.8.  Juan says that it is feature complete with a complete command line interface (CLI).  He  projects that V1.0 of GeoGit will be available by the summer.  In addition Integration with QGIS is in progress and it is expected that a tutorial with GeoGit running with QGIS will be available soon.  GeoGit currently only supports vector data.

Perhaps the first question is why not just use Git and a well known geodata format such as shape or GeoJSON ? Juan pointed out that you can actually do this, but there are a couple of reasons why this may not be optimal.  First of all, Git doesn't support large binary files.  Secondly, this isn't well integrated into the geospatial ecosystem so you'll have to do work to make sense of conflicts, for example.

Juan emphasized the advantages of the GeoGit approach

• No single point of failure - this is a distributed  (peer to peer) geospatial data versioning system.  As in Git everyone has a complete and independent copy of the repository.
• No single point of truth - this may not seem ideal, but as pointed out above the Git model enables you to maintain shared production and development branches.
• The GIt model is proven technology that enables scalable collaboration among remote developers

Juan described and demonstrated basic GeoGit processes including

1.  Create a repository

2. Import geospatial data, for example, shape or OSM data, into the GeoGit staging area

geogit shp import [bounding box]

geogit osm import [bounding box]

3. Edit it in the staging area

4. Commit the changes to the repository

geogit commit -m "first commit"

5. Create a branch

geogit branch <branch1>

6. Import or create some new data in the staging area, for example, digitize a building footprint polygon

7. Commit it

8. Merge it back into the main branch

geogit merge <branch1> <main>

Geogit will find, attempt to resolve and report conflicts

9. View history - view the main branch before and after the commit

10. Synchronize the repository with a remote repository

Geogit will find, attempt to resolve and report conflicts

11. Export the main branch as a shape file

geogit shp export

Following Juan's presentation, Scott Clark, Director of Geospatial Prgrams al LMN Solutions, a solution provider to the intel community described and demonstrated an application for collaborative mapping developed for DoD as a Joint Capability Technology Demonstration (JCTD) that uses GeoGit in addition to other open source geospatial tools including GeoNode, PostGIS, and GeoServer.

 Next steps

Juan outlined the main areas of focus for current and future development activities.

• High performance GeoGit server
• QGIS plug-in
• Web UI
• Python support
• Scalability - large repository support
• Performance optimization

LocationTech attracts open source geospatial projects for managing "big spatial data"

I blogged earlier about Google joining LocationTech as a strategic member (together with IBM, Oracle, Boundless( was OpenGeo), and Actuate). 

After lining up key strategic members, the next step was to attract open source geospatial projects which wanted to take advantage of the governance and intellectual property services including commercially friendly licensing that LocationTech/Eclipse provides.  LocationTech has recently received a number of project proposals that will put LocationTech on the leading edge of open source geospatial development.

SpatialHadoop

Hadoop provides scalable processing for huge datasets by using MapReduce, a programming paradigm for distributed processing, to build an efficient framework for processing large-scale data.  SpatialHadoop is an extension to Hadoop that allows efficient processing of "big spatial data".

GeoMesa

Accumulo is a robust, scalable, high performance data storage and retrieval system based on Google's BigTable design and built on Hadoop.  GeoMesa provides a foundation for storing, querying, and transforming spatio-temporal data in Accumulo.  It implements Geotools interfaces that enable Geoserver and other Geotools related projects to use Accumulo as a data store.

GeoTrellis

GeoTrellis is a general framework for low-latency geospatial data processing developed using Scala and Akka.    GeoTrellis was designed to solve three core problems, initially for raster processing, scalable, high performance geoprocessing web services; distributed geoprocessing services for large data sets; and arallelizing geoprocessing operations to take full advantage of multi-core architectures.

GeoGit

I blogged about this recently because it is a project to develop distributed long transactions for geospatial data. 

GeoGit is a distributed version control system specially designed to handle geospatial data efficiently. It takes inspiration from the source code versioning system Git, but has an approach suited to spatial data (currently Shapefiles, PostGIS, MS SQLServer, or SpatiaLite.) GeoGit efficiently handles very large binary data and is optimized for spatial data using a spatial index.

JTS Topology Suite

The JTS Topology Suite (JTS) is an open source Java software library that provides an object model for planar geometry together with a set of fundamental geometric functions. JTS conforms to the Simple Features Specification for SQL published by the Open GIS Consortium.  JTS provides the core component for vector-based geomatics software or can be used as a general-purpose computational geometry library.

Geoff (Geo Fast Forward)

Geoff is intended to enable existing Eclipse Rich Client Platform (RCP) applications to integrate into the geospatial domain using lightweight components to visualizing geospatial information on a geographical map embedded in the RCP applications.

GeoScript

GeoScript adds spatial capabilities to dynamic scripting languages. GeoScript provides a collection of modules based on the Geotools library for geometry handling, spatial data access, and vector feature rendering.  GeoScript is currently implemented in Groovy, JavaScript, Python, and Scala.

Distributed spatial data management with long transactions

One of the key enabling technologies introduced in the early days of AM/FM/GIS systems was long transactions, also known as data versioning.  Long transactions make it possible to store spatial design data in the same database as as-built data, enabling design engineers to work on new designs at the same time as operations staff are relying on as-built information to operate and maintain active utility networks. 

The first commercial systems implementing this technology in the early 1990's were IBM's GFIS which ran on DB2, Intergraph FRAMME, and VISION* which managed versioned spatial data in an Oracle RDBMS (this was long before Oracle Spatial and Workspace Manager).  As I remember, San Diego Gas & Electric and BC Hydro were GFIS sites.  The largest FRAMME site was Ameritech, now part of AT&T.  The largest VISION^ sites, which are still running and supporting about a thousand concurrent designers,  are US West, now CenturyLink, and Telesp, now Telefonica Sao Paulo.

Smallworld implemented long transactions in 1996.  Since then both Oracle (Workspace Manager) and ArcSDE (now ArcGIS) have implemented support for long transactions. 

Each of these early implementations implemented long transactions in a quite different way.  For example, in some the number of versions at any given time is assumed to be small, but the number of objects in each version is large.  In others the number of versions can be large, and the the number of objects per version is small.

Recently there has been a renewed interest in spatial long transactions with a focus on managing them in a distributed data environment.  It is being proposed that the underlying version management platform be based on the same principles as Git which has become the de facto standard in the open source community.  Git was originally developed by Linus Torvalds for source code management for Linux.  Google, Facebook, Microsoft, twitter, Linked In, Qt, Gnome, and Android are some of the folks using Git. Millions of open source projects are hosted on GitHub.  The GeoGit project is an attempt to apply a Git inspired version management system to spatial data in a distributed environment. Currently, users can import raw geospatial data as Shapefiles, PostGIS or SpatiaLite into a repository where any changes to the data are tracked. These changes can be viewed in a history, reverted to older versions, branched in to sand boxed areas, merged back in, and pushed to remote repositories. GeoGit, still in early development or alpha mode, is written in Java, and is available under the BSD License.

One of the use cases GeoGit is intended to support is a hybrid data management process involving an authoritative database that can be updated by crowdsourced data.  It is being proposed that with distributed versioning, it is possible to have the  best of the crowdsourced (e,g,, OpenStreetMap) and authoritative (e,g,, TIGER) worlds.  "Data stewards could maintain their own data repositories for their particular areas of interest. They could also apply the same quality assurance checks as the central authority, which would enable their data to be easily pulled in to the central database. However, the central authority need not be involved in order for partnering stewards to exchange updates with one another. These inter-steward exchanges would also facilitate quicker updating to the central authority, because there would be fewer conflicts during the merging of the updates the various stewards submit."