Between the Poles

Oracle Spatial and Graph to support NURBS

At the Oracle Spatial and Graph User Meeting, the folks from Oracle outlined some of the things to expect in the next version of Oracle 12c.  I was very impressed.  All of the below is my interpretation of what will be in Oracle 12c.

Parametric curve support

The really big news for the architecture, engineering and construction community (AEC) who use CAD and BIM applications is that Oracle 12c is expected to support NURBS (non-uniform rational basis spline) a type of parametric curve widely used in the design space.  According the Siva Ravada the support will be very general and will be able to handle different ways of representing NURBS. Up to now the only parametric curve supported by Oracle was circular arcs.

To me it looks like Oracle is taking a major step toward being able to store natively and manipulate (in some cases by stroking) the parametrized curves used by CAD and BIM applications.  There is a lot of CAD and BIM data out there and being able to store it and manipulate it in Oracle Spatial and Graph is going to be a game changer.

Orthorectification

Oracle is providing support for orthorectification embedded directly in the database.  This will have important implications for the industry.   Up to now, you had to pull the image out of where ever it was stored, transfer it over the network to someone's application often from an image processing company to orthorectify it, and then transfer it back to store the image or stream it to the end user.  With 12c you will be able to orthorectify images in the database.  This is in line with Oracle's objective of bringing processing to the data, rather than the data to processing.

Raster algebra

Oracle 12c will also have support for raster algebra, again in the database.  For example, you will be able to average in the database12 months of national temperature data stored in monthly raster files.

GDAL and PDAL support

Oracle supports the open source GDAL raster libraries, the industry standard developed by Frank Warmerdam (now with Google) and used by just about everyone for accessing raster images including ESRI.  PDAL is a similar type of open source library for point clouds developed by Howard Butler and Michael Gerlek.

3D

Oracle supports three types of 3D data.  3D vectors, point clouds and terrain (surfaces).  Oracle 12c will provide full support for 3D geodetic data, lon lat and elevation in meters or feet.

Performance

Oracle has developed a new engine for handling massive point clouds.  Dan Geringer showed impressive results for a 2.8 billion point point cloud (286 gigabyte file). 

Processing vector geospatial data is reported to be 50 to 100 times faster in 12c than in 11g. 

Raster operations have been parallelized which can dramatically reduce the time it takes for these operations.

Engineered Database

Since acquiring Sun, Oracle has become a hardware and software company and is offering integrated hardware and software solutions that offer extremely good performance characteristics.  For example, processing LiDAR data benefits from hardware acceleration on engineered machines such as Exadata boxes.

Bundled data

Oracle is planning to bundle vector data from Nokia (Navteq), TomTom and others.

Oracle Spatial and Graph

The name of the product is changing to Oracle Spatial and Graph partly because it has had graph capabilities (network and RDF semantic) already for several years.  But probably more importantly because the demand for graph capabilities is accelerating, especially RDF semantic graphs for linked data, text mining, and for social media analytics, according to Xavier Lopez.  Even Oracle's NOSQL database (aka BerkeleyDB) is apparently getting graph capabilities.

Open source: then and now

I was asked participate on a panel on open source geospatial organized by Eclipse LocationTech at the Location Intelligence in Washington DC.  I decided to look at how software had changed since the early days of open source which I tend to tie to the formation of the Apache Software Foundation (ASF) in 1999.  What I found was that the proprietary and open source software landscape has changed dramatically in the last 13 years.  Here are some statistics that provide an indication of just how dramatic the change has been.

Apache Software Foundation

Apache which started off supporting the Apache http server developed at the National Center for Supercomputing Applications (NCSA) now supports over 100 top level projects. 

According to a Netcraft survey of 649,072,682 web sites in April 2013

• Over 51% run Apache http server
• Almost 20% run Microsoft IIS

Microsoft

In 1998 the infamous Halloween docs were leaked in which Microsoft identifed open source as a threat and discussed ways of combatting it. 

April 2013 is the first anniversary of Microsoft Open Technologies, Microsoft's own open source division.  Microsoft also has another open source group, Outercurve, where more than 50% of the projects are run by non-Microsoft developers.  Microsoft announced that Visual Studio now supports Github.

Browsers

Until several years ago, Internet Explorer was used by over 90% of users.  The most recent statistics from W3Schools for April 2013 show marketshare for browsers is dominated by open source browsers, Chrome and Firefox.  IE is a poor third now.

• Chrome 52.7 % (increasing)
• Firefox 27.9 % (decreasing)
• IE 12.7% mostly IE8 and IE9 (declining)
• Safari 4% (decreasing)
• Opera 1.7% (decreasing)

Operating systems on personal devices

Until several years ago,  PCs running MS Windows were used by over 90% of users.  Now handheld devices such as smart phones lead PCs in annual sales.  IDC 1Q2013 statistics for smartphone shipments show that in the last quarter, open source operating systems are dominant

• Android 75.0%
• iOS 17.3%
• Windows Phone 3.2%
• Blackberry OS 2.9%
• Linux 1.0%
• Symbian 0.6%

Software procurement

In many large organizations until recently the procurement playing field was tilted in favour of proprietary cmmercial and against open source software.  In 2009 the Department of Defense (DOD), the world's largest IT organization, issued a memorandum that leveled the playing field by stating that “OSS [open source software] meets the definition of commercial computer software”.  The memorandum went on to identify some of the benefits of OSS for DOD.

• Fewer defects - many eyeballs means fewer defects
• Flexibility - easier to customize and adapt
• Multiple vendors instead of one - avoids vendor lock-in
• Unrestrictive license - simpler to deploy
• Cost advantage - when many copies are required
• Cost sharing  - reduces DOD costs
• Prototyping and experimentation - OSS particularly suitable for this

Large IT systems

I blogged earlier about a study by Henrik Ingo that found that the world's largest OSS projects including Linux, KDE, Apache, Eclipse, Perl+CPAN, Mozilla+Addons, Gnome, Drupal and GNU are collaborative community projects governed by non-profit foundations like Apache and Eclipse.  These projects are10 times larger than the largest single-vendor projects.  Ingo recommended that vendors seriously consider participating in a collaborative community run by a  non-profit foundation because they  could expect 10 time growth in the project and its addressable market.

Malware

Several years ago virtually all of the world's malware exploits targeted MS Windows.  Now according to F-Secure, exploits against mobile devices are exploding.  In 1Q2013 the main target for mobile exploits is Android which is open source.

• 91.3% Android (49% increase over 4Q2012) *
• 8.7% Symbian
• Apple, Microsoft, and Blackberry - free of malware

In 1Q 2013 for the first time Android malware was identified by F-Secure as being distributed by email in addition to apps.

Geospatial open source

The Open Source Geospatial Foundation was formed in 2006 supported by Autodesk.  Prior to that geospatial software was dominated by proprietary vendors ESRI, Intergraph, Autodesk, Bentley, and others.  Now there are many open source geospatial companies that provide alternatives to proprietary vendors.  They primarily rely on OSGEO software such as PostGIS, OpenLayers, MapServer, GeoServer, and others.  Earlier this year ESRI acquired GeoIQ and now offers both open source and proprietary software.

Open geospatial standards

One of the key reasons for the rapid expansion in the deployment and use of open source geospatial software solutions is the open standards developed by the Open Geospatial Consortium notably the SQL Simple Feature Specification, KML, WMS, WFS, GML, and CSW.  Open source and open standards are very different things, but open source gravitates to open standards.  It is hard to imagine that geospatial open source would have progressed as rapidly and widely as it has without the OGC standards.

Open source geospatial lab announced at University of Southampton

The Open Source Geospatial Foundation (OSGEO) has announced the formation of the Open Source Geospatial Laboratory (OSGL) at the University of Southampton in the UK..

In 2011, the International Cartographic Association (ICA) signed a Memorandum of Understanding (MoU) with the OSGeo to develop global collaboration opportunities for academia, industry and government organisations in the field of open source geospatial software and data.

The Open Source Geospatial Laboratory is supported by the GeoData Institute and Geography and Environment at the University of Southampton and brings together staff from various disciplines including Engineering, Geography, Electronics and Computer Science.

The laboratory will conduct open source research for geospatial software and data, using open source geospatial software and data as tools and contributing to the development of tools, standards and data through research. The lab will seek to exploit new opportunities for research, often in partnership with other OSGLs and members of the ICA-OSGeo initiative. The Laboratory will also deliver open source geospatial training programs.

Open source geospatial LocationTech launched

Today marks the launch of LocationTech, a community of open source projects and consortium of member companies.  LocationTech is designed to help broaden adoption of commercial location aware technologies.  It is vendor neutral, open to all, and operates transparently.  LocationTech is the Eclipse Foundation's industry working group focusing on location aware technologies. Members of LocationTech are also full fledged members of the Eclipse Foundation.

LocationTech is hosted by the Eclipse Foundation.  Eclipse is a vendor neutral community for individuals and organizations who wish to collaborate on commercially-friendly open source software. The Eclipse Foundation is a not-for-profit, member supported corporation that hosts technology projects and helps cultivate both an open source community and an ecosystem of complementary products and services.  Eclipse hosts over 270 active open source projects and is supported by over 185 members.

Commercially-friendly licensing

An important aspect of Eclipse is the focus on enabling the use of open source technology in commercial software products and services. Eclipse consciously promotes and encourages software vendors to use Eclipse technology for building their commercial software products and services. This is made possible because Eclipse projects are licensed under the Eclipse Public License (EPL), a commercial friendly OSI approved license.

LocationTech Governance

LocationTech is governed by bodies consisting of member representatives including a steering committee, architecture committee, and a marketing committee. These bodies make strategic, technology, and marketing decisions for LocationTech.  There are four levels of participation at LocationTech:

Guest Members

Guest Members are people or organizations of note that are invited to participate in the working group by LocationTech's Steering Committee.

Committer Members

Committer members are nominated and elected by other committers to become a committer for a given open source project. Committers members contribute to open source projects hosted at LocationTech.

 Participating members

Participating members view location aware technologies as an important part of their business. They invest to support and participate in the ecosystem.

Strategic Members

Strategic members view location aware technologies as strategic to their business. They invest to further develop the technology and influence the direction of the industry.

Open source reference implementation of the OGC Catalog Service released

The Open Geospatial Consortium (OGC) has made a determined effort to work with the open source community.Osgeologo   For example, the OGC and the Open Source Geospatial Foundation (OSGeo) signed a Memorandum of Understanding (MOU) a few years ago.  One of the concrete results of the collaboration  between the OGC and the open source geospatial community is that OGC official reference implementations are open source.

A new reference implementation of an OGC standard hs just been announced.  pycsw is an official reference Implementation of the OGC Catalogue Service for the Web (CSW).  CSW provides a standard for publishing and the discovery of geospatial metadata.  As its name suggests, pycsw is written in Python.  It is certified OGC Compliant.

pycsw is an open source project under an MIT license and supports Windows, Linux, and Mac OS X.

Cesium open source WebGL virtual globe and mapping Javascript library

Earlier this week on an Eclipse Foundation Locationtech teleconference, Matt Ford of Cesium provided an overview and demonstration of the features of Cesium.  You can watch the video here.

Cesium is an open source virtual globe and map library written in JavaScript using WebGL. It provides hardware-accelerated graphics in a browser without a plugin.  Cesium supports a 3D globe, 2D map, and 2.5D with the same API.   Cesium allows you to layer imagery from multiple sources, including imagery (WMS, OpenStreetMap, Bing Maps, ArcGIS MapServer, and standard image files) and vector data (KML, ESRI Shapefiles, WebGL, JSON).

It supprts vector editing including pionts (labels and markers), polylines, polygons with holes, circles, ellipses, and other shapes.  It also supports cameras and animation.  One of the cool things Matt showed was a camera following a satellite.  And perhaps the coolest capability is materials which alows you to define the appearance of the surface of objects using textures such as water, checkerboard, stripes, dots, brick, cement, asphalt, wood, grass, diffuse map, specular map, reflection, and refraction.

Functionality that Cesium already supports in beta and that looked impressive in Matt's demonstration is digital terrain models.  Functionality currently under development are

• Models - the example shown was a rubber duck
• Integration with other open source packages - the example shown was a plugin for GeoServer

But perhaps the most exciting capability to look forward to is an OpenLayers API on top of Cesium.  This would provide OpenLayers with an extremely rich set of 3D capabilities in the very near future.

What to expect in the OpenLayers 3

Last week on an Eclipse Foundation Locationtech teleconference, Tim Schaub of OpenGeo outlined the plans for the next version of OpenLayers.   You can watch the video and contribute to the effort  here.

OpenLayers2 was released in Aug 2006.  It implemented a lot of the cool stuff based on AJAX and tiles in Google Maps as a Javascript library.  OL2 became the open-source standard for slippy maps and the most widely used geospatial Javascript library. 

The technology ecosystem at that time was very different from today.  For example, IE6 had about 86% of the browser market and mobile devices did not support Javascript.  Since then many things have changed, but since backward compatibiity has always been a priority for OpenLayers, applications that ran in 2006 still run today.  Some of the reasons that OpenLayers became so widely adopted are support for many different formats and protocols, both proprietary and open standards, support for multiple geographical projections,  and advanced vector editing and rendering, first introduced in OL2.4. 

But overtime backward compatibility and the evoluton of the library in a rapidly chaning technical world has meant that cruft has accumulated and it is felt that it is time for a major upgrade to OpenLayers.

Goals of OpenLayers3

The major goal for the next major version release of OpenLayers is to build on existing OL2 functionality, but to make it small for fast loading and fast running.  It is also intended to have broad functionality, all of which is tested together, unlike some of the micro frameworks that have appeared recently.  OL3 also needs to be more friendly, includes a new, clean API and a modern look for UI components.

Some of the important new capabiltiies planned for OL3 include

• Separated renderers - support for both traditional DOM as well as WebGL renderers.  In the future there could be renderers for PDF and Canvas 2D. 
• Google's Closure Compiler - allows you to build javascript libraries that include just the code you need to run your application, to run it in a specific browser or on a specific device, or to run it with a specific renderer.
• Cesium - the intention is to integrate the new Cesium library to enable full 3D spinning globe capabilities in OL3.

Some of the other developments planned include oblique views, animation, cameras, and tutorials and improved documentation.  More technical details about OL3 can be found here.

Venkatesh Raghavan awarded Sol Katz award for his contributions to the Geospatial Free and Open Source community

Every year the OSGEO awards the Sol Katz Award for Geospatial Free and Open Source Software to people who have demonstrated leadership in the GFOSS community and contributed significantly through their activities to advance open source ideals in the geospatial realm.  Sol Katz was an early pioneer of GFOSS and among many other things assisted in the development of a public domain GIS package called MOSS (Map Overlay and Statistical System), arguably the first open source GIS software in the world.  Previous recipients of the Sol Katz award are  2011: Martin Davis, Helena Mitasova, Daniel Morissette, Paul Ramsey, Steve Lime, Markus Neteler, and Frank Warmerdam.

Prof. Venkatesh Raghavan of Osaka City Univiersity was awarded the 2012 Sol Katz Award for Geospatial Free and Open Source Software during the first session of the FOSS4G-India 2012 conference in Hyderabad, India.  Venka has been involved in OSGeo since the foundation was formed in 2006.  I remember his very early involvement in organizing the first open source geospatial symposium,  at an early Map World Forum in Hyderabad if I remember. His committment to open source geospatial and his community work, particularly in Japan and India, promoting open source geospatial is well-known to everyone in the GFOSS community.

The next step in the evolution of open source geospatial software

Apache Foundation and corporate engagement

As I remember the story, back in the late 1990's IBM, seeing a need for a http server for the rapidly developing internet market, looked at a couple of alternatives, developing one in-house or adopting the http server developed at the National Center for Supercomputing Applications (NCSA).  The NCSA http server project had been terminated at the NCSA, but the http server continued to be supported by a group of eight developers.  As I remember it IBM decided that it would be more cost effective and would make more sense from a market perspective to adopt what was the de facto industry standard.  But IBM insisted that for this to work there had to be a legal entity with which IBM as a corporation could commercially and and legally engage.  And so on March 25, 1999, the Apache Software Foundation was formed and the rest is history.   IBM and ultimately many others adopted and distribute the Apache server.  As of September 2012 it was estimated that  Apache serves about 55% of all active websites.  From an industry perspective adopting the Apache server was a much more cost-effective alternative than if HP, Oracle, IBM, Sun, and others had each developed their own http server.

OSGEO

Back in 2005/2006 I was working with Dave McIlhagga to interest Autodesk in becoming a corporate sponsor of open source geospatial software, and I saw a similar scenario played out.  It seemed to many of us that it would be more cost-effective for companies to share web-mapping software infrastructure rather than each develop its own web mapping server.  At that point there was already well-developed and well-known open source geospatial projects whose code was being used all over the world.  The best known examples were the University of Minnesota's MapServer project and Frank Warnerdam's GDAL geo-imagery libraries.  Again Autodesk insisted that there had to be a legal entity with which as a corporation it could engage.  The Open Source Geospatial Foundation (OSGEO) was modeled loosely on the Apache Foundation and was founded in March, 2006 with Autodesk as the major sponsor.  The OSGEO has been incredibly successful.  It has grown rapidly to support 25 or so projects, some in incubation,  20 chapters worldwide, and a national conference, FOSS4G, which last year in Denver attracted about 900 participants.

The next step

As a recent article has pointed out, 15 years ago when open source was just beginning, the discussion was focussed on the total cost of open source vs proprietary, alternative open source business models, and open source as an alternative to monopolistic proprietary.

Since then open source has come a long way.  One of the most important developments is the rise of the open source foundation driven by the need for corporate engagement.   These non profit legal entities offer projects important benefits that are difficult for a project to fund on its own.

• A host for managing fiscal and intellectual property shared resources such as trademarks and shared copyrights
• An employer for staff serving the community and project
• A guarantor and enabler for governance
• An infrastructure provider
• A liability firewall for community participants

Two examples of such foundations are the Apache Software Foundation and the Eclipse Foundation. These foundation encourage trust in the long term stability of the projects they support and, most importantly for enterprise software, they encourage corporate participation.

The other major development is in the area of licensing.  The Open Source Initiative (OSI) lists 69 different licenses that range from the Gnu Public License (GPL) to the very liberal MIT/BSD style licences.  Recent years have seen the development of licences in the middle ground that are designed to be business friendly, in particular to allow mixing of open source and proprietary modules.  The Eclipse Public Licence (EPL) is an example of such as licence and is a critical reason that large corporations such as IBM and Oracle are important supporters of the Eclipse Foundation.

LocationTech

There is a pent up interest for geospatially enabling enterprise information technology. For open source projects interested in encouraging adoption and support from vendors in this area, doing business implies corporate engagement with companies such as IBM, Oracle, and other major IT vendors.

A number of people advocating for open source geospatial software have seen the need for services and facilitates to enable corporate engagement. In my view, the type of projects that are seeking corporate engagement would be attracted to the Eclipse Foundation. The Eclipse Foundation provides services to reduce friction for organizations to re-use and contribute to open source projects. This supports business developing products and services that depend upon open source and in turn, the open source projects benefit from re-use, investment, and increased credibility.

Based on this thinking Andrew Ross, well known for his work in a variety of open source communities, initiated what is now officially known as the Eclipse Foundation LocationTech Working Group along with a team of representatives from notable companies and open source projects.  In my and others view. the LocationTech Working Group fills an important gap in the enterprise geospatial market.  A close relationship between OSGEO and LocationTech is advantageous to the broad open source community because they are both serving the community, but in different, complementary ways.

Major project to improve accessibility to OpenStreetMap data initiated

I have blogged on several occasions about OpenStreetMap (OSM), which is the world's largest volunteered, crowd-sourced database of geospatial information. Apparently, August 9^th 2004 was the day that the first user registered the domain and started the project.  The first time I blogged about OSM was just shortly after March 16th 2009 the day that the 100 000^th user registered on OpenStreetMap.  Since then the amount of data and the number of registered users has continued to increase exponentially.  Currently OSM has about 750,000 registered users.

April 1 of this year OpenStreetMap moved to a new license. The ODbL license is comprised of an Open Database License (ODbL), a Database Contents License (DbCL) covering the database itself and its contents, and a set of upgraded Contributor Terms covering submissions to the database.

One of the challenges for the OSM community is that although there are 750,000 registered users, only about 24,000 of these are active mappers who regularly create maps using OSM data,  Today on an Eclipse Foundation Locationtech teleconference, Alex Barth of MapBox presented an overview of a major project awarded to Development Seed and MapBox by the Knight Foundation that is designed to improve accessibility to OSM data.  The focus of the project is to improve the core infrastructure of OpenStreetMap to make

• it easier to add data to OpenStreetMap
• OpenStreetMap.org more social to support the community as it continues its rapid growth
• it easier for people to access OpenStreetMap data to make their own maps.

The intention is that all of these tools will be open source, with open  licensing, built transparently on open development platforms like GitHub, and worked on collaboratively with the larger OpenStreetMap community.  The technical priorities as outlined by Alex include

• Data API for editors and data consumers (Javascript API) and supporting libraries
• Improved editors using the new API including browser-based editors
• More social OpenStreetMap,org site

The project will begin ramping up over the next weeks and Alex is asking for as much community participation as possible.  The best place to follow the project as it unfolds is the MapBox blog.