Between the Poles

Open source: then and now

I was asked participate on a panel on open source geospatial organized by Eclipse LocationTech at the Location Intelligence in Washington DC.  I decided to look at how software had changed since the early days of open source which I tend to tie to the formation of the Apache Software Foundation (ASF) in 1999.  What I found was that the proprietary and open source software landscape has changed dramatically in the last 13 years.  Here are some statistics that provide an indication of just how dramatic the change has been.

Apache Software Foundation

Apache which started off supporting the Apache http server developed at the National Center for Supercomputing Applications (NCSA) now supports over 100 top level projects. 

According to a Netcraft survey of 649,072,682 web sites in April 2013

• Over 51% run Apache http server
• Almost 20% run Microsoft IIS

Microsoft

In 1998 the infamous Halloween docs were leaked in which Microsoft identifed open source as a threat and discussed ways of combatting it. 

April 2013 is the first anniversary of Microsoft Open Technologies, Microsoft's own open source division.  Microsoft also has another open source group, Outercurve, where more than 50% of the projects are run by non-Microsoft developers.  Microsoft announced that Visual Studio now supports Github.

Browsers

Until several years ago, Internet Explorer was used by over 90% of users.  The most recent statistics from W3Schools for April 2013 show marketshare for browsers is dominated by open source browsers, Chrome and Firefox.  IE is a poor third now.

• Chrome 52.7 % (increasing)
• Firefox 27.9 % (decreasing)
• IE 12.7% mostly IE8 and IE9 (declining)
• Safari 4% (decreasing)
• Opera 1.7% (decreasing)

Operating systems on personal devices

Until several years ago,  PCs running MS Windows were used by over 90% of users.  Now handheld devices such as smart phones lead PCs in annual sales.  IDC 1Q2013 statistics for smartphone shipments show that in the last quarter, open source operating systems are dominant

• Android 75.0%
• iOS 17.3%
• Windows Phone 3.2%
• Blackberry OS 2.9%
• Linux 1.0%
• Symbian 0.6%

Software procurement

In many large organizations until recently the procurement playing field was tilted in favour of proprietary cmmercial and against open source software.  In 2009 the Department of Defense (DOD), the world's largest IT organization, issued a memorandum that leveled the playing field by stating that “OSS [open source software] meets the definition of commercial computer software”.  The memorandum went on to identify some of the benefits of OSS for DOD.

• Fewer defects - many eyeballs means fewer defects
• Flexibility - easier to customize and adapt
• Multiple vendors instead of one - avoids vendor lock-in
• Unrestrictive license - simpler to deploy
• Cost advantage - when many copies are required
• Cost sharing  - reduces DOD costs
• Prototyping and experimentation - OSS particularly suitable for this

Large IT systems

I blogged earlier about a study by Henrik Ingo that found that the world's largest OSS projects including Linux, KDE, Apache, Eclipse, Perl+CPAN, Mozilla+Addons, Gnome, Drupal and GNU are collaborative community projects governed by non-profit foundations like Apache and Eclipse.  These projects are10 times larger than the largest single-vendor projects.  Ingo recommended that vendors seriously consider participating in a collaborative community run by a  non-profit foundation because they  could expect 10 time growth in the project and its addressable market.

Malware

Several years ago virtually all of the world's malware exploits targeted MS Windows.  Now according to F-Secure, exploits against mobile devices are exploding.  In 1Q2013 the main target for mobile exploits is Android which is open source.

• 91.3% Android (49% increase over 4Q2012) *
• 8.7% Symbian
• Apple, Microsoft, and Blackberry - free of malware

In 1Q 2013 for the first time Android malware was identified by F-Secure as being distributed by email in addition to apps.

Geospatial open source

The Open Source Geospatial Foundation was formed in 2006 supported by Autodesk.  Prior to that geospatial software was dominated by proprietary vendors ESRI, Intergraph, Autodesk, Bentley, and others.  Now there are many open source geospatial companies that provide alternatives to proprietary vendors.  They primarily rely on OSGEO software such as PostGIS, OpenLayers, MapServer, GeoServer, and others.  Earlier this year ESRI acquired GeoIQ and now offers both open source and proprietary software.

Open geospatial standards

One of the key reasons for the rapid expansion in the deployment and use of open source geospatial software solutions is the open standards developed by the Open Geospatial Consortium notably the SQL Simple Feature Specification, KML, WMS, WFS, GML, and CSW.  Open source and open standards are very different things, but open source gravitates to open standards.  It is hard to imagine that geospatial open source would have progressed as rapidly and widely as it has without the OGC standards.

Making geospatial data and services meaningful in the real world

The Semantic Web is an effort of the World Wide Web Consortium (W3C).  The idea of the Semantic Web is to provide a common framework that allows data to be shared and reused across application, enterprise, and community boundaries, or put another way a web of data that can be processed directly and indirectly by machines.  Converting the existing web to a semantic web has turned out to be a major challenge and it remains largely unrealized.

However, as a first step ways are being found to add some level of semantic content to existing data and services.  As an important example,  the OGC has announced the adoption of "Semantic annotations in OGC standards" as an OGC Best Practice.  Annotation of Web Services (or data) compliant to OGC standards such as WMS, WFS and GML refers to attaching meaningful (semantic) descriptions to the service.

The authors give an example from a GML document of an XML schema that defines a class or feature type exploitationsponctualsproduction.  They point out that from the information in this schema you can't tell what the data described by this schema represents.  The name is confusing and the attributes don't provide any additional information. Whose name is the value of name, what does the attribute year refer to, and how is the allowedproduction measured?

An OGC Best Practice means a recommended way to add semantic content to OGC web services based on existing standards.  The authors discuss how to do this using not only OGC standards but also standards from the W3C and ISO/TC211.

OGC standards cover the functional dimension of a web service, but they don’t tell us much about what the data represents in the real world. For example, in the case of WMS or WFS the application knows how to load and visualize the data on a map, but provides very little information to a user in a specific domain such as engineering or construction about how to read the displayed map. 

Put another way, semantic annotations can convert geospatial data into infrastructure data.  Semantic annotations are a way for data providers to connect geospatial data to real world domain models, to provide information about data and web services that is meaningful to an engineer or construction contractor.  Semantic annotations not only make data meaningful to a much broader audience of consumers, but make it discoverable by a much broader audience through standard web search engines.

NRECA TechAdvantage: MultiSpeak, where it is and where it is going

At the NRECA TechAdvantage conference in New Orleans Gary McNaughton, MultiSpeak Techical Coordinator gave a comprehensive overview of where the MultiSpeak interoperability standard is and where it is headed.  If you're not familair with MultiSpeak, it has been developed under the auspices of NRECA and is intended as an industry-wide standard for the exchange of information for electricity distribution utilities and all vertically-integrated segments except power marketing and generation.  It is used by more than 600 electric coops, investor-owned, and municipal utilties in 15 countries and is widely recognized and supported by North American software vendors.

Cybersecurity

In light of President Obama's recent Executice Order on cybersecurity for critical infrastructure, security has become even more critical and Gary's message is that utiltiies need to get very serious about it.  To support this a new security standard for MultiSpeak was released in January 2013.  It goes beyond secure sockets (SSL) and transport layer security (TLS) and implements message-level security.  The standard requires that vendors support four security options, which they have agreed to do.

1. None (for debugging)
2. SSL or TLS
3. Message-level - by message
4. Message-level - by session

It is up to the utility to decide which of these it needs to implement.  There is a Cooperative Research Network (CRN) security policy template that is designed to help utilities to choose the appropriate option.

Use cases

The MultiSpeak team has developed over 300 use cases for specific business processes such as connecting a customer.  They cover business processes for demand response, meter data management, meter prepayment, outage management, and asset management.  So far they do not include GIS-related use cases (MultiSpeak has supported geospatial data types for years.)

Harmonization with CIM

MultiSpeak and the Common Information Model (CIM) have been converging in many ways over the years.  But according to Gary, at this point there is no plan to harmonize the two models in Version 5.  And it is not clear if they will ever be harmonized.

MultiSpeak 3

The big news is that there will be no further bug fixes and enhancements for Version 3, which is the most widely used version of MultiSpeak.  Gary said there was no reason for utilities to stop using Version 3, which is 10 years old now, unless the additional functionality in later versions is required.

MultiSpeak 4

Versions 4.1.5 and 4.1.6 have been released, with enhancements specifically to support the MultiSpeak Regional Demonstration project. It is expected that vendors will begin implementing support for this Version 4.1 in the next few months.  The big advantages of 4.1 are a more consistent data model and support for AMI, DR, MDM, AVL, and work management.  A nice thing about. MultiSpeak's web services implementation is that it means that a utility can run several versions of MultiSpeak concurrently.

MultiSpeak 5

Work has begun on a new version which will include work management enhancements and, most importantly, modernized web services support.  A prerelease will be available in May 2013.

Enterprise Service Bus

Gary made a persuasive case for using Multispeak in all but the simplest implementations with an enterprise service bus because it eliminates some of the problems associated with point to point messaging over TCP/IP.  For example, TCP/IP does not guarantee that a message is received or that several messages will be received in the expected chronological order.

Open source reference implementation of the OGC Catalog Service released

The Open Geospatial Consortium (OGC) has made a determined effort to work with the open source community.Osgeologo   For example, the OGC and the Open Source Geospatial Foundation (OSGeo) signed a Memorandum of Understanding (MOU) a few years ago.  One of the concrete results of the collaboration  between the OGC and the open source geospatial community is that OGC official reference implementations are open source.

A new reference implementation of an OGC standard hs just been announced.  pycsw is an official reference Implementation of the OGC Catalogue Service for the Web (CSW).  CSW provides a standard for publishing and the discovery of geospatial metadata.  As its name suggests, pycsw is written in Python.  It is certified OGC Compliant.

pycsw is an open source project under an MIT license and supports Windows, Linux, and Mac OS X.

New directions for geospatial standards: augmented reality, citizen observatories, and the Internet of Things

I have frequently blogged about some of the important geospatial standards that have been fostered by the Open Geospatial Consortium, most recently about a new Energy and Utilities Domain Working Group (DWG) for the electric, gas, oil and water services industries, the recently adopted GeoSMS standard  which is an extended Short Message Service (SMS) encoding and interface for exchanging location content between devices or applications, and the new Geosynchronization Services (GSS) standard for synchronizing distributed geospatial data stores.

Last week at the GIS in the Rockies annual conference in Denver Carl Reed of the Open Geospatial Consortium gave an overview of some of the directions geospatial standards are evolving.  It was really quite amazing to hear about such a range of application areas where OGC standards are making it possible for the exchange of information between different devices from different vendors.

GeoSMS and GSS

Carl gave several examples of how the GeoSMS simple messaging protocol and GeoSynchronization Services (GSS) are being applied in the real world.  An important example is a mobile use case involving an earthquake in a densely populated area.  A combination of GeoSynchroization Services (GSS), GML, and WFS standards allows first responders in the field to get situation assessments including maps on a variety of hand held devices.

Distributed geospatial updates

Another very relevant use case that is being tested as part of an OGC testbed involves using GeoSynchronization Services (GSS) to enable a large enterprise to receive real-time geospatial updates published by many users and sources, review them, and then insert those updates into web feature services (WFS) operated by many organizations in many different locations.

Integrating indoor and outdoor navigation

Imagine not only getting directions from your house to a shopping mall, but getting directions from your house to the specific counter in the hardware store where plumbing fixures are displayed.  This type of  integrated indoor/outdoor navigation applications are being developed using OGC CityGML and WMS standards. 

Augmented reality standard

Augmented reality involved superimposing imagined structures on the real world.  It is being widely used by architects and designers to visualize what a new building or other structure will look like in situ. At the OGC there is a Augmented Reality Markup Language (ARML) that is under development as a profile of the widely used OGC KML standard developed by Google.

Linking citizen observatories

Recently there has developed an wareness that "it is no longer sufficient to develop passive lists or reports to ‘inform’ citizens of changes in our environment. We need to engage with citizens and ask how they can ‘inform’ us."  (Prof. Jacqueline McGlade, Executive Director, European Environment Agency).  As a result the concept of a citizen observatory has evolved which allows citizens to monitor and report on environmental issues such as biodiversity loss, environmental degradation, and the impact of climate change. 

The Citizen Observatory Web (CobWEB) is intended to link these citizen observatories, providing a standards-based way of linking to sensors and mobile devices and enabling crowd-sourced, geospatially-based information about our environment. 

It will rely on the OGC Sensor Web Enablement (SWE) standard, that provides interoperability interfaces and metadata encodings for real time integration of heterogeneous sensor webs involving devices such as flood gauges, air pollution monitors, stress gauges on bridges, mobile heart monitors, Webcams, and robots as well as space and airborne earth imaging devices into the information infrastructure.

Internet of things

It is projected that the Internet of things, the interconnection over the internet of everyday intelligent objects, will include15 billion devices ranging from pollution monitors to refrigerators by 2015.  It will involve indoor/outdoor location, sensor webs, building information models (BIM), intelligent devices comprising the smart grid, devices for environmental monitoring, and household appliances. 

It will require standards such as the Sensor Web Enablement (SWE) standard to allow all these devices, from many different vendors, to intercommunicate.  Carl gave two examples, one involving tracing fresh meat, absolutely critical in tracing meat as a source of infection, and another application used to monitor debris flow in Taiwan.

Importance of location standards

The most important things that I came away from Carl's talk with are first the wide range of domains where geospatial standards are being applied and, secondly, the realization that standards are becoming even more important than before, because we are on the brink of interconnecting billions of devices from hundreds of thousands of different vendors in many different countries over the internet.  Without standards it will be impossible to turn this data into actionable information. 

The other aspect of this new world of interconnected devices is that location is also becoming more important because, thanks to inexpensive GPS chips and the development of new GNSS systems by Europe and China and others, sensors these days can report their location.

New XML-based standard for exchanging information about water sources

The WaterML 2.0 Part 1: Time Series Encoding Standard has been adopted by the Open Geospatial Consortium as an OGC standard.  OGC WaterML 2.0 is XML-based and supports encoding and exchanging hydrological data describing the state and location of water resources, both surface water and ground water. 

WaterML 2.0 is designed to provide a standard for national reporting of water resources as well as for operational echange of data on the state of infrastructure such as water supply systems.  Much of the work was done as part of a joint program between the Australian Commonwealth Scientific and Industrial Research Organization (CSIRO) and the Australian Bureau of Meteorology (BoM). 

The standard is important in Australia because it will be used in the development of the second version of the Water Data Transfer Format (WDTF) which mandates the techniques to be used by the Australian water industry to deliver water data measurements to the Bureau of Meteorology as required under the Water Regulations of 2008. 

The Australian effort was supported by the Consortium of Universities for the Advancement of Hydrologic Science Hydrologic Information System (CUAHSI), which developed WaterML 1.0, and other organizations in the Netherlands, Germany, Canada, France, and the U.S including NOAA and USGS.

Burlington Hydro's real-time smart grid for operations and analytics: Managing big data real-time

At the Oracle Spatial User Conference in Washington DC this week, Don Guatto, COO and VP Engineering at Burlington Hydro and Geoff Cameron, EVP Intelligent Networks at AGSI gave a rivteing presentation on Burlington Hydro's (BHI) smart grid implementaton for the utility's GridSmartCity initiatives.

BHI is a utility near Toronto with 65000 meters, 32 substations and 93 employees.  The GridSmartCity initiatives include

• self healing networks - this initiative has reduced average outage times in BHI's service area to 20 seconds per year
• smart meters - Burlington is in Ontario which has deployed smart meters and time-of-use pricing province wide
• distributed generation - Ontario has a Feed-in-Tariff program which has led to the deployment of over 7000 micro-generation projects, mostly solar
• electric vehicle (EV) charging stations
• factory ride-through systems - enabling factories to continue functioning through outages
• battery-based electric storage

Smart grid management system for operations and analytics

An independent consultant hired by BHI made a recommendation with far reaching implications for BHI. "Two of the main factors affecting GIS today are the move towards using internet technology to be able  to serve GIS data to a much broader user base and efforts by the Open Geospatial Consortium (OGC) to promote interoperability.  BHI needs to implement a new GIS that will fit its technology vision and provide the benefits of today's GIS to BHI operations."

Based on this recommendation BHI determined that the foundation for their smart grid management system would be an open architecture based on open standards.  This would allow them to integrate their SCADA system, 65 000 smart meters reporting power use every 15 minutes, the supporting automated meter infrastructure (AMI), intelligent devices, power line sensors, a customer information system (CIS), their ERP system, engineering analysis, and other systems.  It would utilize bidirectional communications to both receive information from and control smart devices. The system would have handle much larger data volumes than BHI had ever experienced before.  For example, some devices were reporting 60 times per second.  And perhaps the most challenging requirement is that it also had to operate real-time, which as Geoff emphasized, is beyond the capability of today's traditional enterprise GIS systems.

Implementation

AGSI's solution uses their Go360 products which provide a scalable, geospatially-enabled solution built on an open service oriented web architecture. It uses Oracle Spatial for a shared, geospatia data repository and AutoCAD Map 3D for design and geospatial data maintenance.  A web browser, either on the desktop or on a mobile device, is all that's required to access tools for asset maintenance, cable locate, asset managemet, operations, financials and drill downs, automating network pinning and work protection tagging, SCADA, schematic views, automated CAIDI, SAIDI and SAIFI reporting, an outage management system (OMS), mobile workforce automation, automated as-built management, real-time asset monitoring and analytics, and an executive dashboard.

One of the most impressive Go360 applications that Geoff demonstrated was a transformer status monitoring dashboard that not only showed a map with transformer loading in real- in the form of a heat map, but could also report on historical loading and even estimate, based on the history of oveloading on a particuler transformer, how much the lifetime of the transformer had been shortened as a result of the overloading.  It also allowed the operator to reconfigure the grid in real time to reduce the load on overlloaded transformers and redistribute it to others with available capacity.

What impressed me most was to see how an open IT architecture based on open standards together with industry standard off-the-shelf components, a bidirectional communications network and real-time smart devices and sensors has enabled BHI to develop a state of the art real-time geospatially-enabled smart grid operations and management system that integrates with their existing enterprise systems and provides a common point of access to all their operational data. 

BHI has plans to contnue to develop smart grud applications in the future, but I think that they have built a strong IT foundation for their future smart grid development.

Standards-based intelligent modeling of urban multi-utility networks

In a paper Semantic 3D Modeling of Multi-utility networks in Cities presented at GSDI 2012 in Quebec City, Thomas Becker; Claus Nagel and Thomas H. Kolbe have outlined how they are translating a vision of the city as an interactive system comprised of functional components, utility networks connecting components, and interdependencies between utility networks into standards-based intelligent models that can be used to analyze urban environments for a variety of purposes including

• Risk- and disaster management
• Energy consumption
• Carbon balancing
• City life-cycle management

The see ciities as being very complex, diverse and highly interrelated, that is a system of systems, buildings, utility infrastructure, transportation infrastructure, health and social, and so on.  Many different players are involved in designing, building, running and maintaining a city, and each needs a different view of the city. The advantage of having a model of the city is that it provides a simplified representation of the city that provided "ontological and semantic clarity".

Thomas Kolbe and co-workers are the developers of the CityGML standard that has been adopted by the Open Geospatial Consortium.  There are several Application Domain Extensions (ADEs) that have been developed to extend CityGML to other domains.  In 2010 a basic extension UtilityNetworksADE was proposed for city utility networks, and at Quebec it was announced that this was being futher extended.

The intention is to model infrastructure networks both as a 3D topographic, topological and and functional network. In other words, this is not simply a compilation of network as-builts, but includes the function of each component and its relationships to other components.  It knows the difference between different network topologies; water, electric power, steam, wastewater, and communications.  It also knows the interdependencies between different networks.  For example, in case of a flood, from the digital terrain model, it is possible to determine which electric power substations are flooded, from the electric power network topology, which feeders are no longer operative, from the relationships between the electric power and water networks, which pumping stations are no longer operating, and ultimately who is without water and power.

Functionally, every utility network ias modeled by three types of components

1. Distribution elements (conductors, pipes, fiber cables) − For distribution of electric power, gas, and messages.
2. Protection devices (fuses, relays, breakers, and ducts) − Does not actually carry electric current, water, gas, or messages, but supports the distribution system.
3. Functional components (man-holes, water treatment plants, substations, pumping stations and switches) − Needed for linkage, maintenance, and measurement

I like to think of these as connected elements where the connectivity between elements allows traces, from a customer experiencing an outage to a failed device such as a transformer or substation or from a failed device to all affected customers, structural elements, which support the connected elements such as poles or ducts, and devices which generate, transform, switch, and measure.

To better understand this implication of this approach,Kolbe et al have compiled a model database of 1.313,821 infrastructure elements including interdependencies between different networks. Addendum According the Thomas Kolbe (personal communication), for each utility company in Berlin involved in the SIMKAS-3D project (Vattenfall - electricity and district heating, GASAG - gas, and Berlin Water - drinking water and waste water), his team implemented a converter from the respective utility's dataset to the CityGML ADE utility model using Safe Software's FME.

Geospatial World Forum: Dutch 3D standard integrates CityGML/GeoBIM and large scale geography

At the Geospatial World Forum in Amsterdam, Jantien Stoter of the Delft Univeristy of Technology and an advisor to Geonovum, gave an overview of the ongoing work on the Dutch 3D standard, which is being piloted by about 100 organizations in the Netherlands.  Geonovum is the National Spatial Data Infrastructure (NSDI) executive committee in the Netherlands. Its goal is to provide better access to geo-information in the public sector and is responisble for developing and maintaining the standards for geospatial information that are necessary to achieve this goal.

The Dutch 3D standard integrates CityGML and the Dutch IMGeo standard. 

• CityGML is an Open Geospatial Consortium (OGC) standard for 3D information (geometry and semantics) about cities, external to buildings. GeoBIM is a CityGML Application Domain Extension (ADE) intended to extend CityGML to include detailed, semantic information about the inside of buildings.  GeoBIM was initiated in 2009. The last official update was completed in 2010.
• IMGeo is the Dutch standard for the exchange of large scale geography such as roads, tunnels, water bodies, land use, and so on.  IMGeo is intended to be a scale-independent object model providing a uniform nationwide coverage.  IMGeo 2.0 contains a mandatory part, the Basisregistratie Grootschalige Topografie (BGT). which has legal status in the Netherlands, plus additonal data. IMGeo is 2D, but is extensible to 2.5D and ultimately to 3D.  Like CityGML, IMGeo recognizes levels of detail (LOD).

There are six work groups involved in the Dutch 3D standard effort. For example, one is responsible for compiling killer applications using the 3D standard.

Relationship to IFC

One of the work groups could have broad implications, potentially beyond the borders of the Netherlands, This work group is responsible for aligning the CityGML ADE GeoBIM standard and the Industry Foundation Classes (IFC) BIM standard maintained by BuildingSmart.  IFC is intended to enable the exchange of BIM data between applications developed by different software vendors in a vendor-neutral way.  The objective of the work group is to define a standard, semantically meaningful mapping between IFC and GeoBIM.

Geospatial World Forum: Land administration in Germany to include 3D cadastre by 2013

At the Geospatial World Forum in Amsterdam, Peter Creuzer, Director State Survey of Lower Saxony, gave a very interesting introduction to land management in Germany and some of the things that are on the horizon. 

Germany has about 82 million citizens, or about 229 people per km², and the population has been declining since 2003. It has about 64.5 million parcels (land properties) valued at about €9 trillion.  About 48% of households own land.  The land administration system handles about a million transactions per year. 

Germany has the same problem that other jurisdictions have, urbanization, but it has limited land resources.  It loses about 77 hectares/day to urban sprawl.

Land administration in Germany is comprised of two parts, a land register and the cadastre.The land register includes information about owners, land use, encumbrances, and other legal obligations.  The cadastre includes the land parcel geometry, owneship, buildings and public restrictions.  The objectives of the land administration system are security of tenure, some revenue for the government, and support for a transparent planning process.

There are some interesting future goals, one of the most important of which is enabling sustainable development.  This includes land consolidation and revitalization of older cities.

Another major initiative is revision of the land taxation system which is expected to be completed by 2014.  The alternatives that are being considered are taxation based on market value (which is what is used in Ontario), a fixed tax, and a combined system.

There is also a new data model underway based on ISO and Open Geospatial Consortium (OGC) standards referred to by the abbreviation AAA, which stands for AFIS (uniform geodetic reference), ALKIS (uniform cadastre), and ATKIS (uniform topography).  By 2013 the data model will include support for a 3D cadastre and will include the CityGML standard.  It will be accessible via OGC web services; WMS, WFS, and others.

Germany is also working on being compliant with the EU INSPIRE directive.  There is a pilot in progess involving the Netherlands, Lower Saxony, and North Rhine Westphalia.