USGIF GotGeoint Blog USGIF promotes geospatial intelligence tradecraft and a stronger community of interest between government, industry, academia, professional organizations and individuals focused on the development and application of geospatial intelligence to address national security objectives.
The world is changing. Microsoft has been doing things in the open source world for some time, but this last week it has gotten wild.
Microsoft has just kicked off launch activities for SQL Server 2016 with general availability later this year. Spatial data has been supported in SQL Server since 2012. According to a Microsoft blog post this is the most significant release of SQL Server ever. It includes some very cool technology: in-memory database support with dramatic performance increases and advanced analytics based on R that enables customers to do real-time predictive analytics on both operational and analytic data among other things. Microsoft introduced Polybase, a SQL Server connector to Hadoop in 2015. Microsoft has now incorporated that functionality into on-premises SQL Server 2016.
That's all pretty cool, but the biggest surprise is that Microsoft has announced plans to bring SQL Server to Linux. The core relational database capabilities are being previewed right now. SQL Server 2016 will be available on Linux in mid-2017.
Microsoft has just released the source code to an open-source operating system, based on Debian GNU/Linux, that runs on network switches. The software is called SONiC (Software for Open Networking in the Cloud). SONiC builds upon Microsoft's Linux-based Azure Cloud Switch (ACS) operating system.
Earlier this week Github published a graph showing the relative popularity of programming languages among GitHub development project repositories, both private and public, for 2008 to 2015.
But a couple of things really surprised me. Of the three P's, PHP and Python have remained in the top 5 for the entire period, but Perl has fallen off the chart. CSS has risen dramatically. But the most surprising thing for me was the incredible rise of Java from 7th to 2nd since 2008. Not long ago it seemed that only the big system houses like HP, IBM and Oracle were using Java. One reason for its spectacular rise may be that it is suited to building big distributed internet systems (the Hadoop framework is implemented in Java). It is also used for Android which runs more than 50% of the world's smart phones. Something which may be related is the recent spurt in C# (Microsoft's version of Java). In the geospatial community the rise in Java popularity has been paralleled by the rise in the popularity of GeoServer and GeoTools both of which were developed using Java.
Finally, Apple's Swift development language for iOS, which Apple says it will open source, is off this graph, but apparently has already risen to 18th. It's rapid rise perhaps accounts partially for the drop in Objective-C, which has like C++ stayed in the top 10 - until very recently.
At DistribuTECH 2015, Raiford Smith of CPS Energy and Jason P. Handley of Duke Energy, presented their perspectives on the smart grid; what is motivating it in terms of business and technology drivers, a roadmap for implementing it at their utilities, and the benefits that are expected from it for customers and for utilities. They also outlined a smart grid architecture based on open standards to enable seamless interoperability that enables distributed as opposed to centralized intelligence. Some of the advantages of a distributed architecture are scalability, reduced latency and implementing security at the grid edge instead of via the central control application.
Megatrends driving smart grid
Raiford Smith sees four major megatrends that are major motivating forces behind the smart grid. Moore's Law means there are intelligent devices for power networks with greater capabilities and at less cost. Metcalfe's Law means more interconnections and greater interoperability. Big data analytics means the ability to extract more meaningful information and insights from rapidly increasing volumes of data coming from thousands and even millions of intelligent devices. Distributed energy generation (DER) means more complicated power management - balancing intermittent generation and new load profiles from an increasing number of new electronic devices.
From a business perspective a major benefit is greater customer choice. In the future the customer will be able to not only manage his/her consumption of power, but also its generation. With rooftop solar PV and batteries the customer may elect to not even be on the grid, but to create his/her own microgrid. But it also means the utility business model will have to evolve from what it has been for the past 100 years. New York is one of the jurisdictions that is already changing its regulatory framework to enable utilities to move to a new business model.
As an aside, at this year's DistribuTECH if there was one technology that seemed to be everywhere and on almost every utility's radar, it is microgrids. Duke Energy is even playing with the idea of offering microgrids as a service.
Technology roadmap for the smart grid
From Raiford Smith's perspective the technology roadmap for the smart grid involves the deployment of increasing numbers of intelligent electronic devices for sensing and for control. The challenge is federating the data from all of these devices, extracting information from it, and dispatching the information to the right control devices. From an architectural perspective this drives the need for a field message bus which enables interoperability between different devices from different vendors. It also requires a common semantic model, such as the Common Information Model (CIM), adding security at the edge of the grid in addition to the central control room, and analytics to extract information from the huge volume of data collected from the sensors.
To test different smart grid configurations, CPS Energy is assembling a test facility for a three year smart grid testing program. It will have 30,000 customers, 15 circuits, solar generation, smart inverters, battery storage and the ability to disconnect from the grid to form a microgrid.
Benefits of the smart grid
Raiford expects major benefits for customers and for the utility from implementing a smart grid. For customers, perhaps the biggest benefit is that the smart grid avoids divergence of utility services and customers needs. Sometimes referred to as disaggregation, in this context it means a 3rd party coming between a utility and its customers. Historical examples are Microsoft Hohm and Google Powermeter which were perceived as threats because utilities found the idea of a Microsoft or Google insinuating itself between the utility and its customers unattractive. Opower is an example of a different approach that is much more attractive to utilities. Instead of doing an end-run around the utility, Opower focussed on the utility as their direct customer. Services using Opower's solutions is then offered by the utility to its customers.
Secondly, smart grid provides a flexible foundation for providing new services to customers (which also creates new sources of revenue for utilities). Raiford suggested some examples including electric vehicles and charging, premium (high quality) power, premium reliability, and asset control (for ex, inverters and batteries) and advanced demand response (the utility would provide these as a service to customers, rather than customers buying these devices from a 3rd party). The result is greater customer satisfaction and improved brand recognition as CPS Energy is perceived as a leader in providing new and improved services to customers.
The benefits that Raiford sees for utilities are equally important. They include improved operational metrics (SAIDI, SAIFI, asset utilization), better financial metrics (O&M spend, revenue generation), environmental benefits, improved safety, better trained and skilled staff, and more reliable risk modeling (better predictions of revenue, costs, customer satisfaction, and asset condition).
Probably the greatest challenge Raiford sees is managing organizational change, because the smart grid will mean that just about everything will change including the utility business model and most aspects of how we design, build, maintain and operate the grid.
Drivers for utility industry change
Jason Handley, of the Emerging Technology Office at Duke Energy, reviewed some of the major drivers for industry change. Many applications currently used by power utilities are proprietary, with the result that the utility has many application silos that don't interoperate. The rapid adoption of DERs is requiring utilities to move toward faster response times, reduced costs, better safety, and improved reliability. Dynamic load management and low voltage power electronics will mean greater adoption of rooftop PVs and other DERs. Increasingly utilities will invest in standards-based, modular systems for hardware, multi-function devices, and a field message bus for software that will enable interoperability. From a business perspective broader interoperability facilitates more competition which lowers costs, encourages innovation and improves reliability.
Other important drivers that Jason sees that are impacting utilities include demand response, electric vehicles, in-premise automation, cybersecurity threats, aging infrastructure, big data complexity, and avoiding stranded assets. The smart grid is requiring utilities to change how they do things. Utilities realize they have to be more proactive in their operations, rather than waiting for something to happen and then reacting to it. Situational awareness has become a critical capability for utilities in enabling utilities to be more proactive. It is made possible by having a variety of sensors in the field that together can present a snapshot of the status of the grid. The key functionality required to enable this to happen is seamless interoperability.
Centralized or distributed intelligence
As utilities implement thousands and even millions of smart devices in the field, a centralized architecture runs into scalability and latency problems. Duke's solution is an architecture with distributed as opposed to centralized intelligence. Duke sees this as comprised of layers so that with this architecture, not all data needs to go to the central control application. Some can be handled at lower levels. A self healing network is an example where a problem can be handled locally without the central control application knowing anything about it. Distributed intelligence also enables fast edge decisions that can be made without waiting for the central control application. For example, an advantage that cannot be underestimated with this architecture is that it enables security at the edge of the grid, not just via the central control application. Based on this concept Duke has defined a Distributed Intelligence Platform (DIP) Reference Architecture designed to take advantage of the tremendous intelligence that is out in the field in addition to the intelligence in the control centre.
Duke Energy, CPS Energy and 25 vendors, called the Coalition of the Willing (COW) have just embarked on an implementation of this architecture that supports a microgrid. The smart grid requires exchanging data between different devices from different manufacturers in the field. Traditional utility technologies are very often vendor silos utilizing proprietary hardware, telecommunications and software platforms. The goal of the “Coalition of the Willing" (COW) is to demonstrate that data and control commands can be shared across multiple vendor platforms (typically proprietary) to achieve interoperability with lower costs and faster response times. A key part of the demonstration is an open standard field message bus implemented as an open source project. The Smart Grid Interoperability Panel (SGIP) has created an OpenFMB working group to support this effort.
At DistribuTECH 2015 the Smart Grid Interoperability Panel (SGIP) has announced the launch of the OpenFMB Project, a special SGIP working group designed to leverage existing standards and structured processes to create a new paradigm for true interoperability and peer-to-peer communication across vendors' architecture that will increase business intelligence and operational efficiencies while allowing for secure and reliable communication and fast decision-making in the field.
The smart grid requires exchanging data between different devices from different manufacturers in the field. Traditional utility technologies are very often vendor silos utilizing proprietary hardware, telecommunications and software platforms. Duke Energy and six vendor companies formed a group called the “Coalition of the Willing” (COW) whose goal was to demonstrate that data and control commands can be shared across multiple vendor platforms (typically proprietary) to achieve interoperability with lower costs and faster response times. A key part of the demonstration was an open field message bus implemented as an open source project.
Duke Energy has announced the next phase of their interoperability project (COW II) with an expanded group of 25 vendor partners and a utility partner CPS Energy. The second phase will include the operation of a microgrid system which will integrate distributed renewable resources such as solar PV and battery storage with a field message bus-based distributed intelligence platform with wireless communications to devices.
For COW II Duke Energy defined a Distributed Intelligence Platform (DIP) Reference Architecture. According to Duke, the reference architecture is intended to document Duke Energy’s technology roadmap for interoperability using open standards-based distributed information systems. Part of the reference architecture is a field message bus. According to Duke the Field Message Bus (FMB) is intended to be an open standard-based, common logical publish/ subscribe interface that connects multiple disparate grid devices, telecom networks, and information systems. It is the key technology enabler to demonstrate the benefits of the distributed architecture by facilitating interoperability between multiple different vendor’s OT, IT, and telecom systems.
Starting with the Distributed Intelligence Platform reference architecture from the COW project, SGIP has defined the OpenFMB concept as the way distributed applications and open interfaces can enable interoperable peer-to-peer data exchanges between power systems devices. The OpenFMB framework provides a specification for power systems field devices to leverage a non-proprietary and standards-based reference architecture platform, which consists of internet protocol (IP) networking, Internet of Things (IoT) messaging protocols, and standardized common semantic models (such as CIM), to enable communications and peer-to-peer information exchange between devices on the electric grid.
The smart grid requires exchanging data between different devices from many manufacturers in the field. Traditional utility technologies are very often vendor silos utilizing proprietary hardware, telecommunications and software platforms that communicate to a centralized hub. Last year at Distributech 2014 a group of Duke Energy and six companies called the “Coalition of the Willing” (COW) – Accenture, Alstom Grid, Ambient Corporation, Echelon, S&C, and Verizon, demonstrated interoperability between their products. The goal was to demonstrate that data and control commands can be shared across multiple vendor platforms (typically proprietary) to achieve interoperability with lower costs and faster response times.
At Distributech 2014 Duke energy also described the challenges Duke has encountered in cleaning, merging and managing operational data, combining it with other types of data including social media, and developing analytical tools to extract meaningful information. Duke also described an innovative approach for involving vendors in the development of their smart grid platform and applications, which provided the basis for vendor involvement in COW.
COW 1 Volt/Var Optimization
The first phase of this project was to convert a Volt/Var Optimization (VVO) function to run on a distributed platform. It used a “communications node” and a standards-based messaging architecture to enable peer-to-peer communications to integrate data from different vendors' systems. Each of the COW participants exposed some data via a standards-based interface (Field Message Bus). The application read two residential meter voltages supplied by Echelon meters and brought the data back via Power Line Carrier (PLC) to the Ambient communication node. If an under voltage condition was detected, the application sent a close command to the control via DNP3 through peer to peer messaging over the Verizon Wireless network. The capacitor control closed the capacitor bank to alleviate the under voltage condition on the meters. Finally, the Alstom DMS was updated with the new state of the capacitor.
Open standards, open source
The COW 1 project involved open source hardware (Raspberry Pi), open source software and open standards. According to Duke the Field Message Bus (FMB) is intended to be an open standard-based, common logical publish/ subscribe (pub/sub) interface that connects multiple disparate grid devices, telecom networks, and information systems. It is the key technology enabler to demonstrate the benefits of the distributed architecture by facilitating interoperability between multiple different vendor’s OT, IT, and telecom systems. It uses open-source software to translate data to a common publish and subscribe messaging interface using the IEC Common Information Model (CIM) data model. Other open standards that are being used in this project include industry-standard protocols as DNP3 and Modbus and messaging protocols like MQTT (for lightweight applications) and AMQP (for heavy duty messaging).
COW 2 Islandable microgrid
Last June Duke Energy announced the next phase of their interoperability project with an expanded groups of vendor partners. The second phase will include the operation of a microgrid system which will integrate distributed renewable resources such as solar PV and battery storage with a field message bus-based distributed intelligence platform with wireless communications to devices.
The new project will support another open standard. The Object Management Group's (OMG) Data Distribution Service (DDS) for Real-time Systems provides a secure publish-subscribe messaging protocol. Many real-time applications involve publishing “data” which is then available to remote applications that are interested in it. In a utility setting this communications model needs to scale to thousands of publishers and subscribers in a robust manner.
A data model based on the CIM standard will be implemented into an open field message bus to support standardized object model representations. The use case for this demonstration project will be an islandable microgrid which Duke will implement and operate at its smart grid test facility.
FOSS4G 2014 runs from Sept 8th-12th in Portland, Oregon this year. The conference brings to together developers and others in geospatial open source. This is an opportunity to participate in the rapidly growing open source geospatial community. Last year's FOSS4G 2013 in Nottingham attracted 28 workshops, 180 Presentations and 833 delegates.
OpenStack is an open source cloud computing platform. It is free and open-source software released under the terms of the Apache License and is managed by the OpenStack Foundation, a non-profit corporate entity.
Oracle, now the world's second largest software company ahead of IBM, has become a Corporate Sponsor of the OpenStack Foundation and says it s planning to integrate OpenStack cloud management components into Oracle Solaris, Oracle Linux, Oracle VM, Oracle Virtual Compute Appliance, Oracle Infrastructure as a Service, Oracle’s ZS3 Series, Axiom storage systems and StorageTek tape systems.
Other organizations supporting OpenStack include AT&T, Ubuntu (Canonical), HP, IBM, Redhat, Suse, Cisco, Huawei, Intel, VMWare, Yahoo, and EMC.
About a year ago the Obama administration announced a Big Data Research and Development Initiative to focus on extract knowledge and insights from large and complex collections of digital data. It included an initial investment of more than $200 million to improve tools and techniques for accessing, organizing, and extracting knowledge from the huge volumes of data that are available. The list of ongoing Federal government programs that relate to big data is extensive.
Yesterday at a White House Office of Science and Technology Policy (OSTP) event, the Gordon and Betty Moore Foundation and the Alfred P. Sloan Foundation announced a $37.8M funding commitment to develop new data science facilities at the University of California Berkeley (UC Berkeley), the University of Washington (UW) and New York University (NYU).
It is fantastic that data and its management and analysis at long last has been recognized as serious enough to be studied in an academic environment. It will be interesting to see how geospatial data fits into this.
For a compelling open source-oriented perspective on this, take a look at Fernando Perez's blog where he makes the case that open source is a key ingredient in this effort.
In a very recent post from two analysts, Peter Goldmacher and Joe del Callar, from Cowen and Co who attended Hadoop World, Cloudera's annual conference, argue that data management is at a discontinuity, where a new set of open source, big data technologies are set to replace legacy proprietary RDBMS-based technologies.
Recent and repeated top line shortfalls from ORCL and TDC have been blamed on the macro. While we believe this to be the case, we also believe these macro issues are accelerating a technology transition from legacy products to alternative data management systems as users are increasingly evaluating products like Hadoop that offer compelling functionality at materially lower price points. While a user's motivation for an initial Hadoop deployment may be cost, end users we talked to spoke of incremental outcomes around better flexibility and functionality that enabled broader value-added use cases to support the business.
The legacy providers of data management systems have all fallen on hard times over the last year or two, and while many are quick to dismiss legacy vendor revenue shortfalls to macro economic issues, we argue that these macroeconomic issues are actually accelerating a technology transition from legacy products to alternative data management systems like Hadoop and NoSQL that typically sell for dimes on the dollar.
We believe these macro issues are real, and rather than just causing delays in big deals for the legacy vendors, enterprises are struggling to control costs and are increasingly looking at lower cost solutions as alternatives to traditional products. What we are gleaning from multiple conversations with users of these new technologies is that regardless of the initial reason they experiment with these new technologies, the outcome is 1) The realization that not only are these products cheaper, but they are more flexible and better suited to many legacy workloads and 2) They offer end users the optionality to expand project scope beyond the constraints associated with legacy product, whether those constraints are cost or capabilities.
What we think the data center of the future looks like is really a core of these commodity machines [i.e., commodity LInux servers not powered by Oracle] and a collection of these purpose-built machines [like Oracle's Exadata that serve higher-end requirements].
and concludes that it does suggest, like Ellison himself argues, that we're likely to see the legacy vendors take an increasingly peripheral role in an age of Twitter.
I blogged about an insightful article about open source business models posted by Mike Olson, Chief Strategy Officer at Cloudera.
Mike Olson makes the case that there were very few successful large
stand-alone open source vendors. He counts Red Hat as the only one that
has not been acquired or has not failed. The problem Mike Olson sees with
open source products is that as they becme large and generate signfiicant revenue, it is easy for potential competitors to create competive products because the source code is available.
Android is an interesting case that seems to support Mike Olson's hypothesis. According to a recent article on ars technica, Android has always been a combination of open and closed source (proprietary) components. Originally, the closed source code comprised just clients for Google's online services, Gmail, Maps, Talk, and YouTube. Google kept these apps closed source, but built the rest of Android as an open source project (AOSP).
But as Android increased its smart phone OS market share to about 80% at present, new closed source apps were introduced to replace existing open source (AOSP) modules. Apparently, as soon as the proprietary version was launched, all work on the AOSP version was stopped basically turning it into abandonware. According to the article when "Google rebrands an app or releases a new piece of Android onto the Play Store, it's a sign that the source has been closed and the AOSP version is dead." The result seems to be to make more and more of the Android you experience on your smart phone "Google" proprietary. It would seem that this would make it increasingly difficult for potential competitors to create competitive products from the AOSP source code, which supports Mike Olson's argument that when open source products get to a certain size they need to adopt a mixed proprietary/open source business model to remain viable from a business perspective.