Between the Poles

Development of an open spatially-enabled BIM query language

Introduction

Building Information Models are comprehensive digital representations of buildings which include 3D geometry and associated data. They are widely used for designing, engineering, and constructing buildings (vertical BIM) and infrastructure (horizontal BIM). Owners are beginning to see value in BIM models and data for operating buildings. A BIM model includes not just the 3D geometry of all building elements and the enclosed spaces, but also semantic information representing both the type of the elements, the relationships between them (defined by the data model) and data associated with the geometric elements.  With BIM models being mandated in countries like the UK, Finland and Singapore, and in France and Russia in the near future,  and the increasing adoption of BIM by the private sector in the U.S. and other countries, the volume of BIM data is growing rapidly.  By merging models from different disciplines into common virtual models, the amount of information stored in such multidomain repositories quickly becomes large and complex even for average projects.  Centralized or distributed repositories enable the creation, integration and maintenance of large quantities of data from various stakeholders involved in the planning and building process and the wide range of specialized software tools used by these stakeholders.  Many ways of creating, manipulating and visualizing BIM and CAD models have been developed including CAD, Solibri, Navisworks, GTPPM, mvdXML, PQML, EQL, SQL, XSLT/XQuery, SPARQL, Gremlin, Linq, SDAI, and generic programming languages.  However, the development of a general, intuitive query language for BIM models including geometric and related information remains a challenge.

The information needs of individual stakeholders in different parts of the construction process are very different. For example, a structural engineer does not require all the information available in the overall building model such as the detailed specifications of a suspended ceiling. This makes the extraction of partial model subsets or domain specific views on large models necessary.  In addition regulations, energy performance modeling, and building maintenance require finding specific facilities among the building assets. 

In the last few years research has focussed on the development of an open, domain-specific query language for BIM.   Most of the this research has focussed on querying, updating and deleting data stored in Industry Foundation Classes (IFC) models.  The Industry Foundation Classes (IFC) is a vendor neutral and open model that defines more than 650 entities and a few thousand attributes.  Most industry software BIM tools  are able to export IFC models, though there is typically some information loss. The bimserver.org platform is an open repository of IFC models and already provides a Java API to extract partial building information models from a repository.

Open query languages

Many existing database applications use the Structured Query Language (SQL), an ANSI/ISO standard with spatial extensions standardized by the Open Geospatial Consortium (OGC), as the standard language. SQL makes it possible to create, read, update and delete records in a Relational Database Management System (RDBMS).  But the applicability of SQL in a BIM context is limited.  SQL is based on operations on data stored in tables.  Application to IFC, as a BIM example, would require storing 600 entities along with their thousands of attributes in individual tables.  Severe performance and scalability issues are associated with using SQL for BIM.  In addition the complexity of IFC-based queries would rapidly become too complex.

The W3C Data Activity has defined a collection of technologies comprising a “web of data” where information is easily shared and reused across applications.  Key parts of this technology stack are the RDF (Resource Description Framework) data model, the OWL Web Ontology Language and the SPARQL protocol and RDF query language which has been standardized by the W3C.  The Resource Description Framework (RDF) is a directed, labeled graph data format for representing information on the Web.  RDF and SPARQL  enables interoperability among distributed heterogeneous information systems making possible queries that span several repositories and link information from different sources.  Specifically in a BIM context it enables ad hoc queries of sub models that reside in repositories maintained by the different stakeholders in a building project.

Spatial queries

BIM data models can be simple or complex.  A complex data model includes logical relationships between building elements, such as windows and walls, walls and storeys, and so on.  These relationships are created at design time.  A simple model may not include any explicit relationships. For example, there may not be any way to determine which windows are associated with a wall except visually.  Spatial queries enables relationships to be determined after the creation of the BIM model.  For example, a spatial query can find all the windows in a wall, all the walls of a storey, or all the HVAC and plumbing facilities in a storey.

The efficient implementation of spatial BIM queries requires 3D spatial indexing such as octree and multi-dimensional r-tree indexes and algorithms.  It has been known for a long time that spatial indexing is a key technology enabling efficient GIS systems.  Very early on in the development of spatial indexes it was realized that these are domain specific.  Several GIS systems support an extended spatial SQL based on quadtree or r-tree indexes.  These systems evolved historically from a 2D geometric representation approach.  However, the functionality for processing 3D geometry in many GIS systems is restricted and the performance of the available tools is unsuitable for complex building data sets.  Therefore, it has been argued that a specialized tool is required to handle queries against elaborated BIM models containing vast numbers of elements and detailed geometry representations.  Such a query language enables the spatial analysis of BIM models and the extraction of partial models that fulfill certain spatial constraints. Among other features, it includes operators that reflect the topological relationships between 3D spatial objects including "within", "contain", "touch", "overlap", "disjoint" and "equal" in 3D space.  Both octrees and multi-dimensional r-tree indexes have been investigated as a basis for an efficient BIM query language.

Open BIM query language

A BIM query language, dubbed BIMQL, has been proposed as an extendable, open, domain specific query language for BIM models. It can be used to select and update partial aspects in building information models.  A limited implementation of the language has been built on top of the bimserver.org platform.  It is not complete, in particular it does not support spatial queries, but has provided a useful vantage point for future research, discussion and development on the development for end-user interfaces to complex BIM models as well as the composition of service-oriented architectures.

Unlike traditional data modeling approaches which are limited by the scope of their underlying schemas, RDF and related technologies can provide an open and common environment for sharing, integrating and linking data from different domains and data sources.  In comparison with domain-specific query languages, SPARQL is especially applicable for scenarios when data from multiple sources are needed.  Some important scenarios include

• All building objects should be classified according to the local building code
• Where are the locations of companies which produce the materials used in walls placed in the hall ?

The interesting thing about these scenarios is that they require the BIM geometry and data plus other external data. This is the type of query that the W3C semantic web technologies are designed to support.

In the AEC domain, a typical use case is data integrity validation against BIM standards or building code requirements.  A Dutch example is the Dutch BIM standard requires that all building elements should be associated with appropriate building storey.  Provided that the building model is represented in the standardized ifcOWL, it is possible to check this spatial containment relationship using off-the-shelf SPARQL implementation.

Currently SPARQL does not contain the vocabularies or functions required by many use cases in the construction domain.  Recently researchers have proposed extending SPARQL with a set of domain-specific functions for the construction world. There are precedents for this strategy.  In 2011 the Open Geospatial Consortium (OGC) released GeoSPARQL, a geographic query language for RDF data, which extended SPARQL with geospatial functions that enabled spatial queries.  The advantages of this approach is that not only are the extended AEC functions compatible with existing SPARQL environments, but they can also be used to query building data combined with data from other fields.

But standard SPARQL can only crawl the data graph according to the structure of the BIM data model.  There are many use cases that depend on geometric relationships that cannot be handled by off-the-shelf SPARQL. To address this SPARQL has been extended to include spatial operators "touches", "intersects", "disjoints", and "contains" and well as other domain-specific functions.

An example of a simple spatial query using a spatial extension is finding all walls which contain or adjoin doors.  This requires the spatial function "touches" to find doors that are in or near walls.

As in the case of geospatial data, additional domain-specific functions are required to provide the necessary functionality.  For example, functions were added for different ways of measuring distance (spt: defines the spatial extensions namespace)

• spt:distance is a function to return the shortest distance between two products in 3D space.
• spt:distanceZ is a function to return the shortest vertical distance between two products.
• spt:distanceXY shortest distance between the projections of two products on XY plane

Another example is finding all windows that are close (less than 300 units) to a column.

Implementation and testing

As a proof of concept, the extensions were implemented
based on the open source Jena ARQ query engine.  An objective was to try to  minimize low level coding to make the function implementations more portable and more transparent for public reviews.

As a practical example, a case study of building code compliance checking using the International Building Code 705.8.4 (IBC 2006) was tested successfully using a BIM model of a building with 4 floors.

The researchers report that the main limitations of the open SPARQL-based approach they have proposed relate to the way it was implemented.  The implementation is not completely portable and performance could be improved, by integrating existing geometric libraries, for example.   Future research will address these issues and investigate more use cases to extend functions for specific domains and to combine more data from different sources in the construction industry.

Sources

Querying Linked Building Data Using SPARQL with Functional Extensions, 2016, Chi Zhang and Jakob Beetz

Processing of Topological BIM Queries using Boundary Representation Based Methods, Simon Daum and André Borrmann, Advanced Engineering Informatics, Volume 28, Issue 4, October 2014, Pages 272–286

BIMQL – An open query language for building information models, W. Mazairac and J. Beetz, Adv. Eng. Informat. (2013)

Topological analysis of 3D building models using a spatial query language, André Borrmann and Ernst Rank, Advanced Engineering Informatics 23(4):370-385 · October 2009

Thanks to Léon van Berlo of TNO for pointing me to this research.

Microsoft and open source on convergent paths

The world is changing.  Microsoft has been doing things in the open source world for some time, but this last week it has gotten wild.

 

Microsoft has just kicked off launch activities for SQL Server 2016 with general availability later this year. Spatial data has been supported in SQL Server since 2012. According to a Microsoft blog post this is the most significant release of SQL Server ever. It includes some very cool technology: in-memory database support  with dramatic performance increases and advanced analytics based on R that enables customers to do real-time predictive analytics on both operational and analytic data among other things.  Microsoft introduced Polybase, a SQL Server connector to Hadoop in 2015. Microsoft has now incorporated that functionality into on-premises SQL Server 2016. 

 

That's all pretty cool, but the biggest surprise is that Microsoft has announced plans to bring SQL Server to Linux. The core relational database capabilities are being  previewed right now.  SQL Server 2016 will be available on Linux in mid-2017.

 

Gartner has named Microsoft as leading the industry in their Magic Quadrant for Operational Database Management Systems, slightly ahead of Oracle. Microsoft also leads Gartner's Magic Quadrant for Business Intelligence and Analytics Platforms.  Very surprisingly Oracle is not even on the Magic Quadrant for Business Intelligence.

 

And that's not all.

 

Microsoft has just released the source code to an open-source operating system, based on Debian GNU/Linux, that runs on network switches. The software is called SONiC (Software for Open Networking in the Cloud). SONiC builds upon Microsoft's Linux-based Azure Cloud Switch (ACS) operating system.

Java rises to the top of programming languages in the open source community

Earlier this week Github published a graph showing the relative popularity of programming languages among GitHub development project repositories, both private and public, for 2008 to 2015. 

Some things don't seem too surprising. Javascript is at the top. C++ has stayed in the top 10 since 2008. 

But a couple of things really surprised me.  Of the three P's, PHP and Python have remained in the top 5 for the entire period, but Perl has fallen off the chart.  CSS has risen dramatically.  But the most surprising thing for me was the incredible rise of Java from 7th to 2nd since 2008. Not long ago it seemed that only the big system houses like HP, IBM  and Oracle were using Java. One reason for its spectacular rise may be that it is suited to building big distributed internet systems (the Hadoop framework is implemented in Java).  It is also used for Android which runs more than 50% of the world's smart phones.  Something which may be related is the recent spurt in C# (Microsoft's version of Java).  In the geospatial community the rise in Java popularity has been paralleled by the rise in the popularity of GeoServer and GeoTools both of which were developed using Java.

Finally, Apple's Swift development language for iOS, which Apple says it will open source, is off this graph, but apparently has already risen to 18th.  It's rapid rise perhaps accounts partially for the drop in Objective-C, which has like C++ stayed in the top 10 - until very recently.

An open distributed architecture for the smart grid based on seamless interoperability

At DistribuTECH 2015, Raiford Smith of CPS Energy and Jason P. Handley of Duke Energy, presented their perspectives on the smart grid; what is motivating it in terms of business and technology drivers, a roadmap for implementing it at their utilities, and the benefits that are expected from it for customers and for utilities.  They also outlined a smart grid architecture based on open standards to enable seamless interoperability that enables distributed as opposed to centralized intelligence.  Some of the advantages of a distributed architecture are scalability, reduced latency and implementing security at the grid edge instead of via the central control application.

Megatrends driving smart grid

Raiford Smith sees four major megatrends that are major motivating forces behind the smart grid.  Moore's Law means there are intelligent devices for power networks with greater capabilities and at less cost.  Metcalfe's Law means more interconnections and greater interoperability.  Big data analytics means the ability to extract more meaningful information and insights from rapidly increasing volumes of data coming from thousands and even millions of intelligent devices.  Distributed energy generation (DER) means more complicated power management - balancing intermittent generation and new load profiles from an increasing number of new electronic devices. 

From a business perspective a major benefit is greater customer choice.  In the future the customer will be able to not only manage his/her consumption of power, but also its generation. With rooftop solar PV and batteries the customer may elect to not even be on the grid, but to create his/her own microgrid.   But it also means the utility business model will have to evolve from what it has been for the past 100 years.  New York is one of the jurisdictions that is already changing its regulatory framework to enable utilities to move to a new business model.

Microgrids

As an aside, at this year's DistribuTECH if there was one technology that seemed to be everywhere and on almost every utility's radar, it is microgrids.  Duke Energy is even playing with the idea of offering  microgrids as a service.

Technology roadmap for the smart grid

From Raiford Smith's perspective the technology roadmap for the smart grid involves the deployment of increasing numbers of intelligent electronic devices for sensing and for control.  The challenge is federating the data from all of these devices, extracting information from it, and dispatching the information to the right control devices.  From an architectural perspective this drives the need for a field message bus which enables interoperability between different devices from different vendors.  It also requires a common semantic model, such as the Common Information Model (CIM), adding security at the edge of the grid in addition to the central control room, and analytics to extract information from the huge volume of data collected from the sensors.

To test different smart grid configurations, CPS Energy is assembling a test facility for a three year smart grid testing program.  It will have 30,000 customers, 15 circuits, solar generation, smart inverters, battery storage and the ability to disconnect from the grid to form a microgrid.

Benefits of the smart grid

Raiford expects major benefits for customers and for the utility from implementing a smart grid.  For customers, perhaps the biggest benefit is that the smart grid avoids divergence of utility services and customers needs.  Sometimes referred to as disaggregation, in this context it means a 3rd party coming between a utility and its customers. Historical examples are Microsoft Hohm and Google Powermeter which were perceived as threats because utilities found the idea of a Microsoft or Google insinuating itself between the utility and its customers unattractive. Opower is an example of a different approach that is much more attractive to utilities.  Instead of doing an end-run around the utility,  Opower focussed on the utility as their direct customer.  Services using Opower's solutions is then offered by the utility to its customers.

Secondly, smart grid provides a flexible foundation for providing new services to customers (which also creates new sources of revenue for utilities).  Raiford suggested some examples including electric vehicles and charging, premium (high quality) power, premium reliability, and asset control (for ex, inverters and batteries) and advanced demand response (the utility would provide these as a service to customers, rather than customers buying these devices from a 3rd party). The result is greater customer satisfaction and improved brand recognition as CPS Energy is perceived as a leader in providing new and improved services to customers.

The benefits that Raiford sees for utilities are equally important.  They include improved operational metrics (SAIDI, SAIFI, asset utilization), better financial metrics (O&M spend, revenue generation), environmental benefits, improved safety, better trained and skilled staff, and more reliable risk modeling (better predictions of revenue, costs, customer satisfaction, and asset condition).

Probably the greatest challenge Raiford sees is managing organizational change, because the smart grid will mean that just about everything will change including the utility business model and most aspects of how we design, build, maintain and operate the grid. 

Drivers for utility industry change

Jason Handley, of the Emerging Technology Office at Duke Energy, reviewed some of the major drivers for industry change.  Many applications currently used by power utilities are proprietary, with the result that the utility has many application silos that don't interoperate.  The rapid adoption of DERs is requiring utilities to move toward faster response times, reduced costs, better safety, and improved reliability.  Dynamic load management and low voltage power electronics will mean greater adoption of rooftop PVs and other DERs.  Increasingly utilities will invest in standards-based, modular systems for hardware, multi-function devices, and a field message bus for software that will enable interoperability.  From a business perspective broader interoperability facilitates more competition which lowers costs, encourages innovation and improves reliability.

Other important drivers that Jason sees that are impacting utilities include demand response, electric vehicles, in-premise automation, cybersecurity threats, aging infrastructure, big data complexity, and avoiding stranded assets.  The smart grid is requiring utilities to change how they do things.  Utilities realize they have to be  more proactive in their operations, rather than waiting for something to happen and then reacting to it.  Situational awareness has become a critical capability for utilities in enabling utilities to be more proactive.  It is made possible by having a variety of sensors in the field that together can present a snapshot of the status of the grid.  The key functionality required to enable this to happen is seamless interoperability.

Centralized or distributed intelligence

As utilities implement thousands and even millions of smart devices in the field, a centralized architecture runs into scalability and latency problems.  Duke's solution is an architecture with distributed as opposed to centralized intelligence.  Duke sees this as comprised of layers so that with this architecture, not all data needs to go to the central control application.  Some can be handled at lower levels.  A self healing network is an example where a problem can be handled locally without the central control application knowing anything about it.  Distributed intelligence also enables fast edge decisions that can be made without waiting for the central control application.  For example, an advantage that cannot be underestimated with this architecture is that it enables security at the edge of the grid, not just via the central control application.  Based on this concept Duke has defined a Distributed Intelligence Platform (DIP) Reference Architecture designed to take advantage of the tremendous intelligence that is out in the field in addition to the intelligence in the control centre.

Duke Energy, CPS Energy and 25 vendors, called the Coalition of the Willing (COW) have just embarked on an implementation of this architecture that supports a microgrid.  The smart grid requires exchanging data between different devices from different manufacturers in the field.  Traditional utility technologies are very often vendor silos utilizing proprietary hardware, telecommunications and software platforms.  The goal of the “Coalition of the Willing" (COW) is to demonstrate that data  and control commands can be shared across multiple vendor platforms (typically proprietary) to achieve interoperability with lower costs and faster response times.  A key part of the demonstration is an open standard field message bus implemented as an open source project.  The Smart Grid Interoperability Panel (SGIP) has created an OpenFMB working group to support this effort.

SGIP announces OpenFMB project for open standards-based interoperability platform for utilities

At DistribuTECH 2015 the Smart Grid Interoperability Panel (SGIP) has announced the launch of the OpenFMB Project, a special SGIP working group designed to leverage existing standards and structured processes to create a new paradigm for true interoperability and peer-to-peer communication across vendors' architecture that will increase business intelligence and operational efficiencies while allowing for secure and reliable communication and fast decision-making in the field.

The smart grid requires exchanging data between different devices from different manufacturers in the field.  Traditional utility technologies are very often vendor silos utilizing proprietary hardware, telecommunications and software platforms.  Duke Energy and six vendor companies formed a group called the “Coalition of the Willing” (COW) whose goal was to demonstrate that data  and control commands can be shared across multiple vendor platforms (typically proprietary) to achieve interoperability with lower costs and faster response times.  A key part of the demonstration was an open field message bus implemented as an open source project.

Duke Energy has announced the next phase of their interoperability project (COW II) with an expanded group of 25 vendor partners and a utility partner CPS Energy. The second phase will include the operation of a microgrid system which will integrate distributed renewable resources such as solar PV and battery storage with a field message bus-based distributed intelligence platform with wireless communications to devices. 

For COW II Duke Energy defined a Distributed Intelligence Platform (DIP) Reference Architecture.  According to Duke, the reference architecture is intended to document Duke Energy’s technology roadmap for interoperability using open standards-based distributed information systems.  Part of the reference architecture is a field message bus.  According to Duke the Field Message Bus (FMB) is intended to be an open standard-based, common logical publish/ subscribe interface that connects multiple disparate grid devices, telecom networks, and information systems. It is the key technology enabler to demonstrate the benefits of the distributed architecture by facilitating interoperability between multiple different vendor’s OT, IT, and telecom systems.

Starting with the  Distributed Intelligence Platform reference architecture from the COW project, SGIP has defined the OpenFMB concept as the way distributed applications and open interfaces can enable interoperable peer-to-peer data exchanges between power systems devices. The OpenFMB framework provides a specification for power systems field devices to leverage a non-proprietary and standards-based reference architecture platform, which consists of internet protocol (IP) networking, Internet of Things (IoT) messaging protocols, and standardized common semantic models (such as CIM), to enable  communications and peer-to-peer information exchange between devices on the electric grid.

Smart grid multi-vendor interoperability based on open standards and open source

The smart grid requires exchanging data between different devices from many manufacturers in the field.  Traditional utility technologies are very often vendor silos utilizing proprietary hardware, telecommunications and software platforms that communicate to a centralized hub. Last year at Distributech 2014 a group of Duke Energy and six companies called the “Coalition of the Willing” (COW) – Accenture, Alstom Grid, Ambient Corporation, Echelon, S&C, and Verizon, demonstrated interoperability between their products.  The goal was to demonstrate that data  and control commands can be shared across multiple vendor platforms (typically proprietary) to achieve interoperability with lower costs and faster response times.

At Distributech 2014 Duke energy also described  the challenges Duke has encountered in cleaning,  merging and managing operational data, combining it with other types of data including social media, and developing analytical tools to extract meaningful information.  Duke also described an innovative approach for involving vendors in the development of their smart grid platform and applications, which provided the basis for vendor involvement in COW.

COW 1 Volt/Var Optimization

The first phase of this project was to convert a Volt/Var Optimization (VVO) function to run on a distributed platform.  It used a “communications node” and a standards-based messaging architecture to enable peer-to-peer communications to integrate data from different vendors' systems.  Each of the COW participants exposed some data via a standards-based interface (Field Message Bus).  The application read two residential meter voltages supplied by Echelon meters and brought the data back via Power Line Carrier (PLC) to the Ambient communication node. If an under voltage condition was detected, the application sent a close command to the control via DNP3 through peer to peer messaging over the Verizon Wireless network. The capacitor control closed the capacitor bank to alleviate the under voltage condition on the meters.  Finally, the Alstom DMS was updated with the new state of the capacitor.

Open standards, open source

The COW 1 project involved open source hardware (Raspberry Pi), open source software and open standards.  According to Duke the Field Message Bus (FMB) is intended to be an open standard-based, common logical publish/ subscribe (pub/sub) interface that connects multiple disparate grid devices, telecom networks, and information systems. It is the key technology enabler to demonstrate the benefits of the distributed architecture by facilitating interoperability between multiple different vendor’s OT, IT, and telecom systems.  It uses open-source software to translate data to a common publish and subscribe messaging interface using the IEC Common Information Model (CIM) data model.  Other open standards that are being used in this project include industry-standard protocols as DNP3 and Modbus and messaging protocols like MQTT (for lightweight applications) and AMQP (for heavy duty messaging).

COW 2 Islandable microgrid

Last June Duke Energy announced the next phase of their interoperability project with an expanded groups of vendor partners. The second phase will include the operation of a microgrid system which will integrate distributed renewable resources such as solar PV and battery storage with a field message bus-based distributed intelligence platform with wireless communications to devices. 

The new project will support another open standard.  The Object Management Group's (OMG) Data Distribution Service (DDS) for Real-time Systems provides a secure publish-subscribe messaging protocol.  Many real-time applications involve publishing “data” which is then available to remote applications that are interested in it.  In a utility setting this communications model needs to scale to thousands of publishers and subscribers in a robust manner.

A data model based on the CIM standard will be implemented into an open field message bus to support standardized object model representations. The use case for this demonstration project will be an islandable microgrid which Duke will implement and operate at its smart grid test facility.

Duke Energy has defined a Distributed Intelligence Platform (DIP) Reference Architecture for this project.  According to Duke, the reference architecture is intended to document Duke Energy’s technology roadmap for interoperability using open standards-based distributed information systems. 

FOSS4G 2014 in Portland, Oregon Sept 8th-12th

FOSS4G 2014 runs from Sept 8th-12th in Portland, Oregon this year. The conference brings to together developers and others in geospatial open source.  This is an opportunity to participate in the rapidly growing open source geospatial community.  Last year's FOSS4G 2013 in Nottingham attracted 28 workshops, 180 Presentations and 833 delegates.

Call for papers

Presentation Proposals - Due April 15th

Academic Proposals - Due April 15th

Workshop proposals - Due March 15th

Twitter @foss4g

Oracle sponsors OpenStack

OpenStack is an open source cloud computing platform. It is free and open-source software released under the terms of the Apache License and is managed by the OpenStack Foundation, a non-profit corporate entity.

Oracle,  now the world's second largest software company ahead of IBM, has become a Corporate Sponsor of the OpenStack Foundation and says it s planning to integrate OpenStack cloud management components into Oracle Solaris, Oracle Linux, Oracle VM, Oracle Virtual Compute Appliance, Oracle Infrastructure as a Service, Oracle’s ZS3 Series, Axiom storage systems and StorageTek tape systems.

Other organizations supporting OpenStack include AT&T, Ubuntu (Canonical), HP, IBM, Redhat, Suse, Cisco, Huawei, Intel, VMWare, Yahoo, and EMC.

Data science becomes an academic discipline

About a year ago the Obama administration announced a  Big Data Research and Development Initiative to focus on extract knowledge and insights from large and complex collections of digital data.  It included an initial investment of more than  $200 million to improve tools and techniques for accessing, organizing, and extracting knowledge from the huge volumes of data that are available.  The list of ongoing Federal government programs that relate to big data is extensive.

Yesterday at a White House Office of Science and Technology Policy (OSTP) event, the Gordon and Betty Moore Foundation and the Alfred P. Sloan Foundation announced a $37.8M funding commitment to develop new data science facilities at the University of California Berkeley (UC Berkeley), the University of Washington (UW) and New York University (NYU).

UC Berkeley has announced the creation of the Berkeley Institute for Data Science (BIDS), which is scheduled to begin operating in Spring 2014 in partnership with the UW eScience Institute and the NYU Center for Data Science.

It is fantastic that data and its management and analysis at long last has been recognized as serious enough to be studied in an academic environment.  It will be interesting to see how geospatial data fits into this.

For a compelling open source-oriented perspective on this, take a look at Fernando Perez's blog where he makes the case that open source is a key ingredient in this effort.

Do Twitter, Facebook and Google run SAP, Oracle, Microsoft and IBM ?

In a very recent post from two analysts, Peter Goldmacher and Joe del Callar,  from Cowen and Co who attended Hadoop World, Cloudera's annual conference, argue that data management is at a discontinuity, where a new set of open source, big data technologies are set to replace legacy proprietary RDBMS-based technologies.

Recent and repeated top line shortfalls from ORCL and TDC have been blamed on the macro. While we believe this to be the case, we also believe these macro issues are accelerating a technology transition from legacy products to alternative data management systems as users are increasingly evaluating products like Hadoop that offer compelling functionality at materially lower price points. While a user's motivation for an initial Hadoop deployment may be cost, end users we talked to spoke of incremental outcomes around better flexibility and functionality that enabled broader value-added use cases to support the business.

The legacy providers of data management systems have all fallen on hard times over the last year or two, and while many are quick to dismiss legacy vendor revenue shortfalls to macro economic issues, we argue that these macroeconomic issues are actually accelerating a technology transition from legacy products to alternative data management systems like Hadoop and NoSQL that typically sell for dimes on the dollar.

We believe these macro issues are real, and rather than just causing delays in big deals for the legacy vendors, enterprises are struggling to control costs and are increasingly looking at lower cost solutions as alternatives to traditional products. What we are gleaning from multiple conversations with users of these new technologies is that regardless of the initial reason they experiment with these new technologies, the outcome is 1) The realization that not only are these products cheaper, but they are more flexible and better suited to many legacy workloads and 2) They offer end users the optionality to expand project scope beyond the constraints associated with legacy product, whether those constraints are cost or capabilities.

Matt Asay quotes Larry Ellison

What we think the data center of the future looks like is really a core of these commodity machines [i.e., commodity LInux servers not powered by Oracle] and a collection of these purpose-built machines [like Oracle's Exadata that serve higher-end requirements].

and concludes that it does suggest, like Ellison himself argues, that we're likely to see the legacy vendors take an increasingly peripheral role in an age of Twitter.