Between the Poles

Will IBM's adoption of OpenStack mean portable cloud applications ?

OpenStack is an open source cloud computing platform for public and private clouds. The project aims to deliver solutions for all types of clouds by being open, simple to implement, and massively scalable.  The OpenStack Foundation was founded by Rackspace Hosting and NASA.

Last week IBM announced that all of its cloud offerings would be built around OpenStack.  But this could be more important than just an open platform for private cloud management.  IBM seems to be interested  in cloud portability, making applications portable between cloud vendors and avoiding cloud lock-in.  Right now if you develop an application on Amazon Web Services, for example, it is not an easy matter to port it to run on Rackspace.  IBM says that its objective is to make OpenStack a ubiquitous, open source infrastructure-as-a-service layer, analogous to what Apache represents in serving http.  When you recollect what IBM's support of the Apache Foundation, the Eclipse Foundation, and Linux led to, this could have major implications for the future of cloud computing.

Open Modeling Framework for modeling the smartgrid including cost/benefit analysis

At the NRECA TechAdvantage conference in New Orleans Craig Miller and David Pinney of the Cooperative Research Network (CRN) gave an overview of a joint project between the Depatrment of Energy (DoE), NRECA, several electric power coops, Oakridge National Lab (ORNL), National Renewable Energy Laboratory (LREL), and Pacific Northwest National Lab (PNNL), that has important implications for all utilities that are attempting to prioritize future smart grid investments in their distribution networks.

Background

About four years ago DoE invested $33.9 million in a smart grid demonstration project  or more accuratelu a portfolio of projects conducted by CRN.  The research projects focused on distribution automation and demand response.  One of the things that CRN did as part of this demonstration project was to survey National Rural Electric Coop Assn (NRECA) members to determine what they saw as the most important barriers to implementing a smart grid.  The barriers that were reported by the respondents were, in rank order,

1. Customer resistance
2. Data volume and computational complexity (analytics) - utiltiies were facing 100K to 200K times as much as data that they were currently managing
3. Communications - moving the data
4.  Interoperability - many incompatible systems
5. Cybersecurity
6. Cost/benefit - at that time there was little data on the costs and expected benefits of various SG technologies to help prioritize investment

Recently the membership has been resurveyed.  After four years it was found that the perception of the relarive importance of the barriers has changed.  In the most recent survey respondents identified cost/benefit as the highest priority barrier to smart grid adoption and CRN has developed a project to specifically address cost/benefit. 

As a first step the project researchers looked at the cost/benefit models being used by vendors of smart grid product and services.  What they found was such a wide range of models with different assumptions, different data, different grid models, and different ways of estimating costs and benefts, that it was impossible to compare different smart grid investment options and vendors.  As Craig Miller put it, it was not even a question of comparing apples and oranges, it was more like attempting to compare apples, oranges, and '57 chevies. 

Open Modeling Framework (OMF)

This led to a joint project between DoE, CRN, ORNL, and PNNL to develop a common framework for modeling distribution grids and estimating the costs and benefits of alternative investment decisions that woudl allow comparison of different options  The framework would provide standardized data sets, for example, NOAA weather data, and computational modules including modules for modeling grid physics, such as GridLAB-D™, but would allow different data sets and computational modules to be used.  The framework would incorporate a monetization and visualization component that would perform financial analyses that could be compared across different smart grid investment options.  The most important features were

• common data
• common outputs
• multiple solvers (computatuional grid modules)
• run mgmt

All of these components can be customized, but the frameowrk is designed to make the outputs and the subsequent analysis comparable.

OMF is comprised of open source components and runs in the cloud.  PNNL is working on the solvers (grid modeling software), with a particular focus of improving performance.  ORNL is working on the monetization and visualization component.  Currently OMF takes Milsoft grid topology for input, but in the future the Intention is to support other vendors as well as the MultiSpeak and CIM standards.

OMF live demonstration

During the presentation OMF was actually demonstrated.  We were shown how to setup and run a scenario that involved conservation voltage reduction (CVR).  The grid configuration (feeders) were brought in from Milsoft.  Various ways of visualizing network topology were demonstrated (surprisingly none include a geospatial representation though GIS is on the roadmap apparently).   Different ways of modeling demand were shown.  For exmaple, there is a residential demand model that simulates appliances and other sources of domestic demand.  There is a feeder model and ultimately they intend to incoporate vendor specific equipment models, for example for Cooper or Schneider Electric transformers.  For this CVR simulation, they introduced a generalized volt/var controller. 

A network with CVR and without CVR was simulated for a location in California over a period of three months.  Using the output from the grid simulation, financial analyses were performed for the two cases, with CVR and without, and the cost and benefit of the CVR option computed.

It was possible to look at physical parameters, for example, to determine if CVR was effective in shaving peak loads.  It was found that in some cases CVR reduced peak load, but not always.  The financial analysis determined that there was a net benefit from CVR, and that the payback was estimated to be about 2.3 years.

Since the monetization analysis runs using the ouptut from the grid simulation, it can be rerun with different financial assumptions without having to rerun the grid simulation.

The power of OMF is that it can be used to compare different smart grid investment options, for example, residential solar with batteries,  utility solar with batteries, CVR, and wind generation.

Criag Miller said that OMF will be formally released this summer and will be open to all utilities.

Microsoft announces support for open source version control system Git

Microsoft has announced that Team Foundation Server and Team Foundation Service (TFS), Microsoft's collaboration platform, now supports Git, the version control system used by many open source projects. There is a plugin for Microsoft Visual Studio 2012 (VS) available to enable Git support. 

As Microsoft itself said this is a “big milestone for us".

The evolution of the open source software foundation

Steve Walli of the Outercurve Foundation has published a very perceptive discussion of why the phenomenon of the open source software foundation (neutral non-profit) has become so important in the free and open source software (FOSS) community.

Open source is the best software re-use strategy

Steve's fundamental premise is that liberally-licensed collaboratively developed software is the best re-use strategy.  There are a number of different ways to look at this, but the one that has stuck in my mind was the result of attending a military open source conference in Washington, DC some years ago.  The U.S. military, which is perhaps the biggest user of open source software in the world, uses FOSS because they judge it to be more secure and to have fewer bugs than proprietary software all other things being equal.  The reason most often given for this is Eric Raymond's "many eyeballs" thesis that more people seeing code reduces bugs and security problems.

Economic benefit

As I remember from the history of the development of the Apache foundation, there is often a very strong economic motivation for FOSS.  In the early days of the web, there was the opportunity for every major vendor, IBM, HP, Microsoft, Sun, Oracle, and others to develop their own http web server.  Instead IBM helped form the Apache Founation and then encouraged other major vendors to join which just about everyone did (with one important exception).   The major vendors jointly contributed to developing one web server based on the NCSA http server and found it much less expensive than if each vendor had developed its own web server.

Intellectual property rights management is what FOSS foundations do

Steve makes the case that the core capability of FOSS foundations is to manage IP - he refers to them as IP Management Machines.  He says that this was the primary motivation for the Free Software Foundation (FSF), Apache Software Foundation (AS F), OSDL/Linux, and the Eclipse Foundation.  Corporations have more stringent software intellectual property (IP) needs and. neutral non-profit FOSS foundations have proved to be a solution to this problem, providing for the IP management needs of corporations.  FOSS foundations remove the barriers to corporate engagement for FOSS projects.

For example, for all contrbutions to Eclipse projects, the contributor has to agree to a standard IP statement for every submission.  It also must be contributed through standard channels, either Bugzilla or Gerrit, either of which which provide a complete audit trail for any contribution.   

For large contributions (over 250 lines) or code from external projects, then a complete IP review is required.  This can take a  long time because the provenance of all the code is checked, the licences are checked, and the code is scanned for IP problems.  Th reason Eclipse insists that this soemwhat involved process be followed for all contributions is that it guarantees that all Eclipse code can be used legally for commercial purposes under the EPL.

Growing projects into mega projects

Steve refers to Henrik Ingo's research which I blogged about recently.  Henrik found that the largest software projects are governed by FOSS foundations.   He concluded that there appears to be a glass ceiling for single vendor projects prohibiting their growth from the Large category upwards.  Based on his research findings he recommends that companies who want their projects to become mega project should contribute these projects to a FOSS foundation.

What is a FOSS foundation ?

A FOSS foundation is a neutral (not dominated by one vendor) non-profit (but run as a business) organization that provides legal structures, business operations, and technical services to its members, most of whom are firms or individuals involved in some aspect of software development or as software users.   FOSS foundations are different from corporate open source projects like OpenSUSE, Fedora, JBoss, or MySQL in several important ways. 

• FOSS foundations are neutral which encourages participation. 
• FOSS foundations put a lot of effort into managing IP openly and cleanly because clean IP encourages adoption among the user community. 
• FOSS foundations encourage broad participation, something that it is hard for a corporate project to do.

FOSS foundations have many projects they are responsible for.  The Eclipse Foundation for example has 270 active projects.  FOSS foundations also provide technical services and infrastructure such as source code repositories, build tools, test harnesses, code signing, bug tracking,  and so on. 

For example, the Eclipse foundation has implemented a very efficient process for contributors that minimizes for the contributor and the committer the time it take to submit the code, verify that it compiles and links correctly, conduct code reviews, run standard tests, and if all that succeeds actually submit the code to the main tree on Github.

Legal structures

One of the most important services that a FOSS foundation provides is legal liability and risk management.  This includes providing and managing licenses for contributors and users, provenance tracking, and indemnification.

Strong communities

In my view one of the most important things that FOSS foundations provide is a governance structure that creates an architecture of participation.  The objective is inclusion not exclusion.  The governance structure makes it easy to join conversations and to participate in various ways based on merit.  It  also encourages and grows committers (developers who can commit code changes). 

At the Eclipse Foundation there are something like 6 million users, some of whom contribute to the Eclipse ecosystem in various ways,by configuring the IDE, extending in in various ways, developing and publishing plugins, integrating Eclipse in products, and at the highest level fixing problems and developing new modules for Eclipse project code itself.  There is a fascinating video of a presentation on what it means to be a committer by a couple of Eclipse committers at a recent EclipseCON conference in Europe.

One of the most important aspects of being an Eclipse participant is the Eclipse software development process which is part of the Eclipse governance structure. 

This development process is used by all Eclipse projects.  The key objectives of the Eclipse development process are transparency (all decisions, code, and discussions can be seen by everyone), openness (encouraging participation by making it easy to join discussions), and diversity (encouraging broad participation, encouraging as many firms and individuals as possible to particpate).

The world's largest open source projects are governed by non-profit foundations

A while back Henrik Ingo was asked to do a study of the most popular open source projects to see what governance model they used and whether the governance model had any impact on how successful the project was (as measured by the size of the developer community) and  how much business it generated for vendors.  The results are remarkable.

Henrik found that nine projects Linux, KDE, Apache, Eclipse, Perl+CPAN, Mozilla+Addons, Gnome, Drupal and GNU stood out from all the rest because they were so much larger, by about 10 times, than any of the others.  Another common characteristic of all of these projects is that they are collaborative community projects governed by non-profit foundations. Henrik found that there is no single-vendor project, such as MySQL, Qt, OpenOffice, Mono, or JBoss, that comes even close to reaching this size.  Henrik hypothesized that there is a "glass ceiling" that limits the growth of single-vendor projects.

Based on his study Henrik recommended that vendors participating in open source development and business should seriously consider participating in a collaborative community run by a  non-profit foundation.  He suggests that a vendor currently in control of an open source project could, as a rule of thumb, expect this strategy  to result in a 10x growth in the project and product and a 10x larger addressable market, of which the vendor could potentially capture 50% or more as its own market share.  Certainly very attractive from a business perspective and perhaps one of the reasons that companies like IBM and Oracle support non-profit foundations.

The next step in the evolution of open source geospatial software

Apache Foundation and corporate engagement

As I remember the story, back in the late 1990's IBM, seeing a need for a http server for the rapidly developing internet market, looked at a couple of alternatives, developing one in-house or adopting the http server developed at the National Center for Supercomputing Applications (NCSA).  The NCSA http server project had been terminated at the NCSA, but the http server continued to be supported by a group of eight developers.  As I remember it IBM decided that it would be more cost effective and would make more sense from a market perspective to adopt what was the de facto industry standard.  But IBM insisted that for this to work there had to be a legal entity with which IBM as a corporation could commercially and and legally engage.  And so on March 25, 1999, the Apache Software Foundation was formed and the rest is history.   IBM and ultimately many others adopted and distribute the Apache server.  As of September 2012 it was estimated that  Apache serves about 55% of all active websites.  From an industry perspective adopting the Apache server was a much more cost-effective alternative than if HP, Oracle, IBM, Sun, and others had each developed their own http server.

OSGEO

Back in 2005/2006 I was working with Dave McIlhagga to interest Autodesk in becoming a corporate sponsor of open source geospatial software, and I saw a similar scenario played out.  It seemed to many of us that it would be more cost-effective for companies to share web-mapping software infrastructure rather than each develop its own web mapping server.  At that point there was already well-developed and well-known open source geospatial projects whose code was being used all over the world.  The best known examples were the University of Minnesota's MapServer project and Frank Warnerdam's GDAL geo-imagery libraries.  Again Autodesk insisted that there had to be a legal entity with which as a corporation it could engage.  The Open Source Geospatial Foundation (OSGEO) was modeled loosely on the Apache Foundation and was founded in March, 2006 with Autodesk as the major sponsor.  The OSGEO has been incredibly successful.  It has grown rapidly to support 25 or so projects, some in incubation,  20 chapters worldwide, and a national conference, FOSS4G, which last year in Denver attracted about 900 participants.

The next step

As a recent article has pointed out, 15 years ago when open source was just beginning, the discussion was focussed on the total cost of open source vs proprietary, alternative open source business models, and open source as an alternative to monopolistic proprietary.

Since then open source has come a long way.  One of the most important developments is the rise of the open source foundation driven by the need for corporate engagement.   These non profit legal entities offer projects important benefits that are difficult for a project to fund on its own.

• A host for managing fiscal and intellectual property shared resources such as trademarks and shared copyrights
• An employer for staff serving the community and project
• A guarantor and enabler for governance
• An infrastructure provider
• A liability firewall for community participants

Two examples of such foundations are the Apache Software Foundation and the Eclipse Foundation. These foundation encourage trust in the long term stability of the projects they support and, most importantly for enterprise software, they encourage corporate participation.

The other major development is in the area of licensing.  The Open Source Initiative (OSI) lists 69 different licenses that range from the Gnu Public License (GPL) to the very liberal MIT/BSD style licences.  Recent years have seen the development of licences in the middle ground that are designed to be business friendly, in particular to allow mixing of open source and proprietary modules.  The Eclipse Public Licence (EPL) is an example of such as licence and is a critical reason that large corporations such as IBM and Oracle are important supporters of the Eclipse Foundation.

LocationTech

There is a pent up interest for geospatially enabling enterprise information technology. For open source projects interested in encouraging adoption and support from vendors in this area, doing business implies corporate engagement with companies such as IBM, Oracle, and other major IT vendors.

A number of people advocating for open source geospatial software have seen the need for services and facilitates to enable corporate engagement. In my view, the type of projects that are seeking corporate engagement would be attracted to the Eclipse Foundation. The Eclipse Foundation provides services to reduce friction for organizations to re-use and contribute to open source projects. This supports business developing products and services that depend upon open source and in turn, the open source projects benefit from re-use, investment, and increased credibility.

Based on this thinking Andrew Ross, well known for his work in a variety of open source communities, initiated what is now officially known as the Eclipse Foundation LocationTech Working Group along with a team of representatives from notable companies and open source projects.  In my and others view. the LocationTech Working Group fills an important gap in the enterprise geospatial market.  A close relationship between OSGEO and LocationTech is advantageous to the broad open source community because they are both serving the community, but in different, complementary ways.

FCC shows latest U.S. broadband deployment progress in an interactive map

I've blogged about the FCC's broadband map before. Last year the National Broadband Map was launched with more than 25 million records showing where broadband Internet service is available, the technology used to provide the service, the maximum advertised speeds of the service, and the names of the broadband providers.

The FCC has just released an interactive, web-based map illustrating the broadband deployment results reported in the FCC's Eighth Broadband Progress Report. 

 

It shows census block areas of the United States with and without access to fixed broadband of 3 Mbps download and 768 kbps upload.  You can zoom into the map or pan around and explore details of broadband availability in each county in the United States.  These details include population and income numbers from the census, and the percentage of each county that has access to the major fixed technologies providing broadband service.  Perhaps even more importantly, the map provides full access to anyone with a web browser to the raw data used in the map and the Report.

A few other important points about the map.

• Accessibility: no software barriers, terms of service, or other restrictions prohibit use
• Machine readable: the map data is stored in a SQLite file.  SQLite is an open source project.
• Non-proprietary format: no other software is required to look at or analyze the data
• Open IP (License free): data is not subject to any further licensing restrictions.

Major milestones for open source: Red Hat exceeds $1 B and Microsoft accepts external code

It has been reported that Red Hat has become the first company making its money primarily from open source technology to exceed $1 billion in annual revenue.

Microsoft has announced that it will allow developers outside of Microsoft to submit patches and code contributions that the Microsoft development team will review for potential inclusion in its products.  Microsoft has announced that it is releasing pieces of its ASP.NET development framework (ASP.NET MVC, ASP.NET Web API and ASP.NET Web pages) under the Apache 2.0 license.

FOSS4G: Two thirds of this year's attendees were new to FOSS4G

I blogged earlier about this year's FOSS4G conference in Denver, which attracted the highest attendance ever.  I've seen statistics that for two thirds of the attendees this was their first time at a FOSS4G event.

Of these an estimated 100+, according to a tweet I've seen from Arnulf Christl, president of OSGEO, attended the newcomer event, which suggests that a significant number of new FOSS4G attendees are also new to open source geospatial.

Some of the videos of this year's FOSS4G presentations are available on line at FOSSLC, and I expect more will be available in the future.

FOSS4G: Open source geospatial newcomer event attracts a full house

Yesterday at the FOSS4G conference in Denver, one of the workshops that was new this year was a newcomer event targetted on folks who were new to FOSS4G and new to open source geospatial.  I've seen attendee counts that suggest tha attendance this year at FOSS4G will be around 900, an all time high for FOSS4G.  Of these an estimated 100+, according to a tweet I've seen from Arnulf Christl, president of OSGEO, attended the newcomer event, which suggests that a significant number of FOSS4G attendees are new to FOSS4G and new to open source geospatial. 

The newcomer event was moderated by Brian Timoney who besides moderating provided background on open source, open standards, and open data.  Peter Batty, chair of FOSS4G this year, gave the perspective of someone with an extensive background in proprietary, closed source software (Peter has worked at IBM, Smallworld, and Intergraph), who has found significant benefits in using open source software in his current role at Ubisense. 

I talked about why Autodesk saw an important role for open source in the area of interoperability, specificially for an Autodesk developed technology  called Feature Data Object (FDO) API which is designed to allow designers to directly access geospatial data without making copies.  As a concommittant, Autodesk supported the development of the Open Source Geospatial Foundation (OSGEO) back in 2006.  Coincidentally, I was able to announce that just that morning Autodesk had released Revit Industry Foundation Classes (IFC) exporter as open source. I tend to use utility industry examples, and I was surprised to find that there was at least one utility person in the audience, from Xcel Energy.  I suspect that this is just the beginning of a trend and that in future FOSS4G events more folks from industries outside of government will be attending and discovering the role that open source has to play in their organizations.

Michael Byrne, GIO of the FCC and who used to be GIO of California, and Juan Marin also of the FCC gave a fascinating technical and business prespective on the  development of the National Broadband Map, which was developed in seven months from a standing start and got half a million hits in the first ten hours it was up.  Major business objectives mandated by Congress included searchability, interactivity and on-line.  His number one priority was the tight development timeline, 7 months, and his second priority was user response time.  Seven months is an incredibly tight development timeline for any eneterprise system.  Early in the development cycle, the team, which was only eight people, decided that a RESTful architecture (see GeoREST for an example) and open source geospatial software were the only way they were going to make this timeline.  They decided they needed to communicate directly with developers to be able to resolve problems in days, not months.  After deciding on the software stack, the system was developed in four and a half months and survived a deluge of traffic in its first hours and days.

Arnulf Christl, president of OSGEO, gave an impassioned perspective on the advantages of open source in the geospatial world.