Between the Poles

Firm realizes ROI of 203% by integrating CAD and GIS for fiber network management

Integrating CAD and GIS into an efficient workflow for records management (aka network documentation) has been a challenge for utility and telecom operators for decades.  While utility and telecom operators have found ways to address this problem, it is rare to find the benefits for these systems quantified.  At Year in Infrastructure this year iForte Solusi Infotek presented concrete benefits of a records management system that has been in operation for several years for managing their fiber optic networks in Jakarta and on the rest of Java.

I began working in the utility and telecom sector in 1993, specifically in developing software for records management (called network documentation outside of North America).  At that time the magnitude of the AEC+geospatial cultural mismatch problem was just beginning to be appreciated by utilities and telecoms.  Planners tend to use GIS tools, engineers and designers CAD tools, construction folks in the field paper CAD drawings, and asset managers GIS integrated with their FM tools. Fashioning engineering and geospatial data into an efficient data flow from planning through design and construction to operations and maintenance represented a challenge that remains a problem for utilities and telecom companies. The Between The Poles blog is over ten years old and one of the persistent themes from the very beginning in 2006 was the challenge of integrating CAD and GIS data and applications in an efficient workflow (some examples; 2006, 2007, 2008 ).

The winner of this year's Year in Infrastructure Award for Communications Networks was a project of iForte Solusi Infotek.   iForte is a fast-growing fiber network operator with many large corporate customers all over Java island including Jakarta.  From this base it plans to expand into the rest of Indonesia.  iForte was facing many of the problems associated with paper-based records management in a fast-growing network operator. Prior to implementing the new system, their business processes were characterized by a plethora of paper-based manual processes. The data available for network planning was continually out of date making it difficult to compile an accurate situational assessment. Maintenance crews in the field had to rely on their own hand written notes on paper records. When sales staff met with a prospective customer it took an hour or more to collect the required information to determine the availability of fiber optic capacity at the customer's location requiring subsequent exchanges of emails to get the requisite information to the customer by which time the customer may have contacted a competitor.

The primary benefits of the new system were realized in planning, network maintenance and sales. 

• Maintenance crews could redline existing as-builts in the field so that engineering could see immediately where a problem had occurred and any temporary infrastructure the maintenance team had put in place to address the problem.
• Planning could see real-time cable utilization to identify congested networks. When there was an outage they could see the total capacity that was at risk.  It also enabled them to find alternatives for re-routing the network.  It was also found that network expansion planning could be completed four times faster. They were also able to use less fiber, saving on material costs and CAPEX. For example in one case, by looking at available capacity and utilization, it was possible to reduce new cable requirements from 1,400 meter to 100.
• For sales prior to implementing the records management system, determining fibre availability for a customer required on average of 60 minutes necessitating telephone calls to records. After implementing the new system, they were able to determine availability in six minutes and automatically generate a bill of materials and a job estimate.

For these activities iForte estimated that it has realized a return on investment of 203% over three years.

The primary benefits of the new system were realized in planning, network maintenance and sales, but there were intangible benefits as well.  Management was able to see the status of the network in real-time, review progress in building out the network on a map with demographics, assess the status of their network assets, and identify regions for network expansion.  The intangible value to the company achieved with the new system was operational excellence and improved strategic planning which provided for better performance and growth compared to competitors.

Geolocation plays a key role in integrating GIS and engineering data and applications for optimizing asset management workflows.  One of the reasons for the success of iForte's system is that Bentley decided many years ago to geospatially-enable all of their products including Microstation.  Bentley calls this capability geocoordination. It provides a location-aware foundation which facilitates workflows that require integrated engineering and geospatial data and applications.

Open source-based mobile viewer creates value for electric power utilities with multiple GISs

One of the challenges faced by large utilities with multiple GIS, outage management, customer information and other systems, which are often the result of mergers, is how to make spatial and associated data, in different, often incompatible systems, accessible to all users not just GIS professionals.  At Distributech 2018 Duke Energy described an off-the-shelf web-based product based on open source components that is being used by 8000 users, 2000 of them using it 24/7 every day, to access GIS and associated data in several GIS systems, both online and offline, using laptops, tablets and phones.  An important advantage of the the selection process Duke used in selecting the product was that it made it possible for a relative newcomer to compete successfully with the traditional GIS incumbents.

Background

Large investor-owned utilities are often the result of mergers. For example, in 2012 Duke Energy and Progress Energy merged to create the largest U.S. utility with 7.4 million U.S. customers and approximately 52,700 megawatts of electric generating capacity. The challenge for the IT folks is how to manage the many disparate IT systems that such mergers bring together. Historically Duke Energy and Progress Energy have employed all the major GIS systems including Synercom, Intergraph FRAMME, GE Smallworld, ESRI, Intergraph GTechnology, and IBM GFIS. Accessing the valuable operational data in multiple systems is a challenge because each system has its own user interface and command structure. Over the years I have seen a number of solutions that have attempted to address the challenge of providing a common viewer for visualizing data from multiple data sources. But each has had limitations, for example, all data had to be synced to a central Oracle database, or it could only run online making it difficult to deploy to the field, or it had limited support for the many GISs and user files used by utilities, or it supported iOS on handheld devices but not Android, or did not support open standards such as OGC WMS and WFS, or did not support Google Maps, or was so complex users required extensive training and it was like pulling teeth to geth them to use iit.

At Distributech 2018 Ron York, Lead Consultant/ GIS, and Francisco Sarmento, Lead Consultant, both at Duke Energy described how Duke has resolved the problem of a common viewer for several different GIS systems which was brought to a head by the merger of Duke and Progress. They also described the somewhat unusual process by which Duke selected the vendor whose product provided the successful solution.

Open source geospatial components

A key differentiator of the product Duke chose is that it is leverages open source components; PostGIS, GeoServer and Leaflet being the main ones. The product builds on the collective wisdom, code and support provided by the worldwide open source geospatial community.  Leaflet is the leading open-source JavaScript library for mobile-friendly interactive maps. PostGIS adds spatial capabilities to the PostgreSQL relational database. It extends PostgreSQL so it can store, query, and manipulate spatial data. GeoServer is a web server that allows maps and data in a variety of formats to be served to standard clients such as web browsers and desktop GIS programs. Data is published via Open Geospatial Consortium (OGC) standard interfaces, such as WMS and WFS.  These are all Open Source Geospatial Foundation (OSGEO) projects.

Multiple GIS support

The product Duke selected supports ESRI ArcGIS, GE Smallworld, Intergraph GTechnology, spatialNET and Google Maps. At Duke it is used to access data in a couple of GISs as well as non-GIS data in Maximo, SAP, three different outage management systems, four or five customer information systems, and can link to CAD drawings and photos and images.  Importantly for engineering users it preserves the original symbology and background maps of the different GIS systems.

Online and offline

Another key advantage of Duke's solution is that it can run equally well offline and online and is able to seamlessly combine and switch between online and offline data.

Mobile and office devices

It supports mobile as well as office devices, running on laptops, tablets and phones running iOS, Android, and Windows. The user interface adapts automatically for each device, displaying map layers, forms and other data in a way that is appropriate for each device.

Synced or direct access to databases

It provides two ways of accessing data; one is to periodically import or sync data into the system's spatial database, which uses PostGIS, and the other is to directly access the data via a web service. It supports ESRI REST Services, which are used by ArcGIS Online and ArcGIS Enterprise, and the industry standard OGC WMS and WFS web services, which are supported by all major geospatial vendors.

It does not support direct edit of spatial data, but does allow a GIS data correction requests to be routed through Maximo following Duke standard business procedure for making corrections.

It is flexible enough to support different business processes for collecting and organizing data. For example, pole inspections are conducted by circuit in what was Progress service territories and by square area in what was Duke's service territory.

Simplicity of use

Another of the key differentiators of Duke's web-based solution is that it is not targeted on GIS professionals, but on the 95% of people within Duke who can get a lot of value from a really simple Google Maps style interface onto their enterprise data.

Leading up to and during deployment Duke put on road shows, made web demos available online, utilized computer-based-training, and provided job-aid documents to make people aware of the new system.  The tool caught on surprisingly rapidly. Within five months of it being deployed there were 5000 users. Currently there are 8000 users with over 2000 users accessing it 24/7 every day.

Anticipated and unanticipated benefits

The major expected benefit that was realized by the system is that eases the integration of Duke and Progess. All GIS asset data is exposed through a single tool to designers and asset engineers. But there were some unanticipated benefits as well.  It has had a direct impact on tax calculation - it shortened the cycle for internal auditors to complete tax assessments.  It is also widely used during storms and other natural disasters. It enables crews from different area to use a familiar tool to analyze unfamiliar data in areas where they haven't been before.  It is used to provide driving directions for field crews.  Storm managers overlay outages, assets, flood plains, wind, rain, traffic  and other maps for situational assessment.  It allows manager to share staging sites during storms. It has also  proved extremely useful in locating "unmatched outages" - outages called in by first responders with just an address.

Process for choosing a vendor levels playing field for new entrants

Duke used an interesting process to select the vendor. Preliminary vendor interviews were conducted to exclude vendors that clearly didn't have a solution that could meet Duke's needs. Remote web demos provided a cheap and easy way to narrow down the list of vendors. A request for proposal (RFP) process was used to solicit technical-only proposals from product vendors. (Duke was only interested in off-the-shelf products not bespoke solutions.)  Duke picked the top five and asked them to come to Duke for a scripted demonstration. Instead of asking the vendors for a benchmark demonstrating how their product would address Duke's requirements using Duke data, they asked the vendors to demonstrate their product using their own data. They picked the top two vendors and began contract negotiations to decide on one vendor which turned out to be Ubisense MyWorld, a relatively new entrant in the geospatial world.  An important advantage of the selection process Duke used is that it made it possible for a relative newcomer to compete successfully with traditional GIS incumbents.

An open source web-based platform for integrating BIM and 3D geospatial

BIM+geospatial integration is being driven by a trend toward infrastructure construction projects that require maintenance and/or operations during an extended period post-construction.  At the GEO|Design+BIM conference in Amsterdam, Sanghee Shin, CEO of Gaia3D Inc., gave a riveting presentation about a web-based geo-platform integrating massive, complex BIM models using open source geospatial libraries and tools including PostGIS, GeoServer, and Cesium and NASA WorldWind virtual globes.

Drivers of BIM+geospatial integration

Since the world became aware of the importance of human impact on the environment, it has become increasingly difficult to design buildings without reference to geographic context.  Many cities have environmental zoning and other types of by-laws that require designers to take into account neighboring buildings, infrastructure and vegetation.  States and national governments are developing building codes that require designers to assess the energy, water, emissions, and other environmental impacts of buildings.  This means that when designing and constructing a building or renovating existing buildings, the traditional approach ( often associated with a 0,0 coordinate system ) of design in isolation is no longer adequate for sustainable design.

Furthermore integrated geospatial+BIM is essential during the operations and maintenance phase of a building.  Utilities, transportation, emergency planning, first-responder access, energy performance, and evacuation management all require information about internal and external infrastructure.  A common geospatial coordinate system enables a comprehensive operational view of all infrastructure including internal and external structures. 

BIM+geospatial integration provide greater value to projects that involve not just design and construction but also operations and maintenance.  A leader in this space, Rijkswaterstaat, the Dutch transportation authority, began offering design-build-finance-maintain (DBFM) projects a number of years ago which has motivated private Dutch engineering and construction companies to adopt an integrated geospatial+BIM approach to construction.  For example, the firm Royal BAM Group nv/BAM Infraconsult adopted integrated BIM+geospatial because of market developments including more complex construction assignments and an increasing demand from customers for service provision throughout the entire life cycle of a project.

It is indicative of the importance of the integration of these technologies that Autodesk and ESRI, 800 lb gorillas in their respective areas of BIM and GIS, have announced an agreement to collaborate to help bridge the gap between BIM and 3D GIS.  This is their seond attempt.  The first attempt, an agreement between Jack Dangermond and Carol Bartz in the mid 90s, resulted in a product called ArcCAD, intended for civil engineers and planners.

An open source geo-platform for integrating BIM+geospatial

mago3D is a new 3D geo-platform developed by Gaia3D.  It is open source and integrates many well-known open source geospatial products and APis. It is web-based so it is not necessary to install anything to run it. It has an engine for handling massive, complex BIM objects based on an internal format (F4D) that was created to reduce network data traffic, enable LOD (level of detail) control, support octrees, a 3D spatial structure that enables efficient spatial queries and (frustrum and occultation) culling.  Culling refers to the removal of objects that are not visible to the observer and is key to efficient visualization or large, complex structures. It integrates with two of the best known open source 3D virtual globe tools, Cesium (supported by AGI and Bentley) and NASA WorldWind, for visualization in a geospatial context.  Viewing BIM structures in mago3D has some similarities with a Google Earth experience. The source code is available under Apache and AGPL (Affero GPL) open source licenses.  Cesium is an open source project supported by AGI and Bentley.

mago3D includes a F4D Converter that converts BIM models in IFC (buildingSmart Industry Foundation Classes) format as well as 3D objects in .3ds (Autodesk 3ds Max), .obj (Alias/Wavefront) and .dae (Collada) formats into its internal F4D format.  Industry Foundation Classes (IFC) is a vendor neutral and open model that most industry software BIM tools including Revit and Tekla are able to export, although there is typically some information loss. The bimserver.org open source project provides an open repository of IFC models and offers a Java API to extract partial building information models from a repository.

The main components of mago3D are mago3D.js, a javascript plug-in for WebGL virtual globes and open source web server (NGiNX), map server (GeoServer), geospatially-enabled database (PostgreSQL+PostGIS), 2D mapping (OpenLayers, Leaflet) and 3D visualization (Cesium or NASA WorldWind). The mago3d.js javascript plugin is designed to run in a WebGL virtual globe environment in a browser so that moving/rotating objects such as BIM models and heading adjustment happen in-browser and are very fast. 

I went to the mago3D site and viewed a BIM model of a chemical manufacturing facility in South Korea.  I was impressed with the ability to zoom ala Google Earth from the entire virtual globe down to viewing the chemical facility in its geospatial context.  I could continue down to a very detailed, almost nuts and bolts, level showing that the LoD control and the octree-based culling algorithms make visualization very efficient, even for large complex objects.

The importance of this technology is that it shows that it is possible using the available open source, geospatial APis and tools - together with some genuine innovative development - to create an open, non-proprietary 3D geospatial platform for integrating geospatial and BIM.  Given the critical importance of addressing the cultural and technical divide between the AEC and geospatial worlds, a viable open source alternative is essential for developing the innovative solutions to the challenge of interfacing the two worlds. 

The other part of the solution is common, open standards. There has been encouraging progress on that front.  The major breakthrough in bringing the architectural and geospatial views onto a common footing is the OGC LandInfra Conceptual Model developed by the OGC in cooperation with buildingSMART International and approved as an OGC standard in August, 2016.  LandInfra  was developed by Bentley Systems, Leica Geosystems, Trimble, Australian Government Department of Communications, Autodesk, Vianova Systems AS, and buildingSMART International and provides a unifying basis for land and civil engineering standards including the OGC's InfraGML and buildingSmart International's IFC for infrastructure standards.  

GeoBuiz 2016: High volume geospatial imagery processing requires new architecture

DigitalGlobe owns and operates a constellation of commercial earth observation satellites.  The latest is Worldview-3 which is capable of about 30 cm precision.  Digital Globe's satellites collect 3,000,000 square kilometers of Earth imagery every day, that's about 70 terabytes of data per day.  Digital Globe also provides online access to 15 years of earth imagery, about 90 petabytes.  Of all high-resolution commercial imagery collected since 2010, Digital Globe has collected approximately 80% of it.  Clearly as a user you can't download all of this data, so how can users do things like search through this huge data set and generate a product such as a vegetation map for Tanzania ?

At the GeoBuiz 2016 Summit in Bethesda this week, Walter Scott, CTO and Founder, outlined how  Digital Globe is making this vast amount of high resolution data accessible to and usable by customers.  To make it possible for customers to search and process this data efficiently, Digital Globe has put all of the data on the public cloud (Amazon) and has provided a platform that supports efficient search, processing and workflows. GBDX is an online environment for efficiently running advanced algorithms for information extraction from imagery datasets at scale.  It is a set of RESTful APIs, and algorithms that have already been implemented by Digital Globe or its partners include car counting, orthorectification, land use, land cover, and atmospheric compensation. 

One of the most important efficiencies of the GBDX architecture is that it avoids having to move huge volumes of data over the internet.  It does this by bringing the application to the data, rather than the data to the application.  For example, to search through the data, GBDX provides a catalog API that uses 39 different image attributes such as geographic location, cloud cover percentage, sensor type, sun angle and others.  The search runs locally on Amazon, avoiding pushing data over the internet.

The GBDX API includes support for workflows.   A workflow is a sequence of tasks executed against sets of data. A typical workflow is initiated with a search for the data you want to process.  The selected data is assigned to a working set, and then processed  through a sequence of steps to generate your desired product.   For example, a typical product would be an orthorectified vegetation map.  Again the workflow runs locally on Amazon, avoiding pushing data over the internet during processing.  Only the final product resulting from the workflow, such as a vegetation map, would be transferred over the internet to the customer.

GBDX uses the cloud infrastructure of Amazon Web Service (AWS).  Data is stored in S3 ‘buckets’ that are accessed through GBDX RESTful web service APIs. GBDX includes built in tools supporting remote sensing algorithms from DigitalGlobe and ENVI.  GBDX is python-friendly and allows you create your own tools.

One of the application areas that the platform makes possible is deep learning.  The basic concept is that software can simulate the brain's large array of neurons as an artificial “neural network".  Large volumes of input data and some way of quantitatively assessing the results are required to train the system.  With the greater depth enabled by new computers and algorithms, remarkable advances in speech and image recognition are being achieved.  For example, a Google deep-learning system that had been shown 10 million images from YouTube videos proved almost twice as good as any previous image recognition effort at identifying objects such as cats.  Microsoft has demonstrated speech software that transcribes spoken words into English text with an error rate of only 7 percent.

But deep learning requires a lot of compute power and a lot of training.  GBDX provides the computational power to apply deep learning to earth observation and Digital Globe's data library can provide the training data sets.  What is required is a scalable way of assessing and quantifying the results. Walter Scott suggested crowd sourcing as a way of achieving the scale required for earth observation training.

SPAR 3D: Virtual reality, scan to BIM, infrastructure models, and mapping indoor spaces

I have just experienced two remarkable days at the Spar 3D Expo and Conference in The Woodlands, Texas, in itself a remarkable location near Houston.  I will blog in more detail in the near future about the things and people I had the opportunity to experience over the past two days.

First of all, virtual reality.  Everyone is aware of the Oculus Rift (see the new review at CNET if you aren't).  There are other technologies that have come or are coming from Samsung, Sony and Google.  But at SPAR 3D the most remarkable VR technology I personally experienced was what David Smith, CTO of Wearaility, talked about and allowed me and others to try - a set of incredibly light-weight glasses with acrylic lenses designed to work with a smartphone that gives you a virtual experience comparable to IMAX without the IMAX theatre.  I tried it and found it an absolutely amazing surround experience.  And if you have wondered where Michael Jones, of Keyhole and Google Maps fame, has landed, he is now CEO of Wearality.

Secondly, my personal passion is the geolocation of underground utilities.  At SPAR 3D Mark Klusza, Founder & CTO, Real-time Metrology, Inc. described a major new advance in ground penetrating radar (GPR) technology he has developed that can detect a 3/4 inch rebar (among other things) four feet below the surface in clay.

For existing buildings creating a BIM model remains an art rather than science.  But at SPAR 3D a number of new technologies were presented that can help automate, improve the quality, and speed the process.  Foremost among them in my mind was presented by Nicolas Arnold, VP of Product Development, at SKUR.  With SKUR's technology, which runs in the cloud, you can compare the 3D model that you have developed with the point cloud you derived the model from and identify all the points of significant variance.  The same technology can also be used during a construction project to compare what has been built to what was designed - even when the building is only partially completed.

Ron Singh, Engineering Automation Manager/Chief of Surveys, Oregon Department of Transportation (DoT) described progress on his remarkable vision of how automation is upending how we design, build and monitor, operate and maintain U.S. highway systems.  Richard Arrowsmith, Asset Information Group Team Leader, Highways England described the digital model of the English highway system.  Stan Burns, President, Integrated Inventory, who has just retired from Utah DoT, described Utah DoT's comprehensive database of every piece of highway furniture and signage in the Utah highway system.  Together these represent the future of the digitalization of national highway systems.

A new company, Indoor Reality, is the third startup of Dr Avideh Zakhor, who is on leave from the University of California Berkeley.  Her new device is a backpack loaded with sensors including lasers and infrared that enables you to map the interior of buildings by simply walking through a multi-floor building, including up and down stairs.  The resulting data can be used to automatically generate floor plans and 3D models and captures everything you need for an Energy Plus analysis.

Mark Shell and Darrell Gadberry, both of the City of Fort Worth Water Department, described a very versatile device with laser and sonar and camera devices on board that can be used for sewer line inspections of small and large diameter sewer pipes.  Fundamentally this device enables condition-based maintenance of municipal sewer systems.  Mark and Darrell reported that it has saved the Fort Worth Water Department $42 million.

I was able to interview Dr Avideh Zakhor, Nicolas Arnold, Ron Singh, Larry Kleinkemper of Lanmar Services who specialize in creating BIM models for existing buildings, and Greg Bentley of Bentley who has a broad, forward-looking perspective on the "digitalization" of infrastructure.

For the future of distributed energy watch the Hawaiian power utilities

Keep a close watch on the Hawaiian power utilities (HECO) because Hawaii has committed to 100% renewables by 2045 and 65% by 2030. That was the recommendation of Sharon Allan, CEO of the Smart Grid Interoperability Panel (SGIP) at a DistribuTECH2016 MegaSession on distributed energy resources (DER) in response to a question from the audience asking what should utilities do to get ready for GRID 3.0.

Hawaii already has 487 MW of solar PV capacity, 90% of which is residential rooftop panels. At  DistribuTECH2016, Talin Sakugawa from HECO and Matthew Shawver from in2lytics gave an insightful presentation on distributed energy generation (DER), big data and analytics at the Hawaiian power utility.  Hawaii has seen an exponential rise in rooftop solar PV. As a result since 2010 the utility has been experiencing a reduction in net load and revenue. The most dramatic drop in load is during daytime hours as is readily apparent by comparing the utility's load on clear and cloudy days which reflects the impact of residential solar PV.

Challenges facing utilities

The Rocky Mountain Institute (RMI) has published  an analysis of the economics of leaving the grid.  The central thesis is that solar PV and electricity storage enables consumers to leave the grid completely.  Solar PV has already reached grid parity in about 10% of the U.S. and declining PV prices suggest that this trend will continue.  However, without the ability to store electric power, consumers with rooftop PV still require the grid for nights and cloudy days.  The development of combined solar PV and batteries from Tesla and others promises to make solar + storage accessible for increasing numbers of consumers.  These consumers could form their own microgrid, either by themselves or with their neighbours and disconnect from the grid.  Alternatively they could become a source of dispatchable power and become an energy provider.  Increasingly they could find that either of these options is economically advantageous.  This has serious implications for local utilities because if this trend develops it will seriously erode the traditional utility revenue base.   It could also lead to a completely decentralized grid comprised of many microgrids.  From an operations perspective it increases the complexity of managing the grid compared to the centralized model in use today.

The other challenge for a utility with a high penetration of intermittent, distributed generation (DER) is load balancing, ensuring that generation meets demand. The German power industry, the U.S. Department of Energy, Hawaiian power utilities,  California power utilities, ERCOT, and many others are working to address this challenge.  HECO's presentation at DistribuTECH gave an insightful overview of what they are doing to meet this technical challenge.

Data sources

HECO collects a large amount of data from diverse sources. It is comprised of internal operational data plus data from esternal sources. It includes weather forecasts, data from customer sited PV, consumer/public resource data, phasor (PMU) data, renewables power quality, feeder data, irradiance meters, generator and substations power quality, SCADA, and vendor data such as forecasts and data from Independent Power Producers (IPPs). The data has widely different time resolutions ranging from real time such as PMUs sampling 30 times per second, SCADA reporting every 2 seconds, calculated gross load every 2 seconds, PV inverter data reporting every 5 minutes and aggregated monthly, weather forecast reporting every 15 minutes and aggregated daily, transformers collecting data every 15 mins and aggregated monthly, and smart meters reporting every 15 minutes and aggregated quarterly. Much of the data includes location. The data comes in different formats and has different levels of quality.

Fundamental to distributed energy are weather maps.  Maps of irradiance and wind velocity can be directly used to estimate or predict solar and wind generation in different parts of the islands.  Temperature maps are also important because the efficiency of solar panels depends on temperature.  The more accurate the weather forecasts are, the more predictable the generation.  Knowing that there it will be cloudy over the south of Oahu, but sunny over the rest of the island, or cloudy over the whole island, but with moderate to strong winds allows operations to estimate how much backup generation is required and where. 

EMS Integration

Data from new sources relating to distributed renewable generation are integrated with traditional sources in a distributed energy management system which enables operations and planning visibility into how distributed generation is impacting thte grid. It not only provides a view into the status of the grid (situational awareness) but also allows simulations to assess the impact of different future renewable generation scenarios on the grid. This helps to determine where backup generation may be required because high demand and low generation from renewables or curtailment when there is excess generation and insufficient load.  Location is fundamental to the analytics, because wind and irradiance varies widely over the islands.

Traditional input sources include SCADA and data from transmission, generators, protection, and so on. The new input data sources include GIS-based infrastructure models and data from the distributed generation sources. Also new is weather forecasting which enables predicting generation from renewable sources.  It also includes historical and actual weather reports, and satellite and other weather data.

HECO uses an IT platform from In2lytics (a spinoff from Referentia) which integrates all of these time series data with EMS, SCADA, CIS, and GIS and makes it available to operational users, planners, modelers and the public. in2lytics is a high performance time series database that is specially designed to provide instant data accessibility for planning and operational decision making.  in2lytics enables load, query, analysis using MATLAB, and sharing of the "big data" required to monitor and manage today’s complex electric grid. in2lytics has native high performance interfaces for MATLAB in addition to programming languages.  It translates data from different sources into its internal time series database. All data is archived in a spatially enabled time series database for future analysis. Matthew said that in2lytics is able to analyze large volumes of historical time series data very rapidly. For example, a longitudinal analysis on two years worth of archived data can be run in an hour or two.

Applications

Accounting for renewables requires a lot of data. Some areas that use this data include generation planning, load forecasting, distribution planning, and contract administration. One of the first applications is a customer facing web site called Renewable Watch - Oahu that reports the total renewables generation, solar and wind, on Oahu in real-time.

An example of an application that will be used for contract administration beginning next month estimates how much power is lost from Independent Power Producers (IPPs) in the case of curtailment. It uses weather information to estimate wind speed and irradiance meters to estimate wind and solar generation potential.

Another analytical application was used to study how solar PV variability affects transformer tap changes. This application allows HECO to relate frequency of tap changes to solar PV variability and to determine which transformers are most affected by solar variability.

HECO has a number of projects with the Department of Energy and other partners.

1. Department of Energy's Integrating System to Edge-of-Network Architecture and Management" (SEAMS) A federally-funded research initiative on high penetration grids which aims to streamline grid planning and operations for utilities in regions with high concentrations of distributed generation (DG) resources.

2. Department of Energy's Distributed Resource Energy Analysis and Management System (DREAMS) Department of Energy system to provide useful information about the distributed grid to grid operators.

   Partners: Siemens, Alstom, DNV, AWS Truepower, Referentia, Sunshot

   Advisory Team:SMUD, Southern California Edison, California ISO, TEP, APS, Kaua'i Island Utility Cooperative, Western APA

3. Making sense of synchrophasor data for utilities. The Synchrophasor Visual Integration and Event Evaluation for Utilities (SynchroVIEEU) with High Penetrations of Renewables. Accelerate the integration of synchrophasor information into production grade data visualization and analysis platforms/models. Leverage PMU capability at many substations – explore ways to tap resources and provide real-time visibility and real-time data. Make synchrophasor data accessible for efficient and reliable operations of a modern grid in light of high penetrations of renewable resources

   Partners: SEL, DNV GL, Referentia

4. Monitoring, planning, and modeling high PV penetration microgrid. Decision support for microgrid design and operation at Marine Corps Base Hawaii. The Office of Naval Research (ONR) has launched an ambitious program to demonstrate and evaluate energy technologies using Navy and Marine Corps facilities as test beds, known as the Energy Systems Technology and Evaluation Program (ESTEP). Program management is being handled by the Space and Naval Warfare Systems Command (SPAWAR) Systems Center Pacific (SSC Pacific). ESTEP was established in 2013. It brings together the Department of the Navy, academia, and private industry to investigate and test emerging energy technologies at Navy and Marine Corps installations. At present, ESTEP conducts over 20 in-house government energy projects, ranging from energy management to alternative energy and storage technologies.

Other projects include

• Remote light sensor technology at the Kahuku wind farm on Oahu to measure and forecast windspeed and direction to support reliability;
• Sonic detection and ranging technology to remotely measure wind speed and direction;
• Network of solar irradiance sensors to measure light and to make forecasts;
• EV Race to the Sun, a joint Japan-U.S. showcase event for electric vehicles,to take place on Haleakala;
• Battery storage technology development, including the JUMPSmartMaui project.

The ultimate objective for HECO is to become a company offering diversified services providing value to engaged customers, of course with satisfied regulators and sustainable costs and margins.  

Big data and spatial-temporal analytics

It was clear from this presentation that essential for HECO, or to any utility, with a high penetration of solar or wind are weather maps, forecasted, actual and historical because these enable the utility to project generation. The also allow utilities to analyze the historical record in conjunction with other data to discover trends that may help improve forecasting.  Collecting and analyzing temporal geospatial data is fundamental for distributed energy management.  This is in addition to the expanded application of geospatial technology for utility operations and planning in the smart grid era.

W3C and Open Geospatial Consortium announce new collaboration

The social value of the Web is that it enables human communication, commerce, and opportunities to share knowledge. One of World Wide Web Consortium's (W3C) primary goals is to make these benefits available to all people, whatever their hardware, software, network infrastructure, native language, culture, geographical location, or physical or mental ability.  This includes Web access from mobile phones and other mobile devices as well as the use of Web technology connecting intelligent devices in consumer electronics, printers, broadcasting, automobiles, and utility and transportation networks, to name just a few - what is becoming collectively known as the Internet of Things (IoT).

Everything has location.  In the context of the Internet of Things it is increasingly being realized that location is vital to the emerging field of connected devices.   To improve interoperability and integration of spatial data on the Web the World Wide Web Consortium (W3C) and the Open Geospatial Consortium (OGC) are announcing today a new collaboration which is intended to make the geospatial a fundamental component of the Web.

The Open Geospatial Consortium (OGC) is an international industry consortium of over 500 companies, government agencies and universities participating in a consensus process to develop publicly available interface standards. OGC standards support interoperable solutions that "geo-enable" the Web.

There is a gap between the specialized world of geographic data and that of the Web.  Bridging GIS systems and the Web will create a network effect that enriches both worlds.  At a March 2014 Workshop on Linking Geospatial Data, co-organized by OGC and W3C and in partnership with the UK Government Linked Data Working Group, Google, and the Ordnance Survey, participants discussed the challenges of integrating geospatial data and technology into the web.  Based on the challenges identified in that Workshop, the collaboration announced today will enable publishers of spatial data, providers of services that consume the data, and application developers to establish common practices and reduce the cost of integrating spatial data on the Web. The Geospatial and Web communities will document use cases and requirements with the goal of developing best practices for publishing spatial data on the Web.  The collaboration is also expected to result in new W3C Recommendations and OGC standards.

Smart cities market forecasted to reach over $ 1 trillion by 2019

Urbanization is a worldwide phenomenon.  Smart city technology is becoming an essential element in the development of the world's megacities.  For example, the new Indian government's just released budget includes an allocation for initiating the development of 100 smart cities.  Songdo IDB in Korea and Fujisawa in Japan are two smart cities already under development. China has 36 smart cities in development and a low carbon model city in Tianjin.  Singapore plans to become a smart nation by 2015.  Iskandar is Malaysia's first smart city.  The Delhi-Mumbai Industrial Corridor (DMIC) incorporates smart city concepts. 

According to the report "Smart Cities Market - Worldwide Market Forecasts and Analysis (2014 - 2019)", published by MarketsandMarkets, the global smart cities market is forecasted to grow from $410 billion in 2014 to $1.1 trillion by 2019 at a compound annual growth rate (CAGR) of 22.5%.  This includes smart homes, intelligent building automation, energy management, smart health, smart education, smart water, smart transportation, smart security, and related services.  Most of this activity is expected to occur in Asia and the Middle East.

Navigant Research forecasts that cumulative global investment in smart city technologies, including smart grid, advanced water monitoring systems, transportation management systems, and energy efficient buildings, could total $174.4 billion from 2014 to 2023, growing from $8.8 billion annually in 2014 to $27.5 billion in 2023.  Navigant forecasts that annual smart city technology investment in Asia Pacific will increase from $3.0 billion in 2014 to reach $11.3 billion in 2023.

What are cities already doing ?

In a recent study by Arup and University College London, Delivering the Smart City, the researchers analysed the spending patterns of eight U.K. cities; Leicester City Council, Manchester City Council, Leeds City Council, Portsmouth City Council, Sheffield City Council, Liverpool City Council, Bristol City Council and Coventry City Council.  These are medium-sized cities with populations between 200,000 and 760,000. 

This type of analysis has become much easier because of the open data movement in the U.K. which has generated significant amounts of accessible  data about government spending and procurement.   The analysis showed that the eight cities were spending on average 6% of their expenditure on information technology (IT). That's an average of £23 million a year on IT.  Four of the eight spent between 8 and 10% of their budgets on IT. To put this in context the proportion that these city governments are spending on IT is more than many industries and is comparable to the banking and financial services industry which spends on average 8% of operating expenditure on IT.

This research shows that cities are already investing a significant amount on something which is a foundation for smart city technology.  IT already underpins many services offered by city governments today including public security and health, transportation, public works, natural resource management, and permitting and licensing.

Other research also shows that city governments globally are spending a significant proportion of their budgets on IT.  Gartner analysed the IT spending patterns of 99 local governments across 80 countries in the U.S. and found that that IT accounted for 3.8% of their total operating expenses.  U.S. local governments, including 3,200 counties and 19,000 cities, spent approximately $34 billion on traditional IT goods and services in 2013. 

According to Gartner national and regional governments in the Middle East and North Africa will spend US $12.2 billion on IT products and services in 2014, up 1.3 % from 2013. This includes internal services, software, IT services, data center, devices and telecom services.  Saudi Arabia is investing in various digital government initiatives including King Abdullah Economic City (KAEC).  The United Arab Emirates (UAE) is planning multiple smart cities.

City governments are not the only organizations providing IT services to support city operations.  For example, installation of 5,000 smart meters in homes and businesses across London involved investment from private companies including EDF Energy, Siemens, Logica, as well as the electricity transmission and network operators, National Grid and U.K. Power Networks, the transport operator Transport for London, and the city government Greater London Authority.

The U.S. spending analysis also showed that local governments, in addition to more traditional IT products and services, are procuring cloud-based services to modernize citizen services and reduce operational costs.  An example is online portals for tax collection and business licensing.  These IT projects are usually part of wider modernization projects being carried out within departments rather than standalone technology initiatives at the city level.

UK city governments buy IT products and services from large vendors

City governments in the U.K. tend to purchase their IT products and services from large vendors.  According to the UK analysis of eight cities large and middle-sized vendors accounted for the overwhelming majority of IT spending (98%) with small businesses only accounting for 2%.

Michael Byrne receives national award for development of the National Broadband Map

Michael Byrne, who was Geographic Information Officer (GIO) of the Federal Communications Commission (FCC) and led the development of the National Broadband Map (NBM), has just been awarded the Citizen Services Medal for creating a series of online maps (National Broadband Map and derived maps) and geospatial visualizations that helped people make informed decisions about the country’s communications systems.  It was developed in seven months from a standing start, launched Feb17, 2011, and got half a million hits in the first ten hours it was up.

The Service to America Medals are awarded every year to federal employees who achieve amazing results and epitomize public service.

The National Broadband Map allows users to use a web tool to search broadband availability across the United States and compare real download speeds to advertised broadband performance. All of the data is available for download. The map was part of an FCC stimulus (American Recovery and Reinvestment Act of 2009) initiative to improve broadband coverage/speed across the U.S.

The development of the National Broadband Map is a fascinating story.  The business objectives of the National Broadband Map, which were were mandated by Congress, included searchability, interactivity, responsive on-line access and a very tight development timeframe.

The data collection part of the story was a challenge because the FCC had to rely on the states to collect the data.  The FCC ended up with data on 3500 broadband providers, submitted by 50 states, 2 territories and the District of Columbia.  Michael's team had to take all this data in many different formats and load it into the NBM database. The end result was a database covering 1650 broadband providers with 25 million records showing where broadband Internet service is available, the technology used to provide the service, the maximum advertised speeds of the service, and the names of the broadband providers.

Michael and his team's biggest technical challenge was the tight development timeline.  The second major challenge was ensuring fast user response time. 

Seven months is an incredibly tight development timeline for any enterprise system.  Early in the development cycle, the technical team, which was comprised of only eight people including Juan Marín Otero, the lead geospatial architect, decided that a RESTful architecture and open source geospatial software were the only way they were going to complete the project within the timeline because they recognized that they had to communicate directly with the developers if they were going to resolve issues in days, rather than months. 

After deciding on the open source software and open data stack, the system was developed in four and a half months.  It survived a deluge of traffic in its first hours and days.  The first day the map went live, the site received 158 million hits.

The open source software and open data stack used for the NBM includes

• Apache Web Server
• AspectJ
• Cloudmade
• EHCACHE DX
• jQuery
• Geolytics
• GeoServer
• GeoWebCache
• Google Analytics
• Google Geocoder
• Google Maps API
• Hibernate
• Hibernate Spatial
• HSQLDB
• Jersey JAX-RS RI
• Junit
• Mapbox
• Maven
• Mockito
• MySQL
• OpenLayers
• OpenStreetMap
• PHP
• Spring Framework
• Wordpress

Open source geospatial developers will recognize OpenLayers, Mapbox, GeoServer, GeoWebCache, and Hibernate Spatial (on MySQL) software, Google Maps API and Google Geocoder services and OpenStreetMap geodata.  The software incorporates many Open Geospatial Consortium (OGC) geospatial standards.  For example, Hibernate Spatial uses the Java Topology Suite (JTS) as its geometry model. JTS is a Java implementation of the OGC Simple Features Implementation Specification for SQL v. 1.1 (SFS).

Oracle Database Cloud Service supports Spatial

At Oracle Open World in San Francisco 2014, September 28–October 2 the Oracle Spatial and Graph SIG held a Meetup.  By way of background Oracle Spatial and Graph support geospatial data and analytics for land management and GIS, mobile location services, sales territory management, transportation, LiDAR analysis and location-enabled Business Intelligence. The graph features include RDF graphs for applications ranging from semantic data integration to social network analysis to linked open data and network graphs used in transportation, utilities, energy and telcos and drive-time analysis for sales and marketing applications.  

There were several very interesting discussions at the Meetup that revealed some interesting aspects of Oracle Spatial itselef and its interoperability with other products such as ESRI ArcGIS, Autodesk Map3D, and Bentley.  These are taken from a summary recorded by Nick Salem moderator of the Meetup.

The Meetup was attended by about 20 users came from industries such as rail, local city government, national land management agencies, and defense. The SIG Board and members of the Oracle Spatial product development team joined up with them in the Oracle Technology Network Lounge.

Some of the most interesting discussion topics include new features in Oracle Spatial 12c and real world experience interoperating Oracle Spatial with CAD and GIS products from other vendors.

• Oracle’s new Database Cloud Service offering now includes Spatial. 
• A new Oracle Spatial 12c SDO_POINTINPOLYGON function can be used to ingest large point datasets (point clouds) without the overhead of creating a spatial index.
• Bechtel has developed a way to use ESRI ArcGIS versioning with Oracle Workspace Manager (OWM).  They have developed in-house custom tools to usefeature-level metadata to be able to retrieve spatial content from their project and public spatial databases, as well as to compile the revision history of layers in the database to retrieve maps impacted by changes to individual features.
• Dan Geringer, Oracle Spatial Solutions Architect, advised that other large enterprises have also successfully implemented OWM with ESRI ArcGIS.
• Bulk processing and interoperability are two of the important advantages of Oracle Spatial for enterprises. Bechtel has large project-based deployments of Autodesk and Bentley CAD systems, and has worked with Autodesk in recent years to successfully write spatial data to their Enterprise GIS Oracle Spatial database from Autodesk Map 3D. They also have developed a GIS Desktop Procedure that details the required Autodesk client software configuration for Oracle Spatial and ESRI ArcSDE database connections. They will be updating it for use with Autodesk 2014 and 2015 clients.
• Latest version of MapViewer (11.1.1.7) supports HTML5 and JSon and the ability to work in disconnected mode. Standalone, web-based vector editing is also available. Wish Li and Jayant Sharma demonstrated  MapViewer on mobile devices, an Android phone and an iPad.