Between the Poles

Expanding role for geospatial data and technology in the utility sector in BRICS countries

The BRICS countries (Brazil, Russia, India, China, and South Africa) are geographically, culturally and economically diverse, but have one common point on their agenda — the rapid development of the energy industry as a national priority. This is owing to the fact that the primary contribution to the projected increase in world energy consumption (the International Energy Outlook 2013 projects 56% growth between 2010 and 2040) comes from the BRICS. The BRICS countries represent 36% of total global renewable power capacity and almost 27% of non-hydro renewable capacity in 2012. 

The BRICS face a wide range of challenges with respect to energy, the critical ones being universal electrification, especially in rural areas; rapidly increasing demand; the need to decrease energy intensity by deploying more renewable energy sources; reducing the high rate of energy losses, especially non-technical; and improving energy efficiency.

BRICS countries have been employing geospatial technology in various capacities in planning, generating, transmitting and distributing electric power. In a just published article in Geosspatial World we've provided a snapshot of the application of geospatial technology in these emerging countries and how we see it evolving in the future. 

Utilities in BRICS countries are uniquely positioned as they have been using GIS as an operational tool for some time and are familiar with its capabilities. At the same time, they are not encumbered to the same extent by old, legacy IT systems based on operational silos that remain a challenge for utilities in developed economies. Their work forces are younger, more internet savvy, and more willing to adopt new technologies.  The dawn of the data-driven, geospatially aware era promises new opportunities to deliver improved availability, efficiency and affordability. If utility leaders in the BRICS understand the vision and seize the opportunity, they could propel these countries into a leadership position in the electric power utility sector.

Are utilities prepared for managing "big data" and analytics

At the Second Annual Summit on Data Analytics for Utilities in Toronto, Brad Williams, Vice President of Utilities Industry Strategy at Oracle, presented recent results from a survey of of 151 North American senior-level utilities executives with smart meter programs.  The report Utilities and Big Data: Accelerating the Drive to Value is the result of the second annual study conducted by Oracle Utilities in their Big Data series.  They survey showed that the industry as a whole still hasn't really taken advantage of the data that is available, but there is a small group of early adopters who are pushing the envelope.

* Big data preparedness: Utilities are more prepared to manage the huge volumes of data that smart grids are capable of generating in 2013 than they were one year ago, but the majority still say they are unprepared.  This year 17 % of respondents said they are completely prepared compared to 9 % in 2012.

* Mutiple data sources:  In addition to smart meters 95% of utilities gather useful data from other sources such as outage management systems (68 %), SCADA (58 %), customer data and feedback (54%), alternative energy sources (12 %), social media (11 %), weather-monitoring systems (11 %), wholesale market data (11 %), and other grid equipment (11 %).

* Using smart grid data to improve customer service: How are utilities leveraging smart grid data to improve customer service today?

1. Provide customers their usage patterns (57%)
2. Implement demand-response programs (47%)
3. Establish new pricing programs (43%)
4. Target customers for new programs (40%)
5. Alert customers with usage spikes (26%)

The average utility has implemented just two of these measures.

* Strategic decision making: More respondents report using information for strategic decision making.  In 2013 11 % said they were using  smart meter and other data for  strategic decision making.  This is an increase from 4 % in 2012.

* Skills gap: 62 % of survey respondents said they have a "big data" skills gap.  This is a very serious problem. The preferred solution according to respondents is retraining existing employees. 

In addition to Brad many of the other speakers at the Data Analytics Summit mentioned this problem.

* Confidence in analytics: 70 % of utilities said they expect predictive analytics to improve revenue protection.  61 % saw analytics reducing asset maintenance costs.  55% expect an ROI on implementing analytics of less than 5 years.

* Benefits from analytics:  What utility processes are expect to benefit the most from predictive analytics?

1. Improving revenue protection (70%)
2. Reducing asset maintenance costs (61%)
3. Reducing asset replacement costs (57%)
4. Reducing infrastructure costs (54%)
5. Analyzing distributed generation (50%)
6. Reducing generation planning costs (41%)
7. Reducing generation operations costs (39%)
8. Assessing electric vehicle impact (26%)

Applying analytics to smart grid data

Brad outlined a number of areas which represent relatively low hanging fruit where significant benefits from applying analytics to smart grid data can be expected.  Brad emphasized the importance of spatial analytics because many if not all of these of these have a geospatial dimension.

1. Reducing non-technical losses - this has been one of the first areas where analytics has been applied by many utilities.  The payback is typically significant and immediate.
2. Targeting demand response - prioritizing customers for energy conservation and demand response programs using geospatial techniques such as energy density mapping
3. Distribution operations planning -  target customers with high peak load to help them reduce peaks by staggering powering on ventilation, heating/cooling and lighting
4. Transformer load management - identify transformers that are overloaded or underutilized
5. QA/QC data quality - improve the quality of connectivity information including phase
6. Voltage correlation - use analytics to link meters to transformers including phase
7. Energy modeling - analyzing usage patterns including including unmetered usage from street lights and other devices
8. Voltage deviation - identifying transformers with voltage deviating from the rated voltage by 2-3% or more
9. Geospatial outage frequency analysis - analyzing outage patterns geographically
10. Predictive analytics for electric vehicle adoption - identifying PEV owners and predicting demand pattterns to ensure adequate transformer capacity  is in place

A very interesting point that Brad made is that there are really two types of analytics.  Measuring and analyzing things we know - for example, the key performance indicators that are already on many dashboards - and the things we don't know.  The latter is where we can find significant opportunities for improving performance or reducing the frequency and duration of outages where we really don't have any preconceived notion of what to expect.  This is an area where different and  more sophisticated analytical and visualization tools for analyzing relationships and looking for patterns can be very valuable.

Evolution of data analytics at an electric power utility and the importance of data governance

At the Second Annual Summit on Data Analytics for Utilities in Toronto, Lincoln Frost-Hunt, Director of Enterprise IT at Hydro One, gave a riveting presentation on how one utility has evolved from a company that in the past tended to use "gut feel" decision-making to a company that is increasingly relying on data-driven decision making,

Hydro One manages both transmission, almost 30 000 km or 96% of Ontario's transmission network, and distribution, representing about 75% of Ontario, mostly rural but some urban as well.  The total value of their assets is C$17 billion and they manage C$3 billion investment in their assets every year.

Data governance

Lincoln emphasized that a key prerequisite for a successful data-driven analytical approach is quality data.  Bad data means bad decisions.  They found that when they started to make investment decisions based on their analysis of their operational data, the deficiencies in their data sets immediately become apparent.  For example, they found that only 76 % of their connectivity data was reliable, the remaining quarter was unusable.  To ensure the necessary level of data quality and reliability data governance (processes that ensure that important data assets are formally managed)  was a key underpinning of Hydro One's analytics strategy.    They implemented major projects to cleanup their connectivity and other data.  Realizing that data quality is perishable, they also implemented a data governance policy and focused on improving business processes that produced bad data such as the processes for managing as-builts, which I have blogged about on multiple occasions, most recently here.  In addition to ensure that their data quality level does not degrade overtime and that it is maintained at the required level of reliability, they have implemented a system of quarterly audits.

Data analytics

Lincoln described several areas where Hydro One's data-driven approach has yielded value.

Customer billing

Since this is Ontario, all of Hydro One's customers have smart meters.  Hydro One processes 65 000 bills per day.  This is a key essential process because this is a major source of revenue.  A certain proportion of these get kicked out because of problems.  Since seven years ago when they first installed smart meters, tracking down the problems that caused these bills to be rejected was a slow manual process that used a lot of resources.  In the last two years Hydro One has implemented an analytical tool set that allows them to resolve these problems and get the bills out to the customers much more rapidly.  Since bills generates revenues, the value derived from the analytical approach was immediately apparent to managment and helped gain support for the data-driven approach to decision-making.

Asset management

Improving utilization of assets is a key objective of the regulator, the Ontario Energy Board (OEB).  Hydro One took an analytical approach based on modeling equipment failure so that they could estimate the probability of failure based on external conditions such as age and average load.  This allowed them to generate risk curves which enables them to demonstrate to the OEB that their investment strategy had found the sweet spot between the cost of maintenance and reducing the risk of major failure.  Geospatial technology was a key part of the asset analytics toolset that  helped them optimize the lifecycles of their assets.

Workflow optimization

Like most utilities, Hydro One is facing the challenge of an aging workforce, so improving productivity is a key corporate objective.  This is an especial challenge for Hydro One because of its largely rural service area.  Sending a crew to resolve a problem or respond to a service request could require hours of driving to the site and back.  Putting together "work bundles" comprising all the service requests and trouble calls in a geographic area reduces the time the crew spends on the road and maximizes their time actually resolving issues and servicing requests.  To do this they installed GPS, communications and ruggedized tablets on every truck so they knew where every crew was and the crews had access to the information they needed.  They also were able to show all trouble calls and service requests on a map for a three month period so they could easily assemble work bundles by geographic areas.  Another major benefit they found was that when they received an emergency trouble call, they could determine immediately which crew was nearest, thus minimizing travel time.  They also invested in route optimization so that the crew knew the most efficient route to follow.  The bottom line is that crews spent significantly less time driving and more time resolving trouble calls and responding to service requests.

Enterprise architecture

Lincoln described Hydro One's current enterprise architecture as comprised of three major enterprise systems with many heavily customized point solutions.  I suspect that many utilities would find themselves with a similar architecture.  Each point solution is a silo and the enterprise systems are big silos.  Whatever interoperability there is has either been customized or acquired from a third party by Hydro One.  Every time a vendor upgrades a point solution, Hydro One has to retest the links to other systems.

They evaluated three alternative architectures ranging from a best of breed approach with many point solutions focused on automating key business processes to a single enterprise system with bolt-ons or point solutions designed to run with the single enterprise system.  They decided on a hybrid approach, a single enterprise system with minimal point solutions when justified by incremental value.

Basically Hydro One relies on SAP for key business functions like ERP, customer information management, and asset management.  Their analytical tool set is Business Objects, another SAP module, integrated with a GIS designed to integrate with SAP. 

As an example of a bolt-on, they use an advanced visualization tool (Space-Time Insight which is an SAP partner) as part of their analytics tool set.  Lincoln said that visualization had really changed people's view of analytics because it enabled them to visualize operational data in space and time in ways they had been able to do before.  Visualizing based on geolocation, time and network topology has helped them gain insights about their operations that they would never have seen by looking at Excel spreadsheets.   It also created a bit of a challenge for Hydro One's IT folks, because it created an IT traffic jam - everyone wants to use the visualization tool and no one wants to look at Excel spreadsheets.

Smart zone

Hydro one has implemented what they call a "smart zone" around Barrie, north of Toronto, where all equipment are intelligent electronic devices (IEDs) including automated substations, basically a distribution smart grid.  They intend to roll this out to the rest of their service territory.

Real-time big data

Hydro One realizes that the volume of data they need to manage is increasing astronomically, and with IEDs they need to be able to analyze and make decisions in real-time or at least near real-time.  Lincoln said that their next major  IT development focus is  to be better able to manage "big data" in real-time.  I have blogged on a couple of occasions (real-time monitoring smart devices for decision making and using social media streams to improve customer experience) about an example of technology that is designed to help do this.

Mining location-aware social media message streams to improve utility customer experience

In the view of many the utility industry is in the initial phase of adoption of a new generation of IT characterized by real-time, big data and analytics.  I blogged previously about a new application framework implemented to support Burlington Hydro's smart grid deployment.  Here is another, but quite different, example of the new generation of IT.

In a very appropriately titled session Taking Society's Pulse in Real-time at the Creating the Policy and Legal Framework for a Location-enabled Society conference in Boston, there were  demonstration that showed mining information from tweets and other socal media traffic.  One demonstration showed how a person sailing in the Caribbean could be tracked geographically as they tweeted from their boat.

As a more practical example, last week Michael Jander of AGSI gave a riveting webinar on a new, innovative location-aware approach for mining social media message traffic in real-time from Twitter, Flickr, Facebook and other social media streams.  MIke's background is public safety and security so his demonstration tended to focus on this area.  However, this technology is equally applicable to other sectors including utilities. 

The tool that Mike demonstrated monitors social media message streams in real time including  the senders’ locations. In the webinar, Mike demonstrated monitoring Twitter message streams.  A typical scenario involves an event, for example, a fire.   The operator identifies the location of the event on a map and draws a polygon around it to define an area of interest.  At this point all of the tweets emanating from a location within the polygon become visible to the operator, who can see and filter the contents of the tweets based on their contents.   For example he/she may be interested in tweets that mention "fire".  In addition it is possible to  view historical  tweets back as far as 30 days prior to the event.  Since the tweets have geolocation, either from on onboard GPS or via triangulation, it is also possible to track persons of interest by following their tweets. 

People can be notified over the social network if they are in danger.  They can be asked if they saw an incident.  If the incident is still occurring, they can be asked for additional information such as what they can see, hear or smell.  For example,  if they report smelling smoke that can alert first responders to the risk of a fire. Mike related that during one incident, 1500 people were tweeting what they were seeing.  During a major disaster such as Katrina this provides much better coverage over a broader area than would be possible if the only source of information is public officers.

Mike has had a lot of experience in the area of public security and safety and was able to describe real world scenarios where having access to this type of real-time information would be invaluable.  He mentioned the annual Santa Claus Parade in Toronto where this capability can help find lost children.  Define an area where the child was last reported and then tweet to everyone in that area a description and ask them to send a message if they see the child.  Or monitor tweets in that area looking for mention of a "lost child."

The tool allows the operator to search most social media traffic, filter the traffic geographically and by date and time,  identify specific social media users, and track users from their tweets or other social media.  In addition geofencing can be used generate alerts when a specific social media user enters a zone, or when a key word such as "fire" is used within a specific area,  and send social media messages to specific social media users (which don't have to be followers.)

Application to utilities

This approach can also assist electric utilities to search, review, and plot social media traffic when it relates to a customer concern, power outage or other types of incidents.  It also enables the utility's corporate communications staff to interact with all social media users, whether they are following the utility or not.  It enables the utility to communicate important messages to specific users, for example,  in a defined geographical area or region.   Energy conservation programs, notification of planned outages, updates on major outages and other types of information that are important to consumers can be communicated in this way.  Social media messages can be mapped and shared with other employees, for example, showing the location and photos of downed lines during a power outage.  The benefit of the is approach is that it provides utilities the ability to utilize social media for interacting with the public and optimizing customers' experience with the utility.

The graphics show

1. Quilt view of social media tweets during a storm event filtered by geospatial buffer around a electrical distribution feeder.
2. Selection and sharing of applicable social media tweets during a storm event again using a buffer linked to an electrical distribution feeder.
3. Selection of users within a defined geographical area for communicating an important notifcation.  Note that the users are not required to be followers.

Automating image-processing work flows for the Earth observation sensor web

Just about every industry from construction to utilities to roofers to insurance is being impacted, and often transformed, by the availability of high resolution Earth observation (EO) imagery from satellites, planes and helicopters, and UAVs.  Initially 2D, then 3D and in 2014 4D (2D/3D + temporal) imagery is available from many sources.  Imagery from international satellites,  low cost satellite constellations and UAVs is poised to dramatically reduce the cost of EO imagery.

An exponentially increasing volume of EO imagery is being captured every day from a variety of devices. The Committee on Earth Observation Satellites (CEOS) reports 286 Earth observation devices in orbit. The next Digital Globe satellite Worldview-3 scheduled for some time in 2014 will be capable of 31 cm resolution and can store two terabits of data between downloads. Two start-up satellite companies have already started putting satellite constellations in space, that promise to provide high frequency revisits to every point of the Earth's surface at low cost.  Planet Labs and Skybox Imaging each plan to launch constellations of 24+ low cost satellites that will capture high-resolution imagery of every spot on earth many times per day.  They promise to deliver the first ever HD-video of any spot on earth,  allowing users to track changes—from traffic jams to deforestation—in near real time. 

A few years ago I remember that the total amount of imagery being downloaded daily from all EO satellites was estimated to amount to about a terabyte.  Now a single satellite could be responsible for a terabyte daily.   If you add advances in aerial photogrammetry and the rapid development of UAV-based photogrammetry, we are looking at huge volumes of imagery every day, probably approaching a petabyte.

All of this data requires processing before it is made available to customers and end users.  Currently this processing is time consuming and a significant proportion of it stil involves manual and semi-automated processing.  This results in delays between the time the data is captured and when it is made available to customers which can range from days to weeks or even months.  Customers are becoming impatient with the time it takes to process data and are increasingly pushing for near real-time availability.

At the PCI Geomatics Reseller Meeting in Ottawa Wolfgang Lück, whose company Forest Sense cc customizes image processing workflows and provides remote sensing training, gave an overview of the sensor web and how automating image processing is enabling near real-time imagery availability.

Wolfgang says that most of the sensor web including different types of satellites, ground-bases sensors, mobile devices and satellite monitoring systems called sentinels are in place.  

• EDRS Geostationary Relay and supercomputer
• Wide area sytematic Optic/SAR monitoring
• VHR SAR
• Optical VHR/hyperspectral
• Field sensors
• Mobile devices
• DRS (earth receiving station)

According to Wolfgang, the communications protocols and the standards (for example, the OGC Sensor Observation Service) are in place for these systems to talk to each other.  Most of the satellite systems are in space, with just the sentinel systems still needing to go up. These are wide area systematic Optic/SAR monitoring systems being launched into space by ESA.  On the ground there are receiving stations (DRS), field sensors, and mobile devices carried by people.  The mobile devices capture as well as receive information.

Imagery requires involves complex processing to make it usable.  With the huge volumes of data that the sensor web is capable of capturing this processing has to be automated if the data is to be available in near real-time at low cost.  There are many actors that need to be accounted for, but the most important actors are end user customers (machines or human beings) because customers determine the products that are to be delivered.

Typical image processing workflow

There are different work flows depending on the sensors that are used and the final product requested by the customer.  To provide a feel for just how complex this processing is and the challenges involved in automating these work flows,  Wolfgang went through a typical work flow beginning with raw satellite imagery and following through to producing an orthorectified image suitable for classification, for example to identify different types of vegetation.  Even for this pretty basic image processing work flow, there are many steps.

1. Ingestion of data from pick-up point
2. Binary data to supported format conversion
3. Relative radiometric correction and artifact removal
4. Band alignment
5. DN to ToA reflectance conversion
6. Haze cloud and water classification
7. Auto-ground control point collection
8. Orthorectification
9. DSM generation
10. Topographic correction
11. Spectral preclassification
12. Level 4 product generation
13. Image compositing / mosaicking
14. Delivery of data to pick-up point

In the past many of these steps were manual or semi-automated.  The important breakthrough that Wolfgang was able to demonstrate is that he has been able to automate the entire process, which means that end user products, whether orthorectified images, digital surface models or classified vegetation maps are available much faster and at lower cost.

From Wolfgang's perspective the key to enabling this to happen is an image processing platform that provides a modular architecture, robust scripting, "big data" management, support for a broad range of sensor models (cameras),  support for Linux as well as Windows, comprehensive image processing functionality, the ability to incorporate custom leading edge algorithms -  integrated with an accessible user interface and API.  In addition because this level of automation pushes the envelope, responsive support from the vendor is also essential.

In the example Wolfgang used a scripting language, Python or EASI, to implement a work flow primarily comprised of PCI-Geomatica modules, but also custom components implementing leading edge algorithms from current research.

The first step is format conversion.  The data often comes in some proprietary format which the processing engine may not support.  A converter is required to bring it into a format that the processing engine supports. 

The next step is radiometric corrections and artifact removal.  Forest Sense supports a wide range of systems put together by emerging space nations, such as African or Middle Eastern countries.  In the example, you can see two lines lines are artifacts.  New gains and biases have to be calculated for those lines and applied to the image.  Other examples are random noise detection and dropped line removal.

Another problem is systematic noise.  A fast Fourier transform (FFT) reveals low a certain pattern in the high frequency domain which is characteristic of  systematic noise.  We can detect that automatically, do a FFT, create a mask to filter it out, and then do a reverse FFT to reconstruct the corrected image.

The next correction is the flat field correction that ensures that the sensors are calibrated correctly relative to one another.

The next step is band alignment.  The different bands need to be aligned because they are acquired at slightly different angles by different sensors.

The next step is total atmosphere reflectance correction.  What the sensor actually gets is relative voltage, and that is converted to relative radiance, which need to be normalized to total atmosphere reflectance.  This allows images acquired at different times, or from different sensors to be compared and to be used for quantitative analysis.   But the effect of the atmosphere is still there.  We can do atmospheric correction, but it is a bit of a black art and applied incorrectly can introduce artifacts.  For this reason, many people in the domain prefer to use total atmospheric reflectance data because they can trust it. 

At this point the image can be classified.

The first step in classifying the image is haze, water and cloud classification.  To do this requires both blue and red bands. These bands correlate with one another very highly, but the blue band is affected stronger by haze and water vapour.  What is called a clear sky vector can be applied to the image to calculate the haze vector and remove haze.  The right of the side shows the haze that is removed from the image.

The next step is collection of ground control points (GCPs) so that accurate geolocations can be calculated.  Unlike many systems which collect GCPs even over water or cloud which introduces errors, Wolfgang only collects ground control points over land. 

GCPs enable automated orthorectification as well as extracting digital surface models (DSMs) automatically from stereo pairs. 

The next step is preclassification. Spectral preclassification requires topographic normalization using a digital terrain model (DTM).  The image has to be preclassfied into different surface types that have different BRDF (bidirectional reflectance distribution function) properties.  Light reflects in different directions at different intensities on different surfaces at different wavelengths.  This has to be done using bands that are not affected by topography, which makes this step quite complicated. 

The slide shows a Landsat image of a hilly terrain with a lot of shade and exposed slopes which need to be normalized topographically.  Traditionally people have used a range of vegetation indices, such as the normalized difference vegetation index (NDVI).  From comparison with the DTM, it is possible to see that the NDVI is affected by topography.  Although it is a ratio between bands, it is still possible to see shady and sunny slopes.  The reason for this is that there is a lot of scattering in the blue band and shorter wavelengths but no scattering in the red and longer wavelengths so the ratio between the red band and the infrared band is not a pure ratio because the red band is affected by scattering. 

The TVI index was developed to be  resistant to topography.  Of course it doesn't work in areas of full shade.  A DSM is used to to calculate which areas are in complete shade so they can be masked out.  At this point all of these techniques can be applied to generate an automatic spectral preclassification of the image which then can be used for  topographic normalization.  The skylight adaptive topographic normalization is based on techniques that Wolfgang developed using all of these preprocessing steps.  In the corrected image it is still possible to still see some shady areas, but most of the shadow has been removed. 

The next step is image compositing and mosaicing and the generation of Level 4 (geoinformation)  products.  These products might include vegetation indices, biophysical parameters, masks, pixel-based classifications or object-based classifications. 

The final step is to package the data and deliver the data to the customer, typically by transmitting the imagery to a pickup point where it is catalogued.

At this point the product is available for the customer to collect from the pickup point.

Wolfgang demonstrated that downloading raw data and transforming it into something usable by a farmer, construction contractor, or insurance adjustor requires many steps and a lot of processing.  This example of a typical image processing work flow shows just what nearly a petabyte of the raw imagery captured daily must go through to create something usable by the end user customer (people or machines). To get this to customers in near real-time is incredibly computationally intensive and involves rapidly processing massive volumes of data.  This is why algorithmic performance is critical and why multiple processors with hyperthreading and most recently GPUs (graphics processing units) are being harnessed to improve throughput.

LocationTech attracts open source geospatial projects for managing "big spatial data"

I blogged earlier about Google joining LocationTech as a strategic member (together with IBM, Oracle, Boundless( was OpenGeo), and Actuate). 

After lining up key strategic members, the next step was to attract open source geospatial projects which wanted to take advantage of the governance and intellectual property services including commercially friendly licensing that LocationTech/Eclipse provides.  LocationTech has recently received a number of project proposals that will put LocationTech on the leading edge of open source geospatial development.

SpatialHadoop

Hadoop provides scalable processing for huge datasets by using MapReduce, a programming paradigm for distributed processing, to build an efficient framework for processing large-scale data.  SpatialHadoop is an extension to Hadoop that allows efficient processing of "big spatial data".

GeoMesa

Accumulo is a robust, scalable, high performance data storage and retrieval system based on Google's BigTable design and built on Hadoop.  GeoMesa provides a foundation for storing, querying, and transforming spatio-temporal data in Accumulo.  It implements Geotools interfaces that enable Geoserver and other Geotools related projects to use Accumulo as a data store.

GeoTrellis

GeoTrellis is a general framework for low-latency geospatial data processing developed using Scala and Akka.    GeoTrellis was designed to solve three core problems, initially for raster processing, scalable, high performance geoprocessing web services; distributed geoprocessing services for large data sets; and arallelizing geoprocessing operations to take full advantage of multi-core architectures.

GeoGit

I blogged about this recently because it is a project to develop distributed long transactions for geospatial data. 

GeoGit is a distributed version control system specially designed to handle geospatial data efficiently. It takes inspiration from the source code versioning system Git, but has an approach suited to spatial data (currently Shapefiles, PostGIS, MS SQLServer, or SpatiaLite.) GeoGit efficiently handles very large binary data and is optimized for spatial data using a spatial index.

JTS Topology Suite

The JTS Topology Suite (JTS) is an open source Java software library that provides an object model for planar geometry together with a set of fundamental geometric functions. JTS conforms to the Simple Features Specification for SQL published by the Open GIS Consortium.  JTS provides the core component for vector-based geomatics software or can be used as a general-purpose computational geometry library.

Geoff (Geo Fast Forward)

Geoff is intended to enable existing Eclipse Rich Client Platform (RCP) applications to integrate into the geospatial domain using lightweight components to visualizing geospatial information on a geographical map embedded in the RCP applications.

GeoScript

GeoScript adds spatial capabilities to dynamic scripting languages. GeoScript provides a collection of modules based on the Geotools library for geometry handling, spatial data access, and vector feature rendering.  GeoScript is currently implemented in Groovy, JavaScript, Python, and Scala.

James Fee on remaining relevant as a GIS professional

At this year's GIS in the Rockies, James Fee gave an enthralling keynote targeted at GIS professionals on the topic of how to remain relevant in the age of change (or What GIS Pros Can Do to Keep Their Skills in Demand).

He went through his own discovery of maps and technology beginning with atlases and continuing with a lot of the key technologies of the past decades most of which I can remember such as personal computers (Atari 800), the Logo programming language, Turbo Pascal, Hypercard, Freehand, Hamster dance (which I don't remember), the first releases of Arc/Info, Arc Macro Language (AML), spatial databases, the Internet, Sun, Sparcstations and Solaris, ArcView and the Avenue scripting language, mobile devices, big data, real-time, Perl, Python, spatial analytics, UAVs and 3D cities.  His main point is that geospatial folks have always been at the forefront in adopting new technologies and that GIS has a history of pushing the envelope.  His recommendation for GIS professionals is

• A always
• B be
• L learning

He also concluded that a GIS professional is a programmer, not in the restricted sense of a software developer, but in the more general sense of someone who makes computers do things.  James is clearly a fan of scripting languages including AML, Javascript and Perl.  He sees Python becoming the world's primary scripting language ("Python is the bee's knees").  James is a major baseball fan and he asserted that If you like GIS, you like statistics and furthermore if you like statistics, you like baseball. He gave a very simple example of using Python and several Python mathematical libraries to generate an interesting perspective on current baseball standings.

To answer the title question, what should a GIS professional do to stay relevant ?  James asserted that if you

• Put points on a map and throw up a scale bar
• Perform geoprocessing without Python or Model Builder
• Have a job description of “Plotter Operator”
• Have no idea what “fuzzy tolerance” is

you will soon be out of a job.  On the other hand if you

• Embrace Python as your GIS tool of choice.
• Use Model Builder to automate your work flows.
• Learn new tools such as TileMill/Mapnik/PostGIS

(all open source geospatial I would add) your future is bright.

Convergence of technologies transforming surveying and other industries

At the South African Surveying and Geomatics Indaba 2013 today in Johannesburg today Trevor Ventor of Optron gave a very interesting presentation (by Peter Large who wsn't able to make it to Johannisberg) on the topic of the Our Changing World and the Role of Geomatics Practitioners.

. 

By way of a bit of history he mentioned theodolytes which have been around since 1587 and which have been the basis for modern surveying until the advent of GPS in 1989-94.  President Clinton turned off scrambling in 2000 and since then the impact of GNSS on surveying has been enormous.  For survey precision you need an Earth station.  A number of countries such as the UK have implemented national Earth station systems, which allows you to correct GNSS readings over the internet and achieve centimeter accuracy .

,

The Trimble Centerpoint RTX network which provides 4 cm precision in 40 countries worldwide was launched last year.  It is most interesting that it is not surveyors who are the biggest users of the Centerpoint RTX system but farmers, presumably for precision agriculture applications.

.

Convergence of technologies

Another interesting point that Peter made is that at the present time it is not so much new individual techologies that are having the biggest impact but the convergence or integration of multiple technologies.  An example is total stations which are a theodolyte and laser distance measurement device combined in a single piece of equipment.

. 

But on a larger scale the integration of positioning devices such as total stations, laser scanners, wireless and internet connectivity, "big data" management, and analytics is set to transform not only the surveying industry, but agriculture, utilties and other sectors.  Peter suggested that half of the world's GDP of $72 trillion is generated by industries that will be affected by the convergence of these technologies.   For example, precision agriculture has become a major growth area for Trimble.  It is also very interesting that Trimble, which like Hexagon used to be primarily a hardware company, now generates over half of its $2 billion or so annual revenue from software.

ESRI UC: Preview of a cloud-based geodesign app

At the ESRI User Conference Bill Miller, Director GeoDesign Services at ESRI, previewed  a web-based app to support the geodesign workflow.  The app is aimed at the planner and is designed to support land use planning, coastal land use planning, disaster response, and similar types of workflows.

What is revolutionary about this app is that it is not a traditional GIS product designed for GIS professionals with years of experience.  From what I saw it was intuitive, fast, and easy to use with no or little training.  Users can setup their own projects, use Esri or their own data, create plans(scenarios), assess and evaluate them, compare scenarios (ala Urban Observatory), invite people to collaborate synchronously or asynchronously, and share their work with others, all with no or minimal training.  It uses HTML5 and Javascript and so can run in a browser without a plugin on any device. 

Some of the things that Bill demonstrated included

• create/open a project
• invite other people to join
• draw points, lines, polygons freehand
• use symbol palette (land use types)
• set transparency
• choose/change basemap
• show slope, aspect, and elevation
• open scenario/create scenario
• copy polygons from one scenario to another
• open a dashboard
• customize the dashboard
• choose layers and weights for a suitability model
•  run a model, for example, to calculate suitability for open space conservation, or industrial, residential, commercial land use
• compare result maps generated from different scenarios side by side
• create reports by harvesting dashboards, comparisons, and metrics
• collaborate so that two or more users see edits in real-time in their browers on different machines
• synchronize map extents so two or more both users see the same work area

Currently it only supports 2D, but it appears to be the intention that 3D would be supported in the future.

ESRI UC: Geodesign projects in 3D

At the ESRI User Conference Eric Wittner, a 3D Technology Evangelist, at ESRI presented some Geodesign projects that involve 3D that ESRI has worked on.  This was very interesting because it provides a practical perspective on what ESRI considers geodesign to be and secondly how existing software tools can be used for Geodesign.

Geodesign

Based on my experience at the Geodesign Summit last January i've blogged about the different views of what geodesign is, so I won't reiterate that here.  Eric's perspective is practical.  Geodesign is design that considers geography. 

Geodesigning really isn't new.  People have been making this type of design decisions since we have been had buildings and towns.  Many cities are near water bodies for obvious reasons.  Some cities are not near water for other reasons, for example, to avoid mosquitoes and  malaria.  Eric provided a classic example of bad and good geosdesign in the U.S. by comparing a standard American suburban development with Frank Lloyd Wright's renowned waterfall house.

Architecture and GIS

He mentioned the early geodesign pioneer Ian McHarg who developed geodesign techniques before the term was invented.  McHarg’s book Design with Nature, published in 1969, pioneered ecological planning. Geographic overlay maps were not invented by Ian McHarg, founder of the Department of Landscape Architecture at the Universityof Pennsylvania, but were popularized by him within the architectural profession.   Some of the tools used by McHarg that would be familiar to GIS people include overlay maps, transects, diagrams, drawings, bird’s eye perspective, photography and block diagrams.  But McHarg did all this with paper, mylar and acetates because he didn't have the computerized GIS tools we have today.

Modern geodesign is a systematic approach to design that uses computer GIS tools to incorporate geography into the process of designing(planning) buildings and infrastructure.  Initially geodesign used 2D tools exclusively.  But now as 3D is being integrated into GIS tools, geodesign is also beginning to leverage 3D and some of the examples that Wittner showed used 3D. 

Geodesign as a process

From the examples I have seen, the focus of geodesign, at least in how it is used at the present, is on the planning phase, rather than the design phase of the construction lifecycle.  But like BIM processes, geodesign is collaborative, but it is perhaps more multi-disciplinary (involving a wider range of stakeholders including politicians and the public), multi-scale (involving different levels of government) and  iterative (typical in planning) than is typical of most BIM design processes.

As a simple infrastucture example, if you have ever been to Redlands you'll know that it is not a pedestrian-friendly place.  Roads are wide, cars go fast, often there are not sidewalks, or sidewallks disappear into rail lines.  On a hot day, you're likely exposed to direct sun.  Wittner gave a very simple example of how geodesign could be used to encourage people to walk or bicycle in Redlands  by narrowing roads, adding sidewalks, and planting trees to provide shade.  But the key to success in making simething like this happen is that you have to get all the stakeholders participating in the planning process.

City of Honolulu

Another example is the City of Honolulu where the challenge was to accomodate 140,000 new residents without significatly increasing Honolulu's infamous traffic problem and suburban sprawl.  They knew they had to grow up rather than out.

What was being proposed was a light rail system.  They looked at each station, did a 2D walkability analysis to identify areas where Americans would be willing to walk.  They found that they could acoomodate 80% of the 140,000 new residents in areas that were within walking distamce of light rail stations.

They then created a 3D basemap, by using a rule-based procedure to create fairly realistic (textured) models of existing buildings.  Then based on zoning, they created 3D models of potential buildings to show what the maximum development would look like based on zoning restrictions.  But the City wanted realistic representations of buildings, so the project team automatically textured the buildings based on building type.  They found that with a rule-based approach it was fairly easy to accomodate changes made by the contractors and the City.

.

The City then wanted to show what the alternative development would look like if they didn't densify, and allowed more sprawl and worse traffic.  From this analysis they found that by densifying they preserve 105 000 acres from development.

.

In this example, 3D was used primaily to be able to visualize for city leaders and the public what the proposed development would look like.  This is fairly typical of the use of 3D for this type of project.

Philadelphia Logan Square

Wittner described a very interesting geodesign project In Philadelphia where the objective was to rezone a part of Philadelphia to encourage economic development.  For this project 3D was esentail to the analysis.

.

Again, this involved an interative process. They first assessed the existing buildings and spaces, created 3D models of the existing buildings, and then did a multi-criterion suitability map to assess development potential.   Their first attempt showed very little potential for development, primarily because of limited accessability.

.

They then extended a light rail system into the area and found that it enabled much more development potential.  They then created 3D models of potential new buildings and looked at times to walk to the light rail stations from different floors, opportunities for residential, commercial, and retail development at different locations and on different floors with different zoning policies, for example, zoning for higher buildings. 

.

For residential potential they used criteria like proximity to retail and light rail, parking, and quality of view.  For quality of view, they identified historic buildings, waterfront, and other features that would contribute to what people would like to see.  Because they could calculate the square footage available on different floors and at different locations, they had a metric for assessing the development potential for residential, commercial, and retail under different zoning policies.  For example, higher buildings tend to favor more residential because the quality of view is better on higher floors.

. 

This is an example where 3D is essential part of the analytics of the project.