Between the Poles

Sharing Spatial Data

One of the things that I find fascinating are the different ways human society benefits from sharing information.  Let me give you an example in the geospatial arena.  GITA conducts an annual survey of its members.  The 2005 report collected information from 294 members which are utility companies from the electric, gas, pipeline, public, telecom, and water sectors.   The survey asked participants if they were using more than one geospatial platform, and secondly, if they were, how did they share data between the systems.

Over 60% of participants reported were using more than a single platform.  This is up considerably from the previous year's survey, where between a third and a half reported using more than one platform.  For both years two thirds of the companies with more than one geospatial platform were sharing data between them.

But even more interesting, although third party translation software was identified most often as the method for data sharing, an increasing number of participants said they were using "functionality native to their selected GIS" and did not require third party translation software. 

I would suggest that we are seeing an increasing number of organizations who are centralizing their network infrastructure in a relational database.  This is now possible because most relational databases are spatially-enabled including Oracle, DB2 (IBM), Informix (IBM), PostgreSQL/PostGIS, and even MySQL if you use the ISAM storage engine.  Oracle is still the functionality and market leader.  In the last three surveys that IDC conducted, over 80% of companies using a spatial database were using Oracle (Locator or Spatial).

 

It is interesting to see how sharing spatial data has evolved over the years from proprietary files through proprietary schemas for storing spatial data in relational database management systems, to the current state where you can store just about any spatial data with topology in a modern object-relational database management system.  The advantage of ORDBMS's is that data is accessible is through an open query language, SQL, and an open interface standard such as ODBC, JDBC, and OLEDB. 

[Caveat:  But everything is not unalloyed bliss.  If you consider the most basic geospatial data types, points, polylines, and closed polygons, the news is pretty good.  Most geospatial vendors support these. We have demonstrated interoperability (read and update) with Autodesk Map and products from several other vendors.  But if you look at cartographic text, the Open Geospatial Consortium is currently working on this and there does not yet appear to be a consensus.  If you consider more complex things like polygon and linear topology, long transactions, and linear referencing systems, it is not yet possible to say that all vendors support a single model.]

 

The traditional architecture that organizations with more than one geospatial platform employ results in information silos.  Engineering, records, and planning typically used applications from different vendors each with its own proprietary data store.  Paper (data re-entry) and file sharing were used to exchange data.  A typical example is one we have described already, engineering design typically used a CAD application and stored the data in DWG or DGN files.  The records or GIS team used a traditional GIS and stored spatial data in proprietary data files supported by the GIS vendor.  The planning department probably used another application and stored their data in another set of proprietary files.

An open spatially-enabled data store makes it possible to share data between applications including applications from different vendors.  This means that many people with different roles in the organization can share a single point of truth, rather than the more typical situation where the engineering, records, and planning teams use applications from different vendors each with its own proprietary data store.

What this simple concept enables is everyone in different departments in the organization sharing the same information in the same repository, which eliminates data redundancy and the associated redundant processes.  This includes the engineers in the central office, the financial people responsible for managing assets, the install and repair people in the field, surveyors in the field, and outage management staff.

The benefits of a centralized spatially-enabled database are the concrete business benefits that we have outlined in previous blogs about ENMAX and Telesp, better quality of service to customers and reduced costs.

 

 

 

 

Infrastructure Management in Brazil

Managing network infrastructure (often called outside plant) in the telecom industry is challenging.  I'll explain why.  For many years I have used a rule of thumb to estimate the complexity of modeling network infrastructure for different industries.  For water and gas the network infrastructure is similar, pipes and valves, and you may have 30 or so different types of equipment which means roughly 30 classes of objects or 30 tables in your RDBMS schema.   Waste water is similar, but you have to worry about z, because gravity is important.  For electric power grids (secondary), things are more complicated, but still manageable.  You will find that you will need 100 to 150 classes.  But for telecoms, especially in this day and age when copper, fibre, coax, and wireless media are used, you will need hundreds of classes and it is very challenging to develop an application to manage network infrastructure of this complexity.

One of the first companies to attempt to develop software to manage telecom network infrastructure was Telebras, which had a division similar to Bell Labs in the US or Bell Northern Research in Canada. At Telebras Geovane Magalhaes was the technical architect and lead on that project, which resulted in a a system that was remarkable for its time, an integrated outside plant management and provisioning system called SAGRE, that stored spatial data in a relational database (Oracle).  At the time Telebras was the only telephone company in Brazil, and most of the operating divisions of Telebras adopted SAGRE for outside plant and provisioning.

Since then Telebras has been broken up and Telebras's IT division is now called CPqD and their outside plant application is called  CPqD Outside Plant Management (http://www.cpqdusa.com/solutions/outside.html).  CPqD OSPM has been upgraded to support the latest Autodesk and Oracle technology.

One of the operating divisions of Telebras was acquired by Telefonica and renamed Telefonica Telesp.  Telesp runs the telephone network for the State of Sao Paulo, which accounts for 60% of the Brazilian GDP.   Telesp has over 10 million customers and over 15 million access lines.  Every month they process hundreds of thousands of work orders generated by people who want a new phone, to get a new line, or need a repair.   

Several years ago they recognized they had some serious problems, the most important of which from a customer service and competitive point of view was that it took weeks for a customer to get a phone.  The basic problem was that the work flow for provisioning a phone was inefficient. 

Three years ago Telesp initiated a project that was recently completed to automate the entire process of provisioning.  Marcos Faria of Telesp was the technical lead for this project and the Telesp IT team accomplished this by a combination of a centralized data repository for all spatial and other data in Oracle, integration with SAP based around the Oracle RDBMS, CPqD's Outside Plant Management with integrated provisioning support, and resolving the as-built problem.  The result was that they were able to reduce the cost of provisioning by 40%, which is a huge reduction and an incredible achievement.  They also eliminated their backlog and were able to reduce their provisioning life cycle to days from weeks.  Not only were the results of the project impressive, they calculated that their ROI was 100% over the three year period of the project.

If you are interested in more information about this amazing achievement, Geovane Magalhaes prepared a presentation about Telefonica Telesp for GITA2005 in Denver.

Map India 2006

I was fortunate to attend Map India where I gave presentations on some of the themes of this blog.  This year Map India was especially interesting for several reasons.  First of all the theme of the conference was "Finding Mainstream", which brought home to me how all over the world people have recognized that geospatial has become part of mainstream IT and is no longer a niche.  This is one of the reasons that I think 2005/2006 is such an important time for geospatial technology.  Dave Sonnen, who prepares IDC's annual spatial information management report, has been predicting for years that spatial-enabling IT was going to become a much larger market than traditional GIS and would at some point overtake traditional GIS.  2005/2006 is the first time when it feels that this crossover is imminent.

Another reason that Map India was so interesting is Delhi, which is where MapIndia took place this year.  Delhi and the Delhi area were at the centre of the Mughal empire and Delhi is full of remarkable monuments that have endured from that period.  One of the most remarkable is Humayun's Tomb, which with its extensive garden, is located right in Delhi.   It was the first what is called garden tomb in the Indian subcontinent and was a precursor of the more famous Taj Mahal.

Another reason that Map India was so interesting is that I had an opportunity to chat with Sanjay Kumar and Maneesh Prasad of GIS Development, which is one of the organizers of the event.  Sanjay reminded me that Autodesk had been a sponsor of the first Map India nine years ago and that next year will be quite special, because it will be Map India's decenary .  We chatted about some of the interesting trends in geospatial technology and in particular about open source geospatial, to which the January issue of GIS Development was devoted.  To me this is another important reason that 2005/006 is such an important period for geospatial technology because we are seeing a recogniton of the maturing of open source geospatial technology.  At the end of 2005 Autodesk announced that it was open sourcing the next release of MapGuide, called MapGuide Open Source.  And then early in February of this year a number of folks active in the open source geospatial world including people representing GDAL, GeoTools, GRASS, Mapbender, MapBuilder, MapServer, OSSIM and MapGuide Open Source got together to create an open source geospatial foundation (www.osgeo.org), which I recommend that you check out.  I like to draw a parallel between where web mapping is today and to the formation of the Apache Foundation, when most of the major IT vendors recognized that it made more business sense to support the Apache web server project than for each vendor to develop its own proprietary web server.

One of the events that distinguished Map India from other geospatial events is the evening of ghazals that was organized for the attendees.  This is an experience you really have to go to the Indian subcontinent for.  Ghazals are a form of poetry that is sung to the accompaniment of traditional Indian instruments and other instruments, in this case a violin, which you should be able to make out on the right in the photo.  I believe ghazals were historically written in Urdu, but you can find them in a variety of languages including English (http://ghazalpage.net/).

How ENMAX Addressed the Infrastructure Management Problems

In my last blog I described four of the major problems organizations responsible for maintaining network infrastructure face.  These include the "as-built problem", providing field staff access to timely, reliable infrastructure data, enabling field staff to provide feedback in the form of red-lines or markups, and business processes that are not designed to optimize data quality.

This time I would like to describe what Brad Lawrence of ENMAX Power Corp has done, because he has been a leader in first recognizing these problems and secondly in implementing a practical solution.

Quite a few years ago the City of Calgary, which is in western Canada and has a population of around 800,000, in a very forward looking bylaw required all utilities and telecoms within the city to submit data showing the location of their network infrastructure to a consortium call JUMP, Joint Utility Mapping Project (I think).  The actual data format required was DGN files, which tells you how long ago this was.   What this meant for ENMAX was that they had to create digital information showing the location of their network infrastructure, conductors, transformers, poles, and other equipment.  In a very forward looking decision, they decided to store all of the digital information including location in an Oracle RDBMS (relational database management system), which means that right from the beginning they had the concept of a centralized network infrastructure data store.  This made things much easier for them later. If I remember it was a multi-year year project to collect the information.  They quickly realized that the digital information would allow them to provide timely, reliable information to their field staff (linesmen and cable locate staff).   Besides generating DGN files every month, they decided to also generate maps for their field staff to use in the field.  Initially these were microfiche and every truck was equipped with a microfiche reader.  They had to produce and maintain 50,000 microfiche maps, so this was expensive ( I think it cost over 1 $million annually) but it meant that the field staff had timely and reliable maps showing the location of ENMAX's facilities.  This made their job easier and made them much more efficient.  The other thing that ENMAX did which I find absolutely remarkable in its prescience is they instituted a policy that red-lines or markups from the field were to be treated as high priority by the records staff and they guaranteed that they would be captured in the Oracle RDBMS within 24 hours.  Just to make this clear, I'll repeat 24 hours, not the days, weeks, or months which would be typical of most utilities and telcos (if they have a feedback loop at all).  Most recently, ENMAX has implemented a new integrated solution that addresses the as-built problem, so that construction drawings no longer need to be re-digitized into the central Oracle RDBMS.  Previously ENMAX had used Microstation for engineering design and generating paper construction drawings.  The drawings were used by the construction team to build new facilities and were returned as marked up drawings to the records folks who digitized them into the Oracle RDBMS.  If I remember there was team of six or so people responsible for this and of course there was a long backlog.  The backlog meant that the Oracle RDBMS was always out of date. 

Let me suggest some of the business benefits that ENMAX realized.  First of all, improved quality of service.  A concrete example that every utility or telco can relate to is cable locate ( or call before you dig.)  Prior to implementing their geospatial system, ENMAX maintained a fleet of 20 or so vans and roughly the the same number of staff whose sole responsibility was cable locate, which means going out to the site of a proposed excavation to identify where ENMAX's electrical cables were so the excavators could avoid digging them up and causing a power outage.   I would estimate this might have cost ENMAX about $1 million a year in direct costs.  in addition for very 1000 calls they received from folks planning to excavate, on average there were nine incidents or "hits", where the excavator dug up a cable and caused a power outage.  This led to indirect costs associated with restoring power.  After ENMAX implemented their geospatial system, the "hit" rate dropped 36-fold to 0.25 per 1000 calls.  In addition because there was a single JUMP data store containing the location of all facilities in the city, they were able to reduce their cable locate fleet to something like two vans instead of 20, which saved ENMAX a lot of money.   

Secondly, the regulator was much happier because they saw an improvement in the quality of ENMAX's facilities data.  Every year they require ENMAX to undergo an audit, where an independent auditor takes a sample of ENMAX's facilities data out into the field and compares it with reality.  The auditor reported something like 99.6% reliability, which is remarkable given that 70-80% is more typical of the industry.  The regulator likes this because they have confidence that ENMAX knows where its facilities are.  Because they can provide better information, for call before you dig, for example, there are fewer outages and the public is better served.

I think you'll recognize that Brad and the ENMAX team have addressed the four major problems that I described in my last blog.  A couple of questions come immediately to mind.   How has ENMAX been able to recognize the problems so early on ?  Secondly, how have they managed to get everyone in the organization onside and supporting these very innovative changes.

I would suggest a few key factors that enabled this to happen.  First of all enlightened government legislation was a key instigator.  Secondly, there was a single division headed by Brad within ENMAX that was responsible for both records and engineering design.  In many utilities and telcos the engineering design team and the records team, often called the GIS team, have different reporting structures, and there tends to be little cooperation between the two.  Thirdly, Brad is an ex-linesmen and understood from first hand experience what the field staff needed to help them do their job effectively and efficiently.  Finally, IT technology is the key enabler that made it all possible.  Some of this was implemented quite a few years ago starting in the 90s, so ENMAX had to implement, usually through consultants, heavily customized solutions.

In this day and age with the benefit of the experience of Brad Lawrence and ENMAX it is much easier to design the technical architecture and implement such a system using commercial off the shelf (COTS) software.  To the right I've shown a technical architecture that, if implemented together with the requisite organizational structure and support, would address the four major problems that I outlined in the previous blog.  The components of the architecture are a central geospatial RDBMS such as Oracle, a server cache of DWF sheets or tiles generated from the RDBMS that contain maps of the land base and facilities, a standalone viewer with red-line support for viewing the DWF sheets in the field (Autodesk Design Review), an editor for making changes to the central geospatial RDBMS based on the red-lines returned from the field (Autodesk Map3D), a synchronizer that enables the DWF tiles to be efficiently distributed to the field when the field staff connect to the network (for example, we have been using Afaria from iAnywhere), and a DWF cache manager which ensures that the DWF cache is always current.

If you are interested in more information about ENMAX's geospatial solution, Brad has given presentations about this at GITA and GeoAlberta.  In recognition of their achievement in this area, ENMAX won an Innovator of the Year Award from GITA two years ago which Brad accepted on behalf of ENMAX.

Common Network Infrastructure Problems

Hello and welcome to a new blog that will have two key focuses, geospatial IT and how utilities, telecommunications firms, transportation organizations, and others who have to manage network infrastructures have used it to address real-world business problems.  I spend a significant part of my time talking about these topics at geospatial and infrastructure conferences in various parts of the world and I am very excited about having a another forum like this to discuss these things with you, so I am looking forward to your feedback.

I spend an even greater portion of my time talking to our largest customers in various parts of the world, and one of things that I find most fascinating is the commonality in the problems that utilities and telecoms and other folks managing network infrastructure are facing in different parts of the world.  Equally fascinating is the different approaches that people in different geographies take to solve these problems.

Let me outline some examples of common problems.  If you look at the process that a utility or telco uses to design and manage network infrastructure, you will find more frequently than not that the process is similar to what I've outlined in the illustration to the right.   There are some serious problems with this process that inhibits the ability of a utility or telco to improve the quality of service to its customers and to reduce operating costs.

Problem 1: As-built problem

I'll call one of the problems the "as-built problem".  This problem is often the result of having different teams responsible for engineering design and maintaining as-builts or records.  The engineering design folks typically use a CAD tool to design network infrastructure.  The records folks use a traditional GIS application to maintain a record of the network infrastructure after it is constructed.  The problem arises because almost invariably the engineering design deliverable is a piece of paper called a construction drawing.  (It is actually paradoxical that the engineering design team uses sophisticated digital design tools, but the end result is a very old-fashioned piece of paper.  If you are familiar with mechanical design, there is a parallel situation with what are called aperture cards.) The construction drawing is used by the construction contractor to build the network infrastructure.  After construction is complete, the construction drawing comes back to the records team as a piece of paper called an as-built, which may have been marked up or red-lined because the contractor may have had to make minor changes to the original design.  A member of the records staff digitizes the construction drawing into the records database.  At most organizations there is a backlog of as-builts waiting to be entered into the records database, and the backlog can stretch anywhere from 45 days to several years.  Just an aside.  I visited a large telco a few years back and asked them about their as-built backlog.  Their reply was interesting because they said they didn't have a backlog.   I expressed my surprise and and asked the question a different way, how long does it take as-builts to get into the records database.  Their reply was 45 days, which is really quite remarkable for a telco of their size, but underlines how serious the backlog problem is.

Problem 2: Providing accurate and reliable infrastructure data to field staff

Most utilities and telcos have a lot of people in the field.  They are sometimes called install and repair staff, troublemen or linesmen, or field maintenance staff.  They are usually customer-facing. In fact in many cases, they may be the only people you meet face to face from your local utility or telco.  A large telco in the US or UK serving tens of millions of customers, for example, will have tens of thousands of field staff.  A large power company serving millions of customers will have thousands of field staff.  The fascinating thing is that the field staff usually know more about the location, detailed properties, and condition of network infrastructure than anyone else in the organization because they work with it daily.  Not only that, but the field staff need accurate, reliable data to be productive.  When they go out to a site to install or repair equipment, if they know in advance what equipment is up the pole or down the manhole, they can make bring the right equipment and material.  An interesting statistic that many utilities and telcos capture is "returns", which is the proportion of work orders that require one or more return visits.  At many utilities and tel cos returns amount to between 25 and 30% of all install and repair jobs.  This is expensive and it means that instead of a crew completing on average 5 work orders per day, they are only able to complete 4 work orders.  Multiply that by the number of field staff and you can see how expensive this is. 

Problem 3: Limited support for feedback from the field

In many utilities and telcos there is either no feedback loop from the field to records or if there is, data such as red-lines and markups from the field are treated as low priority and are seriously backlogged.  This means that the field staff who need reliable data to do their job productively and know more about the overhead or underground network infrastructure are discouraged from feeding back corrections and updates that would lead to better data quality.

Problem 4: Data quality

A result of the first three problems I described is that the network infrastructure data in the central records is often out of date and inaccurate.  This is the result of backlogs, errors introduced during data entry of construction drawings, and the lack of a feedback loop from the field.

The four problems I've described are common to many utility and telcos around the world.  What I want  to do is tell you how some utilities and telcos are addressing these problems in very innovative ways.  I also want to tell you about some transportation customers I met with recently at a conference in Montreal today and at MapIndia two weeks ago.

Also I want to tell you why I think 2005/2006 is a turning point year for geospatial technology and why I think this is the most exciting period since the introduction of the 8080, for those of you who can remember.

One thing I will pass along that was confirmed by Oracle at the conference in Montreal is that Oracle XE, which is the free version of Oracle RDBMS, will include Locator, which is the basic spatial capability that is available in Oracle Standard Edition.  This means that everyone can download, try, and use Oracle's spatial capabilities.

Finally, I would encourage anyone who hasn't been to Ottawa, Canada, where I live, to seriously consider a visit.  But make sure you do it in January or February.  I took this picture of the Rideau canal last Sunday.  This is only about 1 km of the canal.  You can ice skate more or less in a straight line for 8 km (5 mi), and you usually can do this for 2 and half months of the year, from Boxing Day (Dec 26) through the middle of March.  This year Ottawa has been unseasonably warm.  Last weekend was the first really good day for the canal.  You can see the whole city is out there including all ages from infants to grandmothers/fathers.