Between the Poles

Developing high precision highway maps for autonomous vehicles

Currently autonomous vehicles with some exceptions rely on on-board sensors for the detailed aspects of navigation. But many believe that high precision maps, which contain significantly more detailed information and true-ground-absolute accuracy than current road maps, will be essential for the safe operation of autonomous vehicles. These high-precision 3D maps will be specifically for the self-driving vehicle market and include detailed inventories of all stationary physical assets related to roadways such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and all other critical data needed for the safe navigation of roadways and intersections by autonomous vehicles.  Referred to as high definition (HD) maps they will record the location of physical highway assets to centimeters.

These HD maps have two applications in the autonomous vehicle market.  They can be used by autonomous vehicles in actual driving to augment on-board sensors with near real-time and highly accurate data of the highway and surrounding location where the vehicle is driving.  Secondly because self-driving vehicles would need to travel billions of miles in the physical world to demonstrate a significant improvement on safety for human drivers, simulation has become an essential part of the development of autonomous vehicles.

 At the 2016 SPAR3D conference I had a chance to chat with Ron Singh, then Chief of Surveys at the Oregon Department of Transport about high definition (HD) maps of highway systems.  High definition means accurate to centimeters and Ron believed that HD highways maps would be required to reduce the risk and costs of highway construction.  But he also believed that autonomous vehicles would require this information.  Today autonomous vehicles rely completely on on-board sensors. but many believe that for safe operation they will require other information that can only come from external HD maps of highways and roads.

At the Year in Infrastructure 2017 (YII2017) conference in Singapore, Sharad Oberoi of Sanborn gave an insightful overview of the state of the art for automated feature extraction from combined aerial and mobile scans of highways and adjacent areas. Sanborn develops high precision maps for the autonomous vehicle market.  Sanborn has developed proprietary mapping technology that leverages aerial imagery, aerial lidar data, and mobile lidar data to create standardized, high-precision 3D base-maps focusing specifically on the self-driving vehicle market.

Recently in the UK Zenzic has commissioned the Ordnance Survey (OS) to explore the geospatial data requirements for HD maps when considering the lifecycle of testing and the data interoperability requirements to enable UK plc to provide an exemplar test facility ecosystem and set the foundations for operational deployments.  HD maps for autonomous vehicles are more complex than maps used for simple driver-based navigation systems. Mapping mustbe highly accurate (better than 5cmin resolution) and needs to contain a minimum set of road information, such as lane-level geometry and information relating to street furniture.This study has been conducted with the support of software simulation companies, the Met Office and the British Standards Institution(BSI), with the aim of understanding the data requirements, gaps and sources regarding geospatial data for self-driving vehicle testing.  A report Geodata report - analysis and recommendations for self-driving vehicle testing has been released.  It addresses why self-driving vehicles require real-time high precision maps. With high-definition maps which are updated in real-time, a self-driving vehicle is able to reference the position of other road users against what it already knows to be there. It also provides a back-up in situations where its sensors are unreliable such as poor weather conditions such as snow, heavy-rain or sun reflecting off a wet-road. 

Its key conclusions are that geospatial data is fundamental for autonomous vehicles. Maps can provide an important trusted baseline where the on-boardd sensor feed are unreliable.   To provide the framework required for the testing and driving of self-driving vehicles, current mapping specifications will need to be enhanced to include relevant street-level features, with better than 5cm resolution, more extensive information about street furniture, and interoperability with other sources of data.  An authenticated, authoritative single source of geospatial data accessible to autonomous vehicles is required. Standards relating to geospatial data in the context of autonomous vehicles are required. This is on the Open Geospatial Consortium radar. Real-time updates to mapping through sensor data feeds will also be essential.  Finally it makes some recommendations about the development of data formats (LAS 1.2 or LAZ, OBJ, OpenDrive, and ESRI shapefile), data hosting, governance, minimum safe requirements and standards.

Fully automated classification of road and highway features for autonomous vehicles

Point clouds are fine for visualization to provide context, but to be really useful they need to be classified.  Feature extraction, as the process is often referred to,  is currently a mostly manual process that has been used for many years in the remote sensing sector. A trained person visually inspects the point cloud and graphically selects features which are identified by type of feature, sign, light pole, lane marking, and so on. 

Automated classification of point clouds

Image processing of 3D point clouds has advanced to the point where it has been possible to semi-automate this tedious feature extraction process.  The reliability of the automation process depends on the point density of point cloud, often expressed as points per square meter or per square centimeter.  Point density depends on the technology used to capture the point cloud. Terrestrial lidar, for example, typically captures points at the rate of hundreds of thousands to millions of point per second and is capable of millimeter to centimeter precision. Phodar generates lower point densities and is less precise. Aerial lidar captures many points per second but with less precision because it is covering much greater areas. Single photon lidar is a new technology that can collect 6 million points per second which means that planes can fly quite high, cover very broad areas and still collect 12 - 30 points per square meter.

At the Year in Infrastructure 2017 (YII2017) conference in Singapore, Sharad Oberoi of Sanborn gave an insightful overview of the state of the art for automated feature extraction from combined aerial and mobile scans of highways and adjacent areas.  Different types of laser scanners are appropriate for capturing different types of features. Terrestrial and mobile laser scanners can capture point clouda that are suitable for extracting curbs, gutter drains, traffic signals, road and local terrain information, parking meters, walls, obstructions in the right of way, manholes, sidewalks, overhead clearances, lights, utility wires and connections, garbage cans, benches, fences, guardrails and barriers, retaining walls, and line of sight vegetation obstructions. Aerial lidar will only be able to get gutter drains, road and local terrain information, walls, obstructions in the right of way, and sidewalks with limited ability to get curbs, lights, utility wires and connections, retaining walls, and vegetation.  Lidar does not do well for metal structures such as communications towers and fir these a combination of lidar and phodar is preferred.

Sanborn has developed feature extraction technology that extracts transportation objects from combined aerial and mobile data sets in a semi-automated manner. The mobile and aerial data sets meet the accuracy and information content required for geospatial information for mapping applications as well as the additional information that can be mined for potential asset inventory and infrastructure information content. Mobile technology allows for the low-risk, rapid collection of geospatial information, limiting safety impacts to workers. The aerial LiDAR allows for the rapid collection of elevation information for detailed surface modeling as well as feature extraction. Both technologies collect enormous amounts of point-cloud data that can be merged using the Terrascan capability in Bentley MicroStation. The combined data set is then passed through the automated feature extraction modules to identify transportation objects.  Sanborn feature extraction modules achieve a high proportion of correct feature extraction for road and highway scans.

In the future Sharad expects that it will be possible to improve data collection efficiency by using terrestrial robots or autonomous UAVs equipped with SLAM technology to enable them to operate in GPS-denied areas such as under bridges and in tunnels.  Sanborn has been working with deep learning to improve the automated extraction process.  So far this has yielded mixed results, but Sharad expressed optimism that with AI full automation of the classification process will be achieved.

High precision maps for autonomous vehicles

An application of this technology is high precision maps for the autonomous vehicle market.

Sanborn develops high precision maps for the autonomous vehicle market.  Autonomous vehicles will require high precision maps which contain significantly more detailed information and true-ground-absolute accuracy than current road maps. Sanborn has developed proprietary mapping technology that leverages aerial imagery, aerial lidar data, and mobile lidar data to create standardized, high-precision 3D base-maps focusing specifically on the self-driving vehicle market. These maps include detailed inventories of all stationary physical assets related to roadways such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and all other critical data needed for the safe navigation of roadways and intersections by autonomous vehicles.

Autonomous UAVs using SLAM to navigate without GPS

I have just spent an incredible three days in Houston at the SPAR3D 2017 conference.  There is so much going on right now in the 3D world with several key technologies just coming to fruition and finding broad application in the real world. 

 

One of them is Simultaneous Localization and Mapping (SLAM) technology.  In 2012 a company called GeoSLAM, which was a joint venture with Australia's National Science Agency (CSIRO), began marketing a SLAM-based solution developed by CSIRO.  I blogged about a backpack solution for indoor mapping that Avideh Zakhor from the University of California Berkeley had developed for navigating inside a building, generating floor plans, distinguishing rooms, identifying staircases, texture mapping surfaces, identifying windows, finding heat sources such as computers and human occupants, identifying lights, and even estimating plug load lower consumption - all automatically.

 

At this year's SPAR3D conference Stefan Hrabar from CSIRO described a project he has been working on that combines autonomous navigation with LiDAR-based SLAM to enable drones to safely navigate autonomously without GNSS in areas with complex geometrical structures.  This enables missions that the UAV can do with no pilot intervention.  Some examples that have been implemented include rainforest surveys - looking for invasive species, telco (cell phone) tower inspection and mapping, construction site monitoring including inside buildings, and underground mine mapping, especially, vertical, hard to access shafts called stopes.  For example, Stefan showed a demonstration of telco tower mapping where the UAV was placed near a telco tower, a button pushed to initiate the flight, and the drone mapped the tower without any human intervention.

 

Hovermap is the CSIRO application that provides a SLAM-based 3D LiDAR mapping payload for small UAVs.  It provides omnidirectional collision avoidance, advanced autonomy, and the ability to operate in GNSS-denied areas.  Hovermap uses a rotating LiDAR to produce a virtual spherical shield around the UAV.  Real-time SLAM makes it possible for the drone to navigate in complex geometric environments  without GPS, while the on-board autonomy makes possible mission execution without human intervention.  Some aplications include construction site mapping (inside buildings and outside), underground mine mapping and transmission line mapping and monitoring.

SPAR2016: Digitalizing highway construction at the Oregon DoT

At SPAR 2013Ron Singh, Engineering Automation Manager/Chief of Surveys, Oregon Department of Transportation (DoT) made a startling proposal that would stand the traditional highway construction process on its head by making the post-construction as-built survey the critical source of reliable survey and asset information in the form of a geo-referenced 3D BIM model (also known as horizontal BIM, heavy BIM, VDC, civil information modeling, BIM on its side, and BIM for infrastructure) and associated scanned and other data.  One of the implications of this is that when a new project is initiated in the same area over 80 % of the necessary survey-grade information is already available in the DoT's geospatially-enabled database making the traditional complete resurvey unnecessary - saving time and money.   Ron foresees that one of the consumers of this digital highway information would be autonomous vehicles.

This year at the Spar 3D 2016 Expo and Conference in Houston I had a chance to sit down with Ron to see how his and other DoTs have progressed on his vision since then.

Ron, the last time we chatted you mentioned you were organizing an event for Western State DoTs in July 2014. How did that go?

It was very successful. I did this on behalf of the Federal Highway Administration (FHA) who approached me and said they planned to do an event similar to the one I organized in 2010 which was called Design to Dozer. In that one we used digital technology including automated GPS-equipped dozers, graders, and excavators to build a sub grade of a road.  It was well attended and it really made a change for my DoT.

The FHA asked me to do another one to promote the use of intelligent construction technologies by providing information and training relating to 3D design, automated machine guidance (AMG) and related technologies for highway construction.  This time I decided to go all the way to paving including ditches, culverts, and drainage. I called it Design to Paver.  We invited the DoTs from the18 Western States plus Florida, North Carolina, and Michigan.  From each Dot we invited their chief engineer and their chief construction engineer.  Altogether we had about 250 people at the event.  It was a two day event and combined classroom work in the morning with on-site in afternoon.

This event included building a demonstration road but I wanted to build one we wouldn't have to tear up and restore. I found a piece of land that was a training facility for the Oregon National Guard. It was a big complex of about 40 or 50 acres and had been a training site since WWII. I went out to see the site and met with them and their environmental people.  I sketched out a plan for a road about 1500 feet long. I picked a site where the road would gain 200 feet in elevation because I wanted it to be fairly complex. I wanted to make some radical curves with superelevations and exaggerated superelevations, so we would have some drainage issues to deal with.  I wanted it to look like a real road and have some interesting geometry. No underground utilities. I wanted to take it from bare earth to a finished 1500 foot road with several curves, ditches, culverts, and paved in less than 2 days.  I don't know exactly but with a traditional approach this would probably have required about a week's work.  We could actually have done it in less than two days, but we were demonstrating.

We did all the surveying for the site months in advance.  We also had a geo-referenced 3D design model with rendering and animation.  We brought in the construction equipment the day before.  We had training sessions for the operators on safety including unexploded ordnance since this was a military training site.

I chunked the 1500 feet of road out in pieces and choreographed the whole thing.  For example, the dozer operator on each section of the road did not break ground until the audience showed up.  They stepped out of the buses and on their badges they had colour codes which allocated them to a team of about 25 people. Each group would go to station A at staggered times.  At station A they would be showed automated dozer operation. Then at a certain time they would go to station B and so on. It was very well coordinated.

What impact do you think the event had on the highway engineers who attended ?

It was amazing - the feedback through the emails I received was so positive.  People still talk about it. If you go to any of the DoTs who attended, the staff who had been there were just amazed.  One of the reasons is that what we were able to show them was something that would have taken a single individual many, many years to see, because none of them had experienced a project with automated equipment from breaking ground to paving.  They were able to experience it all in 2 days. We had all types of equipment, dozers, excavators, curb machines, paving machines, and so on.  We even had unmanned aircraft system (UAVs) for monitoring the construction and ground penetrating radar systems for detecting underground utilities.

Is there a way to find out more about this event?

We created a web site DesignToPaver.org which we intend to leave up for two more years. There is a guide book which you can download. It has more than the agenda. It has all the names of all the companies and  presenters.  If you would like to ask somebody questions, you can find that information in there.

Do you think the road map and time frame for 3D models and construction automation that you outlined is still realistic or is it accelerating?

The technology behind autonomous vehicles is really accelerating as is the engineering automation of highway construction, but the social and legal aspects of highway construction may be the barriers that are keeping it in check. If it were only up to to the technology we would be there much quicker, but I think the social and legal aspects are actually providing good balance, by making sure we are not breaking something that we can't replace.

You mentioned the legal aspects, do you think new legislation will be required to enable to the full digitialization of highway construction and maintenance ?

As long as you put a box around the question and restrict it to engineering automation, not autonomous cars, I think we are well-suited to get there with the existing laws.  I could imagine that to further capitalize on this technology, legislation to register another profession, such as remote sensing, might be beneficial.  I don't necessarily feel that it has to be a surveyor's function to do, let's say, mobile mapping or UAV mapping. I feel it needs to be somebody who is licensed, who has gone through training, who is able to prove that they know how to do this work, and who maintains ongoing professional development training.  The key requirement is that they stand behind their products in a legal sense.

There are many different technologies for capturing data now, even a consumer digital camera can be used to create a point cloud. If I  am a highway engineer how do I decide what data I can rely on for designing a highway ?

The data that has the level of rigour required for engineering design is what I call authoritative data. As an example, if you intend to use a boat with a sonar system and cameras for underwater bathymetric surveying in search of treasure, that's fine. I don't think anyone should regulate that. But if you are going to use the same tools to model bathymetry for the design of a public bridge, then it better be authoritative.  It better be data that you can track the lineage and source of.

One of the key questions I have is about the database of digital data including, in the case of Oregon, data from overflights and mobile LiDAR for the entire state highway system plus the digital as-built models and other data from actual construction projects.  Who do you see is responsible for maintaining this database including updates from ongoing highway maintenance work ?

In Oregon we think we at the DoT are responsible for maintaining it. As long as it is inside our corridor, it was either created by us or by our agents, companies that we hired. There is tremendous value in this data, but the value is only there if in the future people feel confident in using the data.  For that they need to know its provenance.  For example, if you consider our as-built plans, we have had these for generations, but we don't rely on them for anything but to help us find things. We don't rely on them for survey data, because the current as-builts were never created with that in mind.

If we collected very accurate, very rich data and we didn't track its lineage including metadata about how it was captured and when, a future engineering user would legitimately say I don't trust it, because I have no narrative about this data, for example, even something as fundamental as what coordinate system it is in.  But if we can compile all this information together in the form of comprehensive metadata that gives the future user trust in its provenance and reliability, then there should be nothing to prevent the data from being reused.

Are contractors who bit to build roads in Oregon interested in the data you are collecting - do they ask you for it ?

For construction project work, when we use mobile LiDAR or terrestrial laser scanning, we are very careful about precision, for example, we put in ground control points. We do the entire survey in such a way that we end up with a certain level of accuracy, a certain level of point density, and a certain level of reliability. 

When we collect mobile LiDAR data for the whole state, it is impossible to do collect data with the same rigour for the whole state.  We don't put ground control points in and we don't do other things that are required for high precision and reliability.  The data is still very usable for many things, but it is not authoritative, high accuracy survey data, that is required for engineering design.

If you are planning a project in an area where there was a previous project with a digital model, high precision data including ground control points and high density laser scans, would you tell contractors planning to bid on the project about the existing data ?

Absolutely. If we don' t tell them then we might as well as had not collected, documented (metadata) and stored it. We have to tell them because it helps drive the cost down.  To give you some background, our first automated machine guidance (AMG) project was in 1997.  In Oregon some contractors had invested in AMG technology and wanted to use it. In those days contractors who had done this were winning the bids by lowering their bid just enough to beat the other contractors.  When they did the work with AMG technology, they saved a lot of money, but most of the savings was going to the contractors, not to the DoT.  To redress this, starting in early 2015, we began specifying on our bid web site that projects were "AMG ready". We provide all the data we have which they can download to see the models, mobile LiDAR, overflight data, and so on.  The effect is to drive bids down, because there is less risk for the contractor.  Asa result we at the DoT are now sharing in the savings.

Is Oregon alone among DoTs in applying these technologies or are other DoTs also applying them ?

There are about five or six that are leaders in using different aspects of digital technology.  For example, the Wisconsin DoT is focusing on 3D design - they are ahead of us in this technology. Michigan is leading in some areas,  for example, data storage and data management. We met with them because we were interested to finding out what they were doing with engineering data management.  In return they were interested in what we were doing with 3D design and AMG. Utah his way ahead of most DoTs in asset management.  None of the DoTs including us are applying every aspect of 3D digital highway construction and maintenance.

Are there any ROI studies demonstrating the financial benefits of these digital technologies ?

We are doing one. We have a research project we call the OSU Engineering Automation project with Oregon State University in Corvalis.  They are studying our entire engineering automation process. They have already put a good project team together.   The final report is due by next April, about a year from now.

Autonomous vehicles forecast to be commercially available by 2020

Google has had a fleet of 10 or more driverless cars on the roads in Florida, Nevada, and California for a couple of years that by now have logged over a million driverless miles.  The latest cars from mainstream vehicle manufacturers already include intelligent functions such as keeping in lane while adjusting speed to the vehicle in front, automated parking, and crash prevention. 

According to a recent report form Navigant Research there is an industry consensus that the first vehicles with comprehensive self-driving features will be brought to market by 2020.  The technology to support autonomous vehicles such as image processing and sensor fusion are now ready for production use, and supervised decision-making software is in trial on public roads.  Navigant Research forecasts that 94.7 million autonomous-capable vehicles will be sold annually around the world by 2035. 

The human toll from highway accidents is immense.  It has been estimated that a total of 3,551,332 motor vehicle deaths have occurred in the United States from 1899 to 2012.  Every year there are an estimated 1.2 million deaths worldwide in automobile collisions - over 30,000 in the U.S., over 240,000 in India, over 270,000 in China.  Autonomous vehicles have the potential to dramatically reduce the number of fatal accidents on the world's highways.

SPAR 2013: Digital highways require integrated engineering and geospatial data

At SPAR 2013 I had the opportunity to hear very forward looking presentations by Ron Singh, Chief of Surveys/Geometronics Manager at the Oregon Department of Transportation.  What Ron sees in the future and what motivates a lot of what Don sees needs to change in the highway construction process is autonomous vehicles, which will provide a major motivation for what Ron refers to as digital highways.

Automation over the last 25 years

If you look at the last 25 years there has been quite a lot of automation in the form of survey automation such as electronic data collection, GPS, and basic 3D laser scanning;  design automation such as computer aided design and drafting, and a bit of construction automation.  The primary focus has been on speeding up the project development process, but the end goal remains  2D paper construction documents.  Some people have been using what is called 2..5 D, a 3D roadbed prism with 2D objects on it.  In Ron's view the current way of designing and building highways has reached the end of it's lifetime and a complete transformation to a new paradigm is required.  The way Ron refers to this is standing the current process on its head and I blogged about it yesterday.  Fundamentally it involves intelligent 3D models replacing the current as-builts and a process that speeds up project development because it leverages existing engineering data rather that requiring a complete resurvey at the beginning of every project..

A new paradigm for intelligent highways

The objective is intelligent highways that will support autonomous and partially autonomous vehicles.  The new paradigm needs to be data-centric and real-time.  It will involve automated construction.   From a data management perspective it will require highway infrastructure lifecycle management, which will involve a very different approach to the survey/design/construct/maintain process that is currently used for highway projects in North America.

First of all, it will involve geospatially enabled 3D digital data. A key requirement is a low distortion geospatial coordinate system that will provide the foundation for geolocatiig all infrastructure including utilities.  It will require interoperability using industry standards for data exchange. Most importantly it will support a construction process that uses post-construction surveys to replace as-built plans with a full 3D engineered model that supports GIS/engineering data integration. 

It will require geospatially-enabled enterprise data management including version control and the ability to search for things geospatially.  It will provide tools for geospatially enabled asset management.  It will provide a framework for developing and maintaining engineering and geospatial data including 3D engineered models as well as scanned data for existiing structures for the entire life cycle of a highway.

SPAR 2013: Turning the construction process on its head

At SPAR 2013 I had the opportunity to hear a very forward looking presentation by Ron Singh, Chief of Surveys/Geometronics Manager at the Oregon Department of Transportation.  It was even more impressive because Ron's background is surveying.  His primary objective is speeding up the project development process.

Ron first looked at the project development process that has been used in the transportaion over the past 25 years.  It starts off with a complete survey of the project terrain to create a basemap - existing as-builts are rarely consulted.  Design of the new project is then conducted based on the basemap produced by the surveyors.  The result is paper drawings which are used by construction contractors to build the project.  After construction is complete construction as-builts are submitted, but they are almost an afterthought.  They are often as-designed rather than as-constructed, and most importantly they are not "sealed", meaning nobody signs off on them putting their professional reputation behind their accuracy and reliability.  Practically what this means is that in the transporation industry, as-builts are treated as unreliable information.

What Ron Singh is proposing is to stand this process on its head by making the post-construction survey the critical source of reliable asset information in the form of a 3D intelligent model. 

Then when a new project is initiated 80-90% of the necessary information is already available in the as-built database making a complete resurvey unnecessary.  All that is required before design can begin is minimal due diligence to validate the as-builts. 

It also implies that there is a reliable asset database that is maintained thoughout the operations and maintenance phase of the lifecycle.