Between the Poles

Mixed reality in the construction industry is becoming a reality

At the SPAR3D 2017 conference in Houston, I had the distinct feeling that we are on the cusp of an important inflection point where technology has evolved to the point that it is going to transform the world's construction sector.  Handheld scanners and mixed reality are two of the technologies that are set to revolutionize construction.  According to Gartner mixed reality is poised to attain what Gartner calls the plateau of productivity where a new technology is applied to real-world problems on a global scale.

In many countries with advanced economies the construction sector, which contributes about $ 10 trillion annually to the world's GDP, has not experienced improvement in productivity for 50 years.  During this same period productivity in the non-farm general industrial sector has more than doubled.  That means that the value produced by an hour of a construction worker's time is the same now as it was 50 years ago, whereas an automobile worker produces more than twice as much value as 50 years ago.  If infrastructure is going to attract the private investment from pension funds, insurance companies, and sovereign wealth funds that is required to maintain and build out the infrastructure we need over the next couple of decades, construction productivity has to improve.  The goal of improving productivity will drive investment in technology.  According to McKinsey the construction industry is poised for rapid digitization. Examples of the technology that is set to digitize and transform the construction industry were demonstrated at SPAR3D this year.

At this year's SPAR3D three companies in the construction sector, Gensler (a collaborative design firm), McCarthy (a commercial construction firm), and Martin Brothers Construction (a heavy civil construction firm), demonstrated how they were using mixed reality on real-world projects.  The tools they used include Enscape, Revizto, Revit, Sketchup Pro, Unity, 3dsMax, Umbra, Hololens, Sketchup Viewer on Hololens, Oculus and HTC Vive.  The application that wasn't shown this year, but I hope will be next year at SPAR3D, is Fuzor.

Cody Nowak of Martin Brothers demonstrated building a bathroom pod with Hololens.  The bathroom pod was designed using a standard design application sush as Revit or Sketchup, then projected onto the reality of a shop floor were it was constructed using the full-scale virtual reality projection of the design into the shop.

Gensler described collaborative design where two or more designers and stake holders can share a mixed reality space where they work collaboratively on a design.

McCarthy demonstrated how easy it was to work in a virtual reality space with a headset and handheld controllers.  They asked a member of the audience with no mixed reality experience to don the headset and controllers.  Within a few minutes she was able to move around in the virtual reality - we could see what she saw on a separate screen.  Using the handheld controllers she was also able to move things around in the virtual space.  She moved furniture and furnishings on the wall.  She even threw a stool across the virtual room.

Many of these mixed reality visualizations can be used on handhelds.  Workers in the field or during operations and maintenance can use these applications to inspect and query facilities to access inspection and maintenance reports for equipment they point at in a mixed reality environment.

The next step, which was not demonstrated at SPAR3D this year but I expect will be next year, is bidirectional communication link between the mixed reality environment and the authoring design tool, whether Revit or Sketchup.  This will make it possible for changes made in the mixed reality environment to be visible in Revit or Sketchup where they can be incorporated in the design.  The reverse of this is where design changes in Revit or Sketchup are visible in real-time in the mixed reality environment.

The key elements of mixed reality as it applies to the construction industry are

• bidirectional link between the mixed reality environment and the design tool enabling users to design in their authoring software and view the changes to their design in real time as well as create design changes directly in the mixed reality environment and send those changes back into their authoring software.
• mobile applications of mixed reality for people in the field or in an operating environment
• real-time collaboration in a mixed reality environment independently or simultaneously from anywhere on the globe from a computer or mobile device

We want to be able to don our VR gear and visualize the model we are working on. If there are design changes, we want to be able to make and update them in real time and then jump right back into the mixed reality environment.  We also want guests to be able to view the our design changes in the mixed reality environment as we make them.  Stakeholders need to be able to explore, review and modify projects independently or simultaneously from anywhere on the globe from their computer or mobile device.  We need real-time collaboration to actively review a design with stakeholders, or let collaborators review and make changes to the model in real-time.  This is not pie-in-the-sky dreams, people are doing these things already with the software tools I mentioned above.  I don't think the question is if, it is how rapidly these tools will proliferate in the construction industry.  BIM and digital scanning of different types have already proliferated over the globe and I expect with this digital foundation, these newer tools will spread even faster.

On-site AR visualization of BIM designs in an urban environment

There are two advances in technology that are having almost as much impact on mapping our world and tracking the location of things in it as the introduction of the GPS.  The development of microelectromechanical systems (MEMS) in the 1980s and 1990s has made possible the mass production of micro inertial measurement units (IMU) comprised of a combination of accelerometers, gyroscopes, and magnetometers.   These have enabled the smart phone which contains a range of sensors, typically an accelerometer, thermometer, gyroscope, pressure sensor, humidity sensor, light sensor, location sensor (GPS),and a magnetometer.  The second major advance is fast structure-from-motion (SfM) algorithms.  One of the first applications is simultaneous localization and mapping (SLAM) technology which was developed for autonomous vehicles. In 2012 GeoSLAM was developed which made it possible to solve a major spatial problem, accurately tracking the location of a person inside a building or in an obstructed environment such as a city.  Recently SfM has been successfully applied to indoor mapping in the form of new professional-level products from Indoor Reality and Leica Geosystems (Pegasus Backpack).

It turns out that structure-from-motion technologies can also be used for on-site visualization of a virtual BIM design in a real urban context.  At this year's Geo Business 2016 I had a chance to talk to Crispin Hoult about an augmented reality product called UrbanPlanAR his company Linknode is developing with SfM technology from Heriot-Watt University that is designed to do this on consumer handheld smart devices such as an iPad.

Background

For years researchers and technologists have been struggling with a major spatial challenge - accurately tracking the location of a person in an obstructed environment such as a city or inside a building in 6 dimensions, 3D location and 3D orientation.  Stucture-from-motion solves this problem and enables you to track your position and your orientation in an obstructed environment where GNSS satellite signals are unavailable or unreliable. Typically input from an inertial measurement unit (IMU) and LiDAR is used to construct in real-time a 3D map of the immediate environment including objects such as walls and other objects.  As you move, the map is reconstructed and this is used to record your trajectory.  The algorithm is robust and helps to addresses some of the problems, such as drift, that results from using just an accelerometer.  

Indoor mapping products, such as those based on GeoSLAM, assume that you are working in an unknown environment.  But what if you already have information about the environment in the form of GIS maps, point clouds, and other georeferenced information ? Tracking your location in a known environment using just a smart handheld device is the problem that Crispin Hoult and his company Linknode are addressing.  The specific business opportunity he is targeting with this technology is on-site (in-field) augmented reality visualization of BIM designs for new buildings with a consumer smart handheld. 

Technically SfM algorithms require multi-sensor input.  The typical smart phone or tablet has many motion tracking MEMS devices built into it.  If you are outside in an unobstructed environment, the GPS sensor can reliably report your location to within 5 or 10 meters.  But when you enter an obstructed environment where you have reliable georeferenced information, you need an algorithm that uses the camera and the inertial navigation sensors in the handheld tablet to accurately track its location and orientation in real-time.  Crispin has been working with Heriot-Watt university to "mobilize" an algorithm that does just this. Basically the algorithm uses “visual-inertial tracking” for the online tracking of the 3D position and 3D orientation of the tablet. This is supported by ‘prior’ learning of the visual and structural appearance of the environment through structure-from-motion (SFM).

Crispin, what is the business problem that UrbanPlanAR addresses ?

We are targeting UrbanPlanAR at on-site visualization using only a consumer smart handheld.  We are making it possible for an architect, owner, or city planner to come to a site with a handheld tablet and in real-time overlay a BIM model showing the design for a new building in the context of the real world.  This is an application of augmented reality.  What you see on the tablet is the real world and into that we superimpose the BIM model.  Our focus right now is to visualize the building design in its urban context - making it possible to walk around in the area and see the building from different viewpoints. We add value by providing more trusted, more contextual visualizations so people can make decisions and engage more with the planning process.

The alternative way of doing this is to create a completely virtual world (often referred to as black box), but then you have problems with resolution and realism.  We have research carried out by the University of Glasgow that shows that by doing on-site visualization and not blackboxing it, it becomes more trustworthy and enables more engagement from stakeholders.

Another way to look at the business value of UrbanPlanAR is that it adds value to your BIM investment. Designers and contractors invest in BIM because it enables automated clash detection, material takeoff, quantity surveying, construction staging and many other applications. One of the important value-adds that hasn't been fully recognized is visualization.  Our technology can be another strong reason for creating a BIM model in the first place.  If you have already developed a BIM model, it adds value to your model. With our software you can take your BIM model, walk out to the proposed site and show all stakeholders - potential customers, city planners, city engineers, and nearby land and building owners what the proposed structure is going to look like on site, in a real world context.

How does this work, what do I need to do to make this augmented reality visualization of a building design ?

The first step is to choose the context.  Unlike the traditional approach where we would choose a location from which we want to view the building design, with UrbanPlanAR we choose an area of interest, which could be a city square, for example.  Then we collect georeferenced information about the area we have chosen. Currently we use imagery captured specifically for that viewpoint site, but this will soon allow other alternatives to be used, such as existing datasets, for example, a city model, LiDAR scan, oblique imagery, or satellite or overflight imagery.

The next step is to get the BIM model of the design from the architect. To do this we plug into BIM 360, which is the Autodesk data store for BIM models. For smaller projects we can take a model published directly from Revit.

At this point we have the context and we have the BIM model. Then we simply take the mobile device out into the field on a tripod or simply held in the hand, and UrbanPlanAR will use the context to work out exactly where it is and enable you to see in real-time your design within the real world – displaying the BIM model not only “onto”, but visually “within” existing reality.

Unlike Virtual Reality systems where there are cost and time limits to the digitization process, what you see in augmented reality is 3D, infinite resolution that all comes for free as the real world.  What we are showing is a combination of real and virtual. As users stand at the site, they see exactly what is there plus the change. That's what adds to the trust - you are on-site, out in the field or the city square, walking around and showing stakeholders what is actually there with the superimposed virtual model. You can also record it so you can replay it back in the office.  That real-world GIS-based augmented reality we call that GIality. Everything is georeferenced, everything is anchored in the real world. That is what links the virtual to the real world.  You need to know where the virtual building is in real-world coordinates.  The technical challenge is that you also need to know  where the observer is and his/her orientation as he or she walks around the site.

Who do you foresee is the end user of UrbanPlanAR ?

I would say that the primary application is early stage project feasibility and development for a new building. It could be used by the architect or by the developer who is funding the architect. It could also be used as a sales tool at this stage.

The second stage is the planning stage. When you are looking at permitting you need the convince the city or town council, who are the custodians of the city. Typically landscape architects are hired to develop and present the environmental statement and impact of the project to the city planner. But we now live in a world where we no longer have to rely on consultants to convey information.  Nowadays the developer may do this him or herself.  The city or town planners may also be users - as validators of what has been presented. UrbanPlanAR broadens the breadth and depth of 3D information available about the project.  It is also interactive in that the architect can make changes to the BIM model and synchronize that to the users.  Another advantage is that the users can go out to the site at any time to inspect some detail of the design that they may have missed.

What we have found useful is to interactively display different surfaces or finishes, different building heights, and different models. Preconstruction changes are very easy to do.  The architect goes away after getting feedback from stakeholders, creates a new or modified BIM model, uploads it the cloud, and then a user synchronizes with the store and takes it out to the site.  By doing this at the feasibility stage, early on in the planning process, communities become more involved and don't see what is being presented as almost a fait accompli. You want them to leave feeling they are actively involved in the design process - then you get more buy-in and there is less likely to be an appeal during the construction stage.  

Does the BIM model have to be a Revit model ?

We integrate with BIM 360 today and we are planning to support Bentley BIM.  In addition Industry Foundation Classes (IFC) is one of our supported file formats.  But this is not yet equivalent to importing files directly from Revit or Bentley BIM, because exported IFC files from some BIM vendors do not support texture.  

What types of hardware does your software run on ?

We support the iPad Pro and we are working on a Microsoft Surface port.

Does UrbanPlanAR require preprocessing in the cloud or on a desktop ?

Neither the BIM model nor the geoferenced context ends up being the operational model we need to do the localization, but both are valid inputs.  All the input data including GIS context and Revit model is preprocessed on a desktop or in the cloud and then synchronized to the handheld.

Crispin, you already have a product that is designed for rural environments but also relies on augmented reality.  Can you tell me about that ?

Yes it is called VentusAR, and it is designed to work in a rural environment.  It has been targeted on visualizing wind farms and solar farms for land owners, communities and planners because these farms are very contentious issues. Before our our product became available, the standard methodology was to take photographs out in the field, go back to the office, process them on the desktop to incorporate the proposed wind or solar farm, send the resulting images to the printer and then show the picttures to the stakeholders. Because of the time and cost it tended to be done late in the planning process, which made it difficult to make changes.

Late last year for the first time we used VentusAR for visualizing transmission towers.  And a few weeks ago we did our first modeling of transmission lines.  As with UrbanPlanAR, VentusAR uses the tablet’s integrated camera, GPS position and 3D gaming capabilities to create an accurate, location-specific, realistic view of transmission tower models in context.  You can investigate different scenarios with different stock models like towers, turbines, and panels. It works well in a rural environment because our georeferenced information about the real world is typically a digital terrain model. In a rural environment, if your GPS is out by 5 meters and you are looking at something a hundred meters away, it doesn't matter that much.  In an urban environment where you need more precision and can't rely on GNSS, you need a more sophisticated approach, which is what we have done together with Heriot-Watt University for UrbanPlanAR.

The project is supported by Innovate UK, the United Kingdom’s innovation agency (Project 102040).

Spar 3D: 3D mapping the inside of multi-story buildings with a backpack

At the Spar 3D Expo and Conference in Houston The Woodlands, Texas, Avideh Zakhor from the University of California Berkeley was onsite with some remarkable products from her new startup Indoor Reality.  I blogged previously about the technology when it was still under development.  The "man portable system" she has developed is a 32 pound backback with a collection of sensors.  The most important feature of the technology are sensors and algorithms for accurately tracking the six degrees of spatial freedom which determine the location and orientation of the backpack in the building.  In addition there are sensors for recording optical and thermal point clouds. She has also developed a handheld device that supplements  the capabilities of the backpack.

The technology can track accurately the location and orientation of the backpack as the operator walks through the building going into all rooms, halls, up and down stairs and into other human accessible spaces.  Software tools then generate different 3D data products including optical and thermal 3D point clouds, a colourized 3D point cloud, surface texture map,  and triangulated mesh surfaces.  The package includes analytical software tools for automatically generating floor plans, distinguishing rooms, identifying staircases, identifying windows, finding heat sources such as computers and human occupants, and identifying lights. Surface textures can be added to create a 3D photo-realtistic rendering.  The sensors and software can capture and generate all required input for the Department of Energy's EnergyPlus energy performance analysis program.  It has GPS and can generate real world coordinates.  Indoor Reality is an ESRI Emerging Business Partner.

I had a chance to sit down with Avideh to talk about her new company and the remarkable products she was demonstrating at SPAR2016.

Avideh, what are the key features of the products you are demonstrating here that distinguish them from some of the other devices vendors are showing here at SPAR ?

We have developed a unique way to track where the operator is in a building and the orientation of the backpack.  We have a lot of sensors. That is the key to all of our intellectual property (IP). It's called sensor fusion and most of our IP is in that area. It does not suffer from drift. It tracks accurately where you are in the building even if you go upstairs. You can even go up in an elevator because it has a barometer.  In addition we detect wifi.  It can tell you what the wifi strength is in different parts of the building. We have an infrared sensor and we have developed an app that will give you real-time occupancy of the building, for example, we can tell you there are four people in this room, two people in that room. We have patents on that.

Based on the data collected by the operator's walk through the building, we allow users to virtually navigate through the space and spatially collaborate with each other through our web based visualization tool. During virtual navigation, you can look around in 3D as if you were there.  That's our second data product. We don't just record dumb point clouds.  We generate intelligent meshes.  Not only can I can pan around, look up and down at each location, but the shape of the cursor shows you the orientation of the surface - for a horizontal surface it's a circle, for vertical surfaces it's a square.  The cursor gets bigger as it gets nearer, smaller as it goes farther away.  We know the normals of all the surfaces because of the 3D mesh behind what you see.  We recognize doors and windows.  Of course, we can measure distances and we allow users to annotate objects and locations.   So this mesh is very intelligent, so intelligent that we can automatically generate a Revit model with a floor plan, rooms, ceilings, doors and windows. That is our third data product.

How do you foresee people will use the tool ?

We went to the University of Illinois just a month ago and we did an energy model for a 14 story bldg. We can do 3d geometry, we can detect lights automatically, we can detect windows automatically, and we can detect AC plugs automatically.   This gets fed into an energy simulation model which is a complete model for an Energy Plus performance audit.  Typically when people do an energy audit they are only interested in the lowest hanging fruit, for example, replacing incandescent light bulbs with LEDs.   This is because collecting the data for a full Energy Plus model has been time consuming.  With the Indoor Reality backpack you can collect whatever you need for a complete Energy Plus audit very simply. For example, we can superimpose the infrared record on a ceiling to show a hot area which is very likely a hot water pipe behind the dry wall.

We have just come out with a visual building documentation tool for stakeholders to collaborate  with each other throughout the lifecycle of a building - during design, construction and operations and maintenance.  If I want to remodel a building using an architect, designer and contractor they can communicate with each other by leaving notes for each other in the virtual building.  For example, I can put a tag on a door for John the contractor. "Please fix this door."  John can come along in the virtual building and edit that note "Avideh, I can do this Tuesday."  So this makes it possible to collaborate virtually on a building project in 3D,  in much the same way that we can collaborate on a Word document.

There is another dimension to this.  Let's say we want to remodel a space in a building. The first thing the architect will want to see is the existing floor plan and the other professionals will also want to see other aspects of the space.  Normally at this point the building owner will ask everyone involved in the planning to come over for a  walk-through the building.   That requires a lot of valuable time from very busy people.  With the virtual model that we can generate, most of this won't be necessary any more. The owner can ask a relatively junior person equipped with the backpack to walk the building to collect the data.  Then the resulting model can be put online and everyone can virtually walk through it.  For bidding on a big renovation project, a relatively junior operator does the walk through and puts it on the cloud and the construction companies who are interested in the project can virtually walk through it online for bidding purposes.

For a new building,  during the actual construction you can conduct a walk through every day and update visually what's going on.  Then if there are lawsuits in the future, you will have a permanent record of who did what and when.   You can show the legal folks where a component was supposed to go and where it was actually installed. It is also useful for monitoring progress, so you know who did what when to determine who gets paid and how much.   If there is a lawsuit and insurance companies get involved, you can step through the record to see who did what.  Then when you want to sell the finished building, the real estate folks can do a virtual walk through of the building with prospective customers.  Property managers and facility managers can use the virtual model for maintaining the building. For example, we talked about the energy performance audit.  Facility managers can use it to identify lights that need to be replaced with more efficient bulbs. Then they can put a virtual tag on the lights for the maintenance folks.  With a virtual model like this, owners, contractors, designers, and facility managers can all collaborate throughout the lifecycle of the building.

There seems to be a lot of areas where the technology you have developed could be used ?

There are quite a number of other application areas for the technology we have identified including historical building and monument recording, scan-to-BIM,  variance detection and virtual inspection, real estate valuations, building plans for first responders, change management and dispute resolution, and detecting thermal anomalies including water leaks and insulation deficiencies to name just a few.

GeoBuiz 2016: Geospatial and consumer devices, reality modeling, big data and analytics

Reality modeling

In the future, the deliverables at the end of a construction project will not only include the completed building itself, but also a digital "reality model" of the structure.  The digital model will include digital as-builts comprised of georeferenced point clouds captured by laser-scanning or meshes generated from images captured with a digital camera on a smartphone.  That was the prediction of Bhupinder Singh of Bentley at the GeoBuiz 2016 Summit in Bethesda.  This is a vision that is shared by other professionals in the construction industry including Ron Singh, Chief Surveyor at the Oregon Department of Transportation, who sees it as dramatically changing the construction process from surveying through design and construction to operations and maintenance.

Consumer devices and professionals

Consumer mobile devices will increasingly be the platform for many professional activities which currently require specialized devices and a PC computing platform.  Bryn Fosburgh of Trimble pointed out that while currently the location accuracy of GPS equipped smartphones is 10 meters, centimeter precision is surely coming because all that is required is a relatively simple antenna redesign.  Bryn sees a continuum in these devices between consumer and professional applications.   But while consumers tend to blindly accept whatever is available on their smartphones, professionals using these devices will have to be aware of the limits of the hardware, what it can be used for and what not.  For example, while smartphones in the future will be capable of centimeter accuracy in ideal locations, the accuracy will be less where there is a lot of scatter in the RF signals from the GPS satellites.  To ensure reliable results a professional surveyor relies on multiple sources including GPS linked to an earth station and traditional total station measurements.  Increasingly we will require other professionals to be involved.  Bryn reported a rising demand for photogrammetrists, as a result of the growing volume and availability of high resolution imagery which needs professional to help interpret.

Platform as a service

For virtually all of the world's big IT companies including IBM, Oracle, SAP, and Microsoft and the world's big content companies such as Digital Globe, the cloud is central to their vision of the future.  Together with others in the IT industry Xavier Lopez of Oracle sees the future in platform as a services (PaaS), but there are challenges.  A major challenge is integrating data in different databases including traditional SQL databases from Oracle and Microsoft and nonSQL databases such as Hadoop.  Another challenge is semantics.  Frequently different disciplines use different terms for the same thing.  I have blogged about the CB-NL project in the Netherlands aimed at solving the semantics problem in the construction industry.  Another big issue that Xavier identified is data fusion without co-location - how do you enable efficient processing of a logically federated database when the data resides in different locations ?

Monetizing the cloud

But perhaps the biggest challenge is finding an appropriate business model.  It is not completely transparent how the cloud is going to be successfully monetized, though the trend is toward some form of subscription or pay-per-use.   An interesting model of the successful application of the cloud in the construction industry is Textura, just acquired by Oracle.  Textura’s cloud serves as a place in which all of the participants in a project from the funding, through design, to construction can come together and access the shared assets necessary to complete the project.  Textura’s network includes about 85,000 general and subcontractors who using Textura to collaborate on construction projects.  Textura handles on average about 6,000 projects each month. The company’s shared database, which hosts project budgets, contractors, invoices, payments, change orders and compliance documents, resides in the cloud and is accessible to contract administrators, contractors, developers, subcontractors, consultants, title companies, and banks.

Big data and analytics

One of big drivers for the cloud is enabling the extraction of information from the huge volumes of data (most of which includes location) which are now available.  I have blogged about the nearly 100 petabytes of earth observation imagery available from Digital Globe.  According to Dr Richard W. Spinrad, Chief Scientist at the National Oceanic and Atmospheric Administration (NOAA) , NOAA is expecting to have 160 petabytes of archived geospatial data by 2020. 

As another example, Jeff Jonas of IBM described a publicly available database which includes 600 billion transactions per day collected by telephone companies.  This data, which includes location, can be mined (after being anonymized) for very interesting and commercially valuable information.  Jeff Jonas described some examples - using it to analyze where people spend most of their time (their "pattern of life"), how many people go into a store, and how long they remain in a store. 

Dipanshu Sharma, Founder & CEO, xAd illustrated live some of the things his company, which is in the location-based marketing business, does with this type of data.  xAd can customize ads to target people who are inside a store ("There's a great deal on aisle 5"), just outside one ("Come on in, we have great deals on umbrellas") or close to a competitor's store ("Checkout XXXX, our prices are lower than YYYY") - the words are mine.

In all of these example, geospatial data and technology are key to the applications.  Jeff Jonas foresees that the apps of the future on your smartphone will require geolocation and will be so compelling that just about everyone will enable GPS location on their smartphones.

SPAR2016: Augmented reality is finally going to happen, on smartphones

I am back from two remarkable days at the Spar 3D Expo and Conference in Houston.  One thing that has stuck in my mind, because it is likely going to impact all of us, is augmented reality (AR) or virtual reality.  AR has been around for a very long time, but has been used for very specific and narrow purposes such as training for space missions, has required a heavy iron computing platform, a heavy head harness and heavy lenses, and has been very expensive.  Paul Davies of Boeing traced the history of AR including its progress on and off the Gartner hype cycle;

• Innovation trigger 2004-2006, 2008-2009 (dropped altogether in 2007)
• Peak of inflated expectations 2010-2012,
• Trough of disillusionment 2013-2015.

According to the hype cycle, AR is positioned for the slope of enlightenment leading to the plateau of productivity where it starts contributing to solving some of the world's challenges.

David Smith, CTO, Wearality gave a very informed presentation on augmented reality that identified the critical issues that have held up the technology.  He made a convincing case for why AR is poised to take off - not on PCs, but on smartphones.

Platform

According to Gartner, PC shipments declined by 8.3 % in 4Q2015.  Smartphone shipments continue to rise, almost exponentially.  Current and future investment in platforms is going into smartphones, not PCs.  While AR for PCs will continue to ship, mobile AR will quickly outstrip AR for PCs.  With one exception all the technologies that AR requires are available on smartphones.  The implication is that the future of mass AR is mobile AR.

3D

You can make the case that there is so much 3D technology and data out there that we have reached the critical mass where we need an inherently 3D platform to take advantage of it.  Games led the way, but now construction (BIM), mechanical design, GIS, and other important sectors have moved into 3D in a major way. 

Last year the Open Geospatial Consortium (OGC) has approved the Augmented Reality Markup Language 2.0 Interface Standard (ARML 2.0).  ARML 2.0 provides a free and open, non-proprietary encoding that content providers can use to specify the location and visual appearance of real world objects and virtual objects in an augmented reality scene. Additionally, ARML 2.0 defines a Javascript interface for communicating how user input selects, queries and dynamically changes the properties of these virtual objects.

Paul Davies of Boeing gave a specific example of an experiment conducted at Boeing where using AR to instruct a worker in the complex process of a wing assembly was found to be much more effective than providing a PDF manual with detailed instructions.  The AR process combined real images of the components and the work space with virtual demonstrations in 3D showing the worker which components went where and exactly how to put them in place.  The results of the experiment showed that the quality of the worker's first attempt to assemble the wing was much better with the AR instruction and the time she required to assemble the wing was 1/3 faster.

Optics

David said that the one technology that is required and is not currently available for mobile AR is lenses.  Current lenses have less than a 100 degrees field of view, suffer from chromatic aberration, are out of focus in the peripheral areas, and are much too heavy.  Google Cardboard has a small field of view (70 degrees), while HTC Vive and Oculus with 95 degrees are not a lot better.   From David's perspective this is the bottleneck that has held up mobile AR.

Now Wearality has developed acrylic lenses that are incredibly light-weight, have a 160 degree field of view when used with a smartphone, enable natural peripheral vision, and are foldable and pocketable.  David allowed us to try them attached to a smart phone.  The light weight, the wide field of vision and that I could use them with my glasses left me with the experience of IMAX, but without the theatre.

David foresees that even the next smartphone, or possibly the one after that, that you buy will be running AR out of the box.  If you have wondered where Michael Jones, of Keyhole and Google Maps fame, has landed, he is now CEO of Wearality, which confirms for me that Wearality is very likely on to something and that all of us with smartphones have AR in our immediate future.

Augmented reality standard adopted by OGC

The Open Geospatial Consortium (OGC) has approved the Augmented Reality Markup Language 2.0 Interface Standard (ARML 2.0).  ARML 2.0 provides a free and open, non-proprietary encoding that content providers can use to specify the location and visual appearance of real world objects and virtual objects in an augmented reality scene. Additionally, ARML 2.0 defines a Javascript interface for communicating how user input selects, queries and dynamically changes the properties of these virtual objects.

Augmented reality involves a smartphone or a tablet that has a screen, camera, GPS and an accelerometer that detects camera motion and orientation.  The real world "background" is comprised of data, including location, of real world objects or visible objects .  For example, in the context of construction virtual objects representing the design of a new building or new infrastructure or a 3D model of underground infrastructure can be superimposed on a map showing existing conditions or a map of surface structures such as roads and buildings.  The augmented reality application superimposes the virtual design objects or the underground infrastructure on the map of existing objects or the surface map.  (Image showing 3D model of Las Vegas underground infrastructure from VTN).

SPAR 2013: Developing an intelligent 3D model of above and below ground infrastructure for the City of Las Vegas

At SPAR 2013 Keith Warren, BIM/3D SUE Manager at VTN Consulting gave a fascinating presentation on the 3D above and below ground model of city infrastructure including roads, utilities, telecom, and buildings that VTN has developed with the City of Las Vegas.  Keith believes that at the present time Las Vegas' 3D city infrastructure model is unique in North America.

This model also represents a classic example of the benefits of convergence, the integration of engineering design data including building information models (BIM), geospatial data including digital terrain models, high resolution photogrammetry, and point clouds derived from laser scanning, together with 3D visualization technology.

Education

About five years ago the City of Las Vegas approached VTN because they saw some potential value in 3D modeling.  At that time the city had had little experience with 3D modeling.  VTN has a broad range of in-house capabilities spannng engineering (traffic, public works, and 3D subsurface utility engineering), survey including laser scanning, GIS, building information modeling, and visualization.  Over the next two years VTN focused on education, working with the city and other utility agencies to introduce them to 3D modeling and visualization technology.  VTN organized workshops with the city, local utiltiies including the water district, and other public agencies to educate them in some of the capabilities of the new technology.  At the same time they worked with the city and other public agencies to understand the problems they were facing.

Challenge: subsurface utility engineering (SUE)

One of the problems that was identified repeatedly in these workshops, not only at the City but also at the utilities, was hitting underground infrastructure during excavations.  According to national statistics, in the United States an underground utility line is hit on average every 60 seconds.  The total cost to the national economy is estimated to be in the billions of dollars.  The problems is that in most municipalities in North America, for years underground utility lines have been put in the ground not according to plan but wherever it has been easiest and cheapest to build them.  In addition 2D as-builts of underground infrastructure are unreliable.  The result is that in most municipalities the location of underground utiltiies is very poorly known.  This was the situation in Las Vegas.

Since the city had been educated on 3D modeling technology by VTN, they decided that this technology could provide a solution for the problem of locating underground utilities.  The City decided to go ahead with a project to model one and half miles of Main Street in the older part of Las Vegas.  The project was intended to model below and above ground facilities including roadways, utilities and telecommunications, and buildings.

The project also specified implementation of a new low distortion geospatial coordinate system to make it possible to support engineering grade accuracy for geolocating infrastructure.

Reality capture

Buildings were modeled in several ways, including extruding a building footprint from a GIS, using building models from the Sketchup 3D Warehouse, laser scanning existing structures, and complete BIM models for newer buildings provided by architects. 

A variety of techniques were used for reality capture of underground utilities and other infrastucture includng GIS, survey, design records, test holes, and ground penetrating radar (GPR).  Above ground utilities were captured by combining GIS data with mobile laser scanning. 

In the future to improve data quality for underground infrastructure, since the City owns most of the right of ways within Las Vegas proper (some is owned by the Nevada DOT), it has mandated that in the future contractors are required to use open trenching for utiities and all work has to be surveyed before the trench is closed. 

Intelligent city infrastructure model

The deliverable was a set of 3D models of all the underground and aboveground infrastructure and buildings for the one and a half mile corridor of Main Street in old Las Vegas.   Engineering design and other data was combined with the City's geoimagery, digital terrain models and other GIS data. The models were designed to run with the software that Las Vegas already had including Civil 3D, Navisworks, ESRI ArcGIS and Autodesk Infrastructure Modeler (Infraworks).

Augmented reality

Working with a partner VTN developed an application for the iPad that allowed underground facilities to be viewed virtually under the actual roadway.

Map-21 and Every Day Counts

Map-21 and Every Day Counts are Federal Highway Authority (FHWA) initiatives that are designed to encourage the adoption of 3D modeling.  MAP-21 (Moving ahead for progress in the 21st Century) requires 3D modeling and virtual construction and visualization technology for all eligible projects.  The Federal government will provide matching funding on projects using intelligent 3D modeling and visualization of up to 5% fo project costs.  The Map-21 initiative was announced last July, but it turns out that what Las Vegas has done is in Keith's estimation 80-90% compliant with Map-21 guidelines.

Every Day Counts is another FHWA initiative that encourages 3D modeling.  According to the FHWA 3D modeling in transportation construction allows for faster, more accurate and more efficient planning and construction. The FHWA foresees that as the benefits are more widely recognized, many in the U.S. highway industry will transition to 3D modeling over the traditional two-dimensional (2D) design process.

Benefits of the intelligent 3D city infrastructure model

One of the major benefits that Las Vegas has experienced as a result of developing the intelligent 3D city infrastructure model is increased safety because of the reduced risk of unexpectedly hitting underground utilities especially hazardous ones like gas mains. 

The benefits of being able to accurately locate existing underground infrastructure are immense.  They include automated clash detection to identify potential problems when plannng, designing and constructing new undergound infrastructure.  Also there are reduced operating costs because of reduced truck rolls for cable/pipe locate operations. 

Overall the City has found that the 3D model approach provides more information per dollar invested, in other words more capabilities at lower cost.

New directions for geospatial standards: augmented reality, citizen observatories, and the Internet of Things

I have frequently blogged about some of the important geospatial standards that have been fostered by the Open Geospatial Consortium, most recently about a new Energy and Utilities Domain Working Group (DWG) for the electric, gas, oil and water services industries, the recently adopted GeoSMS standard  which is an extended Short Message Service (SMS) encoding and interface for exchanging location content between devices or applications, and the new Geosynchronization Services (GSS) standard for synchronizing distributed geospatial data stores.

Last week at the GIS in the Rockies annual conference in Denver Carl Reed of the Open Geospatial Consortium gave an overview of some of the directions geospatial standards are evolving.  It was really quite amazing to hear about such a range of application areas where OGC standards are making it possible for the exchange of information between different devices from different vendors.

GeoSMS and GSS

Carl gave several examples of how the GeoSMS simple messaging protocol and GeoSynchronization Services (GSS) are being applied in the real world.  An important example is a mobile use case involving an earthquake in a densely populated area.  A combination of GeoSynchroization Services (GSS), GML, and WFS standards allows first responders in the field to get situation assessments including maps on a variety of hand held devices.

Distributed geospatial updates

Another very relevant use case that is being tested as part of an OGC testbed involves using GeoSynchronization Services (GSS) to enable a large enterprise to receive real-time geospatial updates published by many users and sources, review them, and then insert those updates into web feature services (WFS) operated by many organizations in many different locations.

Integrating indoor and outdoor navigation

Imagine not only getting directions from your house to a shopping mall, but getting directions from your house to the specific counter in the hardware store where plumbing fixures are displayed.  This type of  integrated indoor/outdoor navigation applications are being developed using OGC CityGML and WMS standards. 

Augmented reality standard

Augmented reality involved superimposing imagined structures on the real world.  It is being widely used by architects and designers to visualize what a new building or other structure will look like in situ. At the OGC there is a Augmented Reality Markup Language (ARML) that is under development as a profile of the widely used OGC KML standard developed by Google.

Linking citizen observatories

Recently there has developed an wareness that "it is no longer sufficient to develop passive lists or reports to ‘inform’ citizens of changes in our environment. We need to engage with citizens and ask how they can ‘inform’ us."  (Prof. Jacqueline McGlade, Executive Director, European Environment Agency).  As a result the concept of a citizen observatory has evolved which allows citizens to monitor and report on environmental issues such as biodiversity loss, environmental degradation, and the impact of climate change. 

The Citizen Observatory Web (CobWEB) is intended to link these citizen observatories, providing a standards-based way of linking to sensors and mobile devices and enabling crowd-sourced, geospatially-based information about our environment. 

It will rely on the OGC Sensor Web Enablement (SWE) standard, that provides interoperability interfaces and metadata encodings for real time integration of heterogeneous sensor webs involving devices such as flood gauges, air pollution monitors, stress gauges on bridges, mobile heart monitors, Webcams, and robots as well as space and airborne earth imaging devices into the information infrastructure.

Internet of things

It is projected that the Internet of things, the interconnection over the internet of everyday intelligent objects, will include15 billion devices ranging from pollution monitors to refrigerators by 2015.  It will involve indoor/outdoor location, sensor webs, building information models (BIM), intelligent devices comprising the smart grid, devices for environmental monitoring, and household appliances. 

It will require standards such as the Sensor Web Enablement (SWE) standard to allow all these devices, from many different vendors, to intercommunicate.  Carl gave two examples, one involving tracing fresh meat, absolutely critical in tracing meat as a source of infection, and another application used to monitor debris flow in Taiwan.

Importance of location standards

The most important things that I came away from Carl's talk with are first the wide range of domains where geospatial standards are being applied and, secondly, the realization that standards are becoming even more important than before, because we are on the brink of interconnecting billions of devices from hundreds of thousands of different vendors in many different countries over the internet.  Without standards it will be impossible to turn this data into actionable information. 

The other aspect of this new world of interconnected devices is that location is also becoming more important because, thanks to inexpensive GPS chips and the development of new GNSS systems by Europe and China and others, sensors these days can report their location.

Augmented reality technology preview

There is a new technology preview at Autodesk Labs, Augmented Reality for Showcase.  Augmented reality makes it possible to overlay information (graphics, text, video, sound) on to a live video feed of the real-world in real-time.  Based on Autodesk Showcase, which is an application that produces photorealistic high quality interactive 3D real-time rendering of design models, the Augmented Reality plug-in makes it possible to visualize models in the real-world as viewed through your web or video camera.

The process to overlay a design model on the real world as seen through your video camera is pretty simple.   You prepare your model in Showcase and then associate a supplied marker (jpg file) with your virtual object.  You print the marker and place it in the real world and point your web camera at the printed marker.  Then Showcase replaces the marker in your video recording with the model you've created in Showcase.  The technology preview will be available until October 31, 2012.  There is a video on Youtube that gives you an introduction to the augmented reality preview.