DevSummit 2019 Presentation: Augmented Reality Paired with Computer Vision
Argis Solutions’ CEO Brady Hustad presented on computer vision and augmented reality and how to use the Open Computer Vision Library with ArcGIS at Esri’s 2019 DevSummit.
At Esri’s 2019 DevSummit held in Palm Springs, CA on March 5-8, machine learning and ArcGIS REST JS created a big buzz. Another hot topic among DevSummit attendees was how developers are switching to 4.x JavaScript libraries, which allows for more 3D capability and better functionality. This conference is the annual opportunity for Esri’s expert developers to share their technical knowledge with other software developers to help them write better code, build better systems, and create state-of-the-art apps that utilize ArcGIS mapping technology. While there are 8 rooms simultaneously hosting a full daily schedule of talks, there’s also a little time for dodgeball and beer!
Argis Solutions CEO Brady Hustad had the honor of sharing how to integrate computer vision and Esri, explaining how to use the Open Computer Vision Library with ArcGIS. The presentation covered the technical setup of Open CV, some interesting tips on how to successfully connect it to ArcGIS, and coding tips that will help Open CV function properly for developers interested in using computer vision in their next project. Reflecting on his talk, Hustad remarked, “It was exciting to see how people are getting creative and how computer vision and machine learning are impacting the way they are doing business. People are seeing ways to go beyond doing maps and creating systems software.”
Tackling Decision-Making with Technology
Computer vision enables a computer to see something and make a decision that a human no longer has to make. For example, if hundreds of miles of roadways need to be analyzed for damage, computer vision could allow the computer to look through thousands of images and determine which images contain road damage. In robotics, computer vision has been one of the hardest problems for programmers to solve. Now with access to libraries such as OpenCV and some some finesse to connect it to ArcGIS, programmers are able to attempt to integrate this complex technology in ways that will shape the future of business.
Big cities can be GNSS black holes—their large volume of concrete and metal throws off GNSS service and blocks signals. It can be difficult to get accurate geographic data. Using the rough satellite location of the mobile device, the computer can compare and compute location using two known points, giving improved accuracy in the city. Computer vision could enable the mobile device to detect an asset such as a manhole cover, storm drain, or hydrant. It just needs to be trained to see these known above-ground facilities, enabling the ability to generate accuracy where before none existed.
Together, AR and computer vision could be used to document missing GIS assets. The computer would be programmed to be passively viewing in the background with the ability to notice an asset that is not documented in the ArcGIS data. Then the computer would generate a basic record with spatial location. Its final step would be to ask the end user for further details, for instance: “This asset is not found in your data. Is this correct?” This would be an efficient way to quality check data.
Computer vision could also be used to train a computer to look at an asset when a field worker is looking at it and then automatically pull up the right manual to work on that asset, streamlining field work. As AR visualizer improves with computer vision and image recognition this will all be possible. Computer vision is a game changer in improving accuracy and processes.
Computer Vision: Next Steps
How do we get there? Spatial referencing is required. Once you know where something is in space by way of dual cameras, you can extrapolate 3D space around it. Most mobile devices now come standard with dual cameras, paving the way for programming these functionalities. Feeling inspired? The code used in Brady Hustad’s talk is available on GITHub for developers interested in building projects using the OpenCV library and ArcGIS.
What is needed to program computer vision? A knowledge of a modern programming language like Swift, C++, Java, Python, Kotlin, etc. A developer will also need to be savvy with mobile software like ARCORE, ARKit, OpenCV, ArcGIS, etc. Lastly, a project using Open CV and ArcGIS will need decent software and high availability GIS data. If you would like tips and further information on programming, please visit GITHub, where our example is built in Android using Java and Esri’s ArcGIS Runtime SDK for Android.