The Use of Augmented Reality in Geomatics

. User interfaces are in continuous progress. As the computing power of modern machines grows, they become more user-friendly and intuitive. Not all solutions are widely accepted, sometimes they become only a “curiosity”, while another ones achieve success. Lately, some user interface designers strive for such solutions, in which the user will have the impression of “staying” or “permeation” of the system with reality and therefore some kind of software integration with the environment. This is achieved by various methods utilizing interfaces controlled by voice or touch. Quite spectacular and very interesting are solutions that integrate image generated by a computer with a real view. This technology is called AR – Augmented Reality, and is the core of the author’s considerations about its application in contemporary surveying and GIS practice. In this article, are presented issues related to the possibilities that lie in the use of this technology in the daily work of geo-engineer.


Introduction
Modern society has long been fascinated by the technologies enabling people to cross the borders of their natural abilities. The man wants to improve his physical function, memory and concentration, wants to see more, or to communicate over long distances in a fraction of a second. Initially, such ideas occur mainly in the field of science fiction, largely due to technical limitations. Then gradually, with the development of technology, they are implemented or abandoned -depending on the idea. One of the many ideas that fires the imagination and hope of people since years, is a virtual reality and its different varieties. Modern computers for a long time have sufficient computing power to implement this idea in practice. The advent of widespread use of mobile devices (smartphones, tablets), which, in fact, are computers with quite big computing power and many sensors, made it possible to implement virtual reality in completely different way. It is called: Augmented Reality (AR). AR means any system that "augments", or overlays, the real world with digital information that seems to co-exist with the real world (Azuma 1997). Over the past few decades, this once theoretical field has matured into a mass medium (Ahonen 2012) with applications spanning countless industries. Speaking of AR used in mobile devices, many authors (Höllerer, Feiner 2004;Henrysson 2007) use the term MAR (Mobile Augmented Reality), to clarify the scope of application of this technology. AR technology, can be compared to a real time digital image editing, or a special video effect, similar to the techniques used in film editing or television. This allows to create the illusion of an "extension" of the observed reality with additional information and images. View generated this way can be named as first "perspective view", which also depends on the position and orientation of the user's device. Therefore, most commonly used as the output device is portable equipment such as smartphones, tablets or displays worn over the head like glasses, which are generally referred to as Head Mounted Displays (HMD), rather than standard PC computers or laptops.
The effect of combining digital image with the real view is obtained by the use of the digital camera and the sensors like GPS/GNSS receiver, inclinometer, magnetometer, accelerometer, compass and other, which provide information about the location and orientation of the device. As Siltanen (Siltanen 2012) writes that AR research must take into account such two fields like: -computer graphics (photorealistic rendering and interactive animations), -computer vision (marker and feature detection and tracking, motion detection and tracking, image analysis, gesture recognition and the construction of controlled environments containing a number of different sensors). I would expand this list with three more fields: -GPS/GNSS and indoor navigation (localization purposes), -digital cartography and mapping (information presentation and georeference), -WWW applications, Service Oriented Architecture and Databases (for data storage, obtaining and communication).
Bednarczyk, M. 2017. The use of augmented reality in geomatics 2 -Augmented Reality's potential in different fields of usage seems to be very wide. Applications of this technology can be found in such fields like, for example: -architecture -presentation of three-dimensional models of buildings, housing estates and cities. Visualization of the details of construction, including installation and equipment (Broschart et al. 2013;Xiangyu 2009; VTT 2013) -interior designing and decoration -e. g. virtual furniture or household articles may be overlaid in real time to check how they fit in a room (Augment 2017), -sale and marketing -e.g. to see augmented 3D models of the products presented in catalog, or even virtually "place" them at home using smartphone (IKEA 2017) -spatial designing and planning -presentation of three-dimensional scale models, visualizations of projects in a real environment (Moloney, Bharat 2011) -augmented reality maps and GIS -maps, the content of which is expanded with additional information in the form of descriptions, graphics or 3D objects (Moloney, Bharat 2011) -tourism -tourist attractions, which can be observed directly on the smartphone screen, giving additional information during the tour (Kounavis et al. 2012). Observing the variety of disciplines that implements AR, even only those mentioned above, the author decided to consider whether and how to use it in the field of geodesy and GIS. Although there are reference attempts to this subject (Halik 2012a(Halik , 2012b, but still in the field of geodesy, AR seems to be rather little exploited.
When considering the use of AR in other areas, the hypothesis can be formulated that AR technology might be used and helpful in surveying. The purpose of this work is to refer to mentioned problem which seems to be quite wide. For this reason, the complete and exhaustive consideration of this issue goes beyond the scope of a single article.
To achieve the objective of this paper, the author decided to use AR in practice in two experiments. First was to use computer-vision based AR for "augmentation" of data contained on a traditional map, the second was to indicate the location of geodetic control points, and to consider other possible applications of AR technology in GIS and surveying. This preliminary raise of the subject led the author to a deeper exploration of the problem and to undertake further work in the near future.

AR technology
As mentioned in introduction, AR mixes virtual objects (graphics, sounds, videos,3D models etc.) with the view of actual world. Augmented Reality is a system that has the following three characteristics (Ronald Azuma AR Definition): -combines real and virtual, -interactive in real time, -registered in 3-D.
As it is a fairly new technology, here are a few definitions related to AR. They were taken from AR Glossary, document published by the AR Community (AR Community 2017) to standardize terminology.
-AR implementation or AR application -means any service that provides augmentations to an AR-ready device or system. -Device -the hardware unit the AR implementation is running on.
-Augmentation -a relationship between the real world and a digital asset. The realization of an augmentation is a composed scene. An augmentation may be formalized through an authoring and publishing process where the relationship between real and virtual is defined and made discoverable. -Digital asset (digital object or virtual object) -data that is used to augment users' perception of reality and encompasses various kinds of digital content such as text, image, 3d models, video, audio and haptic surfaces. A digital asset is part of an augmentation and therefore is rendered in a composed scene. A digital asset can be scripted with behaviors. These scripts can be integral to the object (for example, a GIF animation) or separate code artifacts (for example, browser markup). A digital asset can have styling applied that changes its default appearance or presentation. Visual Assets are digital assets that are represented visually. -Composed scene -produced by a system of sensors, displays and interfaces that creates a perception of reality where augmentations are integrated into the real world. A composed scene in an augmented reality system is a manifestation of a real world environment and one or more rendered digital assets. It does not necessarily involve 3D objects or even visual rendering. The acquisition of the user (or device)'s current pose is required to align the composed scene to the user's perspective. Examples of composed scenes with visual rendering (AR in camera view) include a smartphone application that presents visualization through the handheld video display, or a webcam-based system where the real object and augmentation are displayed on a PC monitor. -Camera View or AR View -the term used to describe the presentation of information to the user (the augmentation) as an overlay on the camera display.
-Pose -six degrees of freedom -a real object in space can have three components of translation -up and down (z), left and right (x), forward and backward (y), and three components of rotation -pitch, roll and yaw. Hence, the real object has six degrees of freedom. -Reference image -an image representing a real object, that is used by image recognition algorithms to match a frame from composed scene so that an anchor point for augmentation can be identified. -Due to the variety of technologies used to achieve the effect of "augmentation", there are two primary techniques for image generation in AR (scene composing) (Lechner 2015; AR Community 2017).

Computer-vision based AR
This technique uses computer vision algorithms to process the image, viewed by the camera, and compare it with the reference images. The AR effect takes place when the application recognizes and identifies the fragment (or fragments) of an image (based on the pattern) and displays on this place virtual objects. Figure 1 shows the example of "augmentation" performed this way. In geodesy, this technique can be used for example with paper maps or maps displayed on the screen.

Geospatial AR
Geospatial AR means AR experiences based on the user's location and orientation in a geographic coordinate space. Location determining and object tracking relies on geo positioning techniques such as GPS/GNSS, WiFi or cellular positioning. Sometimes, the user can enter the location manually eg. by scanning specially prepared marker. The user's orientation is approximated, from the movement of the device using sensors such as a digital compass, accelerometer or gyroscope. Once approximated device's orientation may be also refined by using computer vision techniques. To compute the AR effect, the real feature's location is needed. The real feature is represented on the display, by any digital object (virtual object). The "augmentation" effect takes place when a user is in the vicinity of the real object and points on it with a camera. Then the software will automatically determine the location and orientation of the device and display respectively generated image (virtual object). Figure 2 shows the example of "augmentation" performed this way. In geodesy this technique can be used for example while surveying.

Examples of AR in geomatics
After getting acquainted with the subject, the author decided to make an attempt to use AR technology in the selected range in the fields of geodesy and GIS. For this purpose, two experiments were performed. The first concerned the use of computer-vision based AR for "augmentation" of data contained on a traditional map. The second concerned the use of geospatial AR to indicate the location of geodetic control points in the field. Both tasks were performed by creating own software using Wikitude library, providing AR functionality. Although described software works on mobile devices (smartphone, tablet) on Android, it is, however, created as multi-layer software. Therefore some of its components are server-side scripts and the database.

Computer-vision based map augmentation
The purpose of this experiment was to examine the possibility of "augmentation" of the data contained on a traditional map. The tests were carried out both on the paper map and on a computer screen. On the map was selected fragment, representing the object (a building). It was subsequently used as the reference image for computer vision algorithms (Fig. 3). The virtual object was a small picture of the building (sample photo was used). After entering the data into the application, real time tests were performed. It was observed, that the composition of the AR scene takes place very quickly. The algorithm immediately detects the specified portion of the image and displays the virtual object (Fig. 4). This occurs, however, under certain circumstances: -first of all, the camera has to focus properly, -the distance between camera and the map can not be too large, although the recognition occurs at quite a wide range of distances, which is convenient. -proper lighting conditions. In case of this experiment, the reference image was created from a raster image with perfectly white background and black drawing. Natural light reflected from the paper changes the shade of original color, which makes that the algorithm has difficulties with recognizing the image. This results in "blinking" of displayed virtual object or complete lack of it on the screen. Ultimately, the problem was solved by applying the additional lightsource.
Also, the angle of view is quite broad. After the first recognition and composing a scene, the angle of the camera can be changed in the range of several tens of degrees without losing the visibility of the virtual object, which can be seen in the figure 4 (b) and (d).

Location of geodetic control points in geospatial AR
The aim of the second experiment was an attempt to use geospatial AR to support surveying by visualization of the approximate location of geodetic control points. The application might be helpful in finding points in the field during the measurement. In this task, it was decided to use the existing database of points, which are located around campus Kortowo in Olsztyn. This aim was achieved by creating a network service that provides data from database management system, which then goes to the application on the smartphone. Authorial software used in this experiment therefore consists of three primary elements (Fig. 5): − database with geodetic points data − network service used to retrieve data from the database − mobile AR application running under Android OS Application communicates with the database by a network service using HTTP protocol. Location of the device is determined all the time while the application is running. On a basis of device's coordinates (latitude, longtitude), a request to the service is constructed, which searches the database and provides data concerning points in the immediate vicinity of the received location (e.g. within 1000m). This query is performed each time the device's location is changed. Example of request to the service transmitted by HTTP GET method is as follows (the network address is an example): http://www.exampleadress.com/ar_points.php?lat=53.7619565305&lon=20.4595679909&range=1000.
The application displays virtual objects indicating the approximate location of the control points, depending on the position and orientation of the device (Fig. 6). In geospatial AR, position of virtual markers depends on 6 factors and it is not always determined precisely enough. According to the error propagation law, any measurement error affects the value established on the basis of measured quantities. In this case we are dealing with geolocation errors and sensors errors. Due to a multiplicity of factors, obtaining accurate result is difficult. The measurements must be sufficiently precise to put a virtual object accurately on the screen. During the tests it was observed that the virtual object does not always point the exact place that it should. In addition, the effect of "swimming" virtual objects positions can be observed. This results from the need to stabilize the operation of sensors after making a movement. Of great importance here is the accuracy of a GPS fix, as well as the distance between the device and the real object. The greater the error of GPS position and closer distance to the real point, the larger the position error of virtual object on the screen. These are the preliminary conclusions from the observation of this experiment's implementation. The accuracy of geolocation can be significantly improved. Modern methods used in the measurement of GNSS, especially the use of GNSS permanent stations (Rapiński, Kowalczyk 2016), gives the opportunity to determine the position and height with high accuracy, but they are not supported in today's smartphones.
In the near future the author intends to undertake more detailed study focused on the factors, affecting the accuracy of composing AR scene, for the purpose of surveying and looking for solutions to improve this process. However, the results obtained in this experiment indicate that, the use of AR technology to indicate the approximate location of geodetic control points is potentially a good idea and can be applicable in geodetic practise.

Conclusions
Conducted experiments showed that the use of augmented reality technology, in presented range of geomatics, is possible. Nevertheless, there is a need to continue research on the accuracy of generating AR scene. Many issues remains unresolved, such as the presentation of other than point type objects and three-dimensional models for the needs of geodesy and GIS. The use of AR in surveying will be justified only if it will be possible to achieve a sufficiently high location precision of virtual objects on the screen. Author believes that it is possible, however, there should be undertaken studies for obtaining higher precision of parameters of the mobile device's orientation and location. Nevertheless, examples mentioned in this article indicate a high potential of AR technology and even in the form presented in this paper can be a tool to facilitate the work of the geodesist. The author assumes the continuation of research in this area, both regarding the processing algorithms and the search for hardware solutions if needed.
AR technology provides new opportunities to build the user interface for the applications. Each virtual object can be interactive, which makes that a part of the application's functionality can be realized from the "augmented" view. In addition, we gain an additional functional dimension of GIS systems. They don't need to be associated only with the digital map. With AR, paper map also becomes interactive. In the future, it'll be possible to develop a system that performs analyzes on "augmented" paper map.