“The main task of the augmented reality system is to seamlessly integrate the real world and virtual objects, and it is necessary to solve the problem of synthesis consistency between real scenes and virtual objects. In order to ensure the seamless integration of real world and virtual objects, according to Ronald Azuma’s definition of augmented reality technology, three key problems and three key technologies must be solved in the development of AR application systems.

“

Augmented reality system

The main task of the augmented reality system is to seamlessly integrate the real world and virtual objects, and it is necessary to solve the problem of synthesis consistency between real scenes and virtual objects. In order to ensure the seamless integration of real world and virtual objects, according to Ronald Azuma’s definition of augmented reality technology, three key problems and three key technologies must be solved in the development of AR application systems.

01
Augmented Reality

Three Key Issues in Application System Development

The three key issues in the development of augmented reality application systems are how to solve the consistency problems of real scenes and virtual objects in terms of geometry, lighting, and time. Geometric consistency is the most basic requirement to solve the spatial consistency between virtual objects and real scenes; illumination consistency is the requirement for realistic rendering of virtual-real fusion scenes; temporal consistency is the requirement for real-time interaction. Among the three major problems, geometric consistency and temporal consistency are the prerequisites for studying illumination consistency, because only by efficiently and real-time recovery of the geometric representation of the scene can accurate illumination restoration be performed, and a fusion effect with a strong sense of reality can be obtained.

02
Augmented Reality

Three Key Technologies of Application System Development

According to the definition of augmented reality proposed by Ronald Azuma, the three key technologies of augmented reality application system development can be summarized as three-dimensional registration technology, virtual-real integrated Display technology and human-computer interaction technology. In order to solve the three key problems faced in the development of augmented reality application systems, three key technologies must be solved.

(1) 3D registration technology

3D registration technology is the basic technology for realizing mobile augmented reality applications, and it is also the key to determine the performance of mobile augmented reality application systems. Therefore, 3D registration technology has always been the focus and difficulty of mobile augmented reality system research. Its main task is to detect the pose state of the camera relative to the real scene in real time, determine the position of the virtual information that needs to be superimposed in the projection plane, and display the virtual information in the correct position on the screen in real time to complete the three-dimensional registration. .

There are three main criteria for the performance judgment of registration technology: real-time, stability and robustness.

At present, the research on mobile augmented reality systems based on mobile terminals mainly adopts the following registration methods: computer vision-based, hardware sensor-based and hybrid registration methods, as shown in the figure:

Figure: Classification of Mobile Augmented Reality 3D Registration Technology

Registration algorithm based on computer vision: mainly refers to the process of identifying, tracking and locating the real scene through the knowledge of image processing after using computer vision to obtain the information of the real scene. The registration algorithm based on computer vision is further divided into registration algorithm based on traditional logo and registration algorithm based on natural feature points without logo.

Registration algorithm based on hardware sensors: The hardware sensor tracking technologies of traditional augmented reality systems mainly include inertial navigation systems, global positioning systems (GPS), electromagnetic, optical or ultrasonic position trackers, etc. Among them, the main problem of the inertial navigation system is that the tracking error of the angle and position of the tracked object will increase with time, the drift will be large, and the volume and weight of the equipment will also be large; GPS positioning error is large, indoor, canyon or In the case of other complex terrains, GPS signals are often unable to be received normally; electromagnetic, optical or ultrasonic position trackers adopt the working method of transmitting and receiving to track, and the use occasions are fixed and the range is limited.

In application fields such as maintenance induction, education and training, the matching accuracy is relatively high, and a large registration error will destroy the user’s correct perception of the surrounding environment and change the coordination of the user’s actions in the real environment. Therefore, to achieve accurate augmented reality 3D registration, it is necessary to have high-precision tracking equipment. Commonly used hardware sensors on mobile terminals include gyroscopes, speed sensors, magnetic field sensors, and orientation sensors. This registration method is easily disturbed by the environment, and the registration is imprecise.

Hybrid registration method: Dularch and Mavor have concluded that due to the inaccuracy of the system and the limitation of system delay, the current single tracking technology cannot well solve the azimuth tracking problem of the augmented reality application system. Therefore, using the hybrid tracking method to track and register the augmented reality system is also the research direction of the researchers of famous universities and scientific research institutions at home and abroad. The hybrid tracking registration algorithm is mainly to achieve more accurate registration results, combining the registration algorithm based on computer vision and the registration algorithm based on hardware sensors.

(2) Virtual reality fusion display technology

At present, the main devices for augmented reality systems to realize virtual-real fusion display are generally divided into: helmet display type, handheld display type, and projection display type.

Head-mounted displays are widely used in augmented reality systems to enhance user immersion. According to the realization principle, it is roughly divided into two types: optical see-through type and video see-through type, as shown in the figure. The optical perspective augmented reality system has the advantages of simplicity, high resolution, and no visual deviation, but it also has the defects of high positioning accuracy, difficult delay matching, relatively narrow field of view and high price. The video see-through HMD (Video See-through HMD) based on the video synthesis technology adopted by the video perspective augmented reality system uses the video information of the real environment collected by the camera to be integrated with the three-dimensional virtual information generated by the computer, so as to enhance the user’s perception of reality. Cognitive ability of world data information.

Figure: Classification of helmet-mounted displays

Handheld display devices generally refer to the displays of mobile terminal devices such as mobile phones and tablet computers. They have the advantages of high portability and can be used anytime and anywhere, and the handheld display devices have the characteristics of touch control, which is convenient for human-computer interaction. design.

Projection display is an enhanced display technology that directly projects the generated virtual object information into the real scene that needs to be fused. Projection display can project images into a wide range of scenes, but the projection equipment is bulky and easily affected by changes in lighting. It is suitable for indoor scenes, but not suitable for large outdoor scenes.

There are two main problems in the research of virtual-real fusion scene display research:

One is how to complete the fusion and superposition of real scene and virtual object information, and the other is how to solve the phenomenon of virtual object information delay in the fusion process.

For the optical see-through head-mounted display, the user can see the scene in the surrounding real environment in real time, and the virtual object information that enhances the real scene can only be displayed on the head-mounted display after a series of system delays. When the user’s head or the surrounding scene and objects change, the system delay will cause the “drift” phenomenon of the augmented information in the real environment. The use of video perspective display mode can solve this problem to a certain extent. Developers can control the display frequency of video display and virtual object information through programs, which can meet real-time requirements and alleviate or even eliminate the phenomenon of “drift”. This paper studies the augmented reality technology based on mobile terminals, which is similar to the video perspective to a certain extent, but the handheld display can see a wider scene, but the immersion is not as strong as the video perspective helmet display.

(3) Natural human-computer interaction technology

Augmented reality system interaction technology refers to the process of inputting the user’s interactive operation into the computer, and then displaying and outputting the result of the interaction through the display device after processing.

At present, there are three main types of interaction methods in augmented reality systems: external devices, specific signs, and hand-to-hand interaction.

External equipment: such as mouse and keyboard, data gloves, etc. Traditional PC-based augmented reality systems are used to using keyboard and mouse for interaction. This interaction method has high precision and low cost, but the immersion is poor. The other is to use data gloves, force feedback devices, magnetic sensors and other devices to interact. This method has high precision and strong immersion, but the cost is relatively high. With the development of wearable augmented reality systems, voice input devices have also become one of the interaction methods of augmented reality systems, and have great development prospects in the future.

Specific Logos: Logos can be designed in advance. Through a more advanced registration algorithm, the logo can have a special meaning, and the user will know the meaning of the logo after seeing the logo. Therefore, the interaction based on the specific logo can make the user clearly understand the operation steps and reduce the learning cost. This method is slightly more immersive than traditional external devices.

Freehand interaction: One is a natural gesture interaction method based on computational vision, which requires the help of complex human hand recognition algorithms. Firstly, the human hand is extracted from the complex background, and then the movement trajectory of the human hand is tracked and positioned. Finally, the operator’s intention is estimated and correctly mapped to the corresponding input according to the gesture state, the current position and movement trajectory of the human hand and other information. in the event. This kind of interaction has the strongest sense of immersion and low cost, but the algorithm is complex, the precision is not high, and it is easily affected by conditions such as lighting. The other is mainly for mobile terminal equipment. Nowadays, the display devices of mobile terminals all have a touch function, and even support multi-touch. So you can interact by touching the screen. Almost all mobile applications today use this type of interaction.