Description of work content and effort of the work steps
- Conceptual design and implementation: Four components are required for the development of an AR streaming solution: a) methods for rendering digital 3D objects (rendering), b) pose-tracking methods, c) interaction capabilities, and d) a server-client communication. At the beginning, a literature review is conducted to identify suitable methods. Subsequently, it is conceptually worked out how AR streaming can be realized with the help of the individual methods. Based on the concept, the individual components will be implemented and turned into an overall solution.
a
Methods for rendering virtual objects
In order for the virtual objects to be visualized in AR, they must be processed by the system. In the first step, an interface for reading in 3D models is implemented. A parser is used to capture the model data and transfer it to the system. The raw data of the 3D models is then processed for rendering. This includes, for example, triangulating the data and converting it into an efficient data structure (e.g. Octree). For rendering the 3D data, a virtual environment is created with a virtual camera for AR.
b
Pose tracking procedure
For the accurate positional insertion of virtual 3D models in reality, the indoor positioning (AP 2) is extended by a visual pose tracking method (Natural Feature Tracking) for the highly accurate real-time determination of the position and rotation (pose) of the AR system. The coarse pose from AP 2 serves as the basis for determining the fine pose. Factors to be considered include different coordinate systems and update rates to merge the coarse and fine pose. Finally, the fine pose is used to synchronize the virtual world with the real world.
c
Interaction possibilities
For the interaction with the AR environment, input methods are designed and implemented, for example to retrieve and provide information of the virtual objects in AR. For this purpose, various gestures (tapping, swiping, etc.) are to be captured and processed from the smartphone screen and transferred to the AR.
d
Interface for server-client communication
The complex rendering and pose tracking procedures are computed on a server and the results are transferred to the mobile device. For rendering, methods for sending render data as well as object information are implemented on the server and methods for receiving on the client. For pose tracking, methods are implemented for sending raw data (e.g. sensor data, camera images) on the client and receiving it on the server, and methods are implemented for sending result data (pose information) on the server and receiving it on the client. - Laboratory tests and empirical evaluation of the results (milestone 1).
By means of initial investigations, the implemented components and the overall solution are evaluated by means of laboratory tests in order to identify optimization potential and to plan further steps to improve performance. - Increasing the update rate of the AR streaming solution
Solutions for performance improvement based on the identified potentials will be developed and implemented. Among other things, the relationships between visual quality and computational effort for rendering and between accuracy and computational effort for pose tracking are investigated and optimized. - Verification of the robustness of the streaming performance in the TIP (milestone 2).
Finally, the AR streaming solution will be evaluated in the TIP with respect to robustness and performance. In particular, stress tests with large 3D models will be performed to determine the performance limit of the solution and to confirm the robustness.
Outcomes and risks; milestones with termination criteria.
- Conceptual design and implementation
- Laboratory tests and empirical evaluation of the results (milestone 1).
- Increasing the update rate of the AR streaming solution
- Verification of the robustness of the streaming performance in the TIP (milestone 2).
Milestone 1
Laboratory tests and empirical evaluation of results successfully completed
Milestone 2
Successful verification of the robustness of the streaming performance in the TIP
Risks/Consequences
The AR streaming solution or a component may be inadequate (e.g., in terms of data throughput or due to errors in data transmission), so that the visualization quality, update rate (speed), accuracy, or performance are too low for the purposes of the project. In this case, ways must be found to improve the streaming solution (e.g., bundling of channels) and/or compromises must be made in the individual components to obtain a runnable but reduced overall system.
