Goal: Assistance system
Description of work content and effort of the work steps
- Conceptual design of the AS on mobile devices
A concept for data management, graphical user interface (GUI) and the associated interfaces is created. In the environment of a mobile application, which is also used in motion, special emphasis must be placed on simple and intuitive use, which is accessible to the user in just a few moments. In addition, the data management and the response of the overlying app layer must be designed in such a way that the app remains operable even if large amounts of data are streamed in the background and even if the data transfer does not work as intended. - Creation of the alpha version of the AS (milestone 1)
a
Creation of the functional prototype
The prototype is developed as a Minimum Viable Product (MVP) to demonstrate the basic approach. The productive interfaces to the streaming server are not yet realized, static laboratory data is used.
b
Development of an intuitive UX and transfer of the prototype
In this step, the prototype is provided with an intuitive user interface that takes the special usage situation into account; the mobile client must be intuitive and naturally operable in motion and provide visual and, if necessary, haptic feedback on user inputs. - Integration of the IP and AR solution into the AS
a
Implementing the interfaces as well as data handling and caching
The IP algorithm is integrated into the app so that the local context required for AR visualization is available with sufficient accuracy, especially indoors. In addition, the streaming server services are addressed via the defined interfaces instead of the lab data so that the context-specific data can be triggered reliably depending on the location and the content can be streamed in the correct context and displayed by the AS. If necessary, a cache is implemented in the AS that receives the streaming data and forwards it to the user interface in a harmonized manner in order to ensure a maximally "fluid" display of the AR streaming data.
b
Continuous visualization of the dynamized data in the app as an AR scene
The synchronization and display of different AR streams depending on the user context must be managed. Depending on movement and dwell time, the AS must be able to trigger different streams accordingly so that the object in focus is delivered with priority. - Development of the IoT module to provide the data from the IoT sensors and integration of the IoT-based real-time sensor data into the AS.
A solution architecture is being developed that enables sensor data to be transferred to the cloud via the 5G network. For this purpose, it is specified which data is relevant for the project and a corresponding sensor is selected. On the cloud side, IOX will provide an infrastructure with broker and IoT device management, which will store the sensor data and make it available for access from the mobile device.
Outcomes and risks; milestones with termination criteria.
- Concept for an AS on mobile devices with associated interfaces
- Alpha version of the basic AS (milestone 1)
- AS with integrated IP and AR visualization
- IoT module for providing the data from the IoT sensors and integration into the AS
Milestone 1
Alpha version of the AS successfully created
Milestone 2
IoT module successfully created
Risks/Consequences
The large number of mobile end devices available will pose a particular challenge in practice. The ideal balance must be found between the computationally intensive work on the mobile end device and the data transmission and latency of the network with decentralized computation. In any case, sufficient performance of the mobile device is required despite the outsourcing of processing (e.g., rendering processes) to dedicated server processes. There is a risk that the performance is not sufficient. In this case, the algorithms have to be optimized and, if necessary, different rendering qualities have to be delivered depending on the performance level of the used end device and the speed and stability of the network in order to ensure a smooth visualization. Furthermore, the orchestration of the streaming focus as well as the cache management is a factor that decisively influences the usability, because here the data streaming must be sensitively coordinated with the user movements.
There is a risk that the latency of the data from the sensor to the cloud and from there to the mobile device cannot be transmitted with the desired authenticity using NB-IoT. In this case, one could look for and integrate suitable 5G sensor hardware in the field of uRLLC. When using uRLLC, the building penetration of the 5G network may not reach all areas of the building. In this case, real-time capability would need to be coordinated and balanced with user needs (real-time vs. data availability).
