VR-based Assistance System for Semi-Autonomous Robotic Boats

established Updated: 2026-05-31
vr teleoperation robotic-boats asv human-ai-interaction maritime hci hmd unity ros remote-control

VR-based Assistance System for Semi-Autonomous Robotic Boats

Status: established
Last updated: 2026-05-31
Sources: 3610978.3640750.Pdf
Tags: [vr, teleoperation, robotic-boats, asv, human-ai-interaction, maritime, hci, hmd, unity, ros, remote-control]

Summary

Reitmann & Jung (2024) present the concept and a working technical prototype of a virtual-reality teleoperation system for semi-autonomous Autonomous Surface Vehicles (ASVs), built around a Clearpath Kingfisher boat, a HTC VIVE Focus 3 head-mounted display, a 360° camera, and a Unity scene fed by ROS over a TCP connection. The operator monitors autonomous driving or takes direct manual control, with live sensor data (camera, GPS, IMU, depth/sonar) and past measurements registered into one immersive view. Initial field tests on a small inland lake confirmed the concept works and surfaced the challenges the paper then addresses: connectivity and bandwidth over distance, motion and sea sickness, the design of a context-driven assistance handler, and AI-based context identification across autonomous, semi-autonomous, and manual modes. Within this knowledge base the paper is the maritime-remote-operation anchor that grounds the VR/HMD strand in an applied teleoperation use case rather than abstract device-tracking comparisons. The paper is a verified, text-based ACM conference paper (HRI '24 Companion), so it is treated as established.

Body

Context

Reitmann & Jung (2024) describe the design, technical architecture, and first field tests of a VR-based human-machine interface (HMI) for remotely supervising and controlling semi-autonomous robotic boats, presented as a five-page paper at the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24 Companion). The work is a concept-and-prototype report rather than a controlled user study: the authors build the system, run initial trials on a lake, and derive the open challenges. Within this knowledge base the paper bridges maritime remote operation and the VR/virtual-environments strand — it is the first source here that places head-mounted-display VR inside an applied teleoperation loop with live robot sensor data, as opposed to the device-level eye-tracking and engine surveys (Openxr Eye Gaze Interaction, Unity Eye Tracking, Eye Tracking Engine Comparison). Its use of a HTC VIVE Focus 3 HMD and a Unity rendering scene connects it to the engine and HMD material already in the KB, while its treatment of VR sickness links it to the broader human-factors concerns running through the VR articles.

Key Points

The system targets a teleoperation loop in which a remotely located operator perceives an ASV's state through an HMI and issues control commands, with the boat itself switching between autonomous, semi-autonomous, and manual modes depending on context (PDF p. 1, orig. p. 877). The motivating problem is that fully autonomous operation of outdoor robots reaches its limits under difficult conditions; an AI-based, adaptable catalog of criteria lets the robot identify the most suitable mode and communicate it to the operator, while the VR interface gives the operator an immersive impression of on-site obstacles and sensor data. Reitmann & Jung frame the VR advantage as improving conflict-resolution strategies and avoiding the problem of mixing different projections (full ASV view versus camera image, map view, or external shore view) (PDF p. 1, orig. p. 877). The prototype's named features are streaming of camera images with the boat position as pivot point to an HMD operator, integration of location and depth sensing, visualization of expected trajectory and bearing, and interactive control for live path-planning adjustment (PDF p. 2, orig. p. 878).

The background situates the work in two literatures. ASVs are defined as robotic boats and swimming platforms used mainly for geo-environmental monitoring and bathymetry, requiring position systems, propulsion, an IMU, and surroundings sensors (Dunbabin, 2015; Dunbabin & Grinham, 2010, 2017; Eichhorn et al., 2017, 2018; Pose et al., 2023; Specht et al., 2017) (PDF p. 2, orig. p. 878). Teleoperation is presented as a long-standing field with results for analogous platforms such as drones, cars, and ships (Di et al., 2011; Kheddar et al., 2007; Lv et al., 2006; Merwe et al., 2019; Monferrer & Bonyuet, 2002; Navarro et al., 2018; Neumeier et al., 2018; Nava-Balanzar et al., 2017) (PDF p. 2, orig. p. 878). The authors draw on prior findings that HMDs enhance situation awareness and task performance (Hosseini & Lienkamp, 2016), that enriched processed views aid orientation and scale (Schimpe et al., 2022), and specifically that VR/AR-HMI users detect collision or grounding situations of surface vehicles better than users of conventional or 3D-GUI interfaces and rate the VR-GUI as both the easiest and the best expert tool (Lager & Topp, 2019; Walker et al., 2019) (PDF p. 2, orig. p. 878). They also flag at the outset that VR-based systems can induce motion sickness from disagreeing sensory stimuli, citing a review of VR sickness causes and measurements (Chang et al., 2020) (PDF p. 2, orig. p. 878).

The technical setup is concrete. The platform is a Clearpath Kingfisher ASV (350 × 980 × 320 mm, 28 kg, electrical water-jet propulsion, 40 N thrust, max 1.7 m/s, GPS + IMU + front camera). The operator wears a HTC VIVE Focus 3 standalone HMD with inside-out tracking for outdoor mobility; a RICOH Theta Z1 360° camera (23 MP, 6720 × 3360) on a telescopic pole captures the surroundings; an Intel NUC barebone running Ubuntu Linux serves as the board computer; and depth sensing uses LiDAR above water and R2Sonic 2020 sonar below (PDF p. 2, orig. p. 878). Connectivity uses a UDP/GStreamer camera stream to the board computer and a 2.4 GHz WiFi link between ASV and remote station rated for up to roughly 100 m in the first prototype, with connectivity flagged as an issue at larger ranges (PDF p. 2, orig. p. 878).

The software core is ROS (Noetic, ROS 1) chosen as the de-facto robotics middleware, with a publish-subscribe topic model; the authors run ROS 1 to match the ASV but plan ROS 2 for ASV-independent scripts (PDF p. 2, orig. p. 878). Unity connects to ROS not as a ROS node but through Unity Technologies' ROS-TCP-Connector and ROS-TCP-Endpoint packages over a TCP connection to the ROS master (PDF pp. 2–3, orig. pp. 878–879). In Unity, a VR camera at the origin is surrounded by a sphere onto which the 360° image is textured (lower half culled to make room for augmented sensor data); a low-poly boat model with an attached compass sits below the camera, a green line shows trajectory and bearing, grey points render a previous sonar point cloud, and a toggleable head-up display exposes network and control settings plus parameters (delay, FPS, speed). The ASV position is synchronized via WGS 84 and the ROS GPS stream (PDF p. 3, orig. p. 879).

The field example tested the model on an inland lake of roughly 100 m × 100 m. Under normal conditions the ASV navigated autonomously and recorded parameters; control in the trial was done both manually (a Logitech F310 gamepad, which replaced the HMD controller because interactive path planning was not yet implemented) and automatically via prepared waypoints (PDF pp. 3–4, orig. pp. 879–880). The authors envision three operating modes the assistance should help distinguish: autonomous normal mode (VR supervision), graded semi-autonomous interactive control (VR supervision plus interactive path planning), and manual mode (VR visualization plus controller input) (PDF p. 4, orig. p. 880). They plan a user study on the observed correlation between ASV–station distance and delay, focused on connectivity, reliability, and usability (PDF p. 4, orig. p. 880).

The conclusion enumerates four challenges the field tests surfaced. Connection: providing sufficient bandwidth for high-resolution video over a large operating range is essential, with 5G integration planned as an alternative to WiFi. Sea and motion sickness: permanent VR use is difficult and sea sickness compounds it, partly mitigated by an initial IMU-based or 3-axis-gimbal camera stabilization (Ryu et al., 2023; Watanabe & Takahashi, 2020) (PDF p. 5, orig. p. 881). Assistance handler: a context-driven set of assistants (exploration, planning, collision) that either suggest actions for the operator to confirm or check and correct operator-proposed routes, with interactive VR path planning requiring fusion of planar image data and depth data so GPS waypoints can be linked to the imagery. Context identification: a state machine whose transition functions between control modes are concretized with AI, taking a navigation/environment model (water depth, shoreline, obstacles, wind) and a data-collection model (hydrological, chemical, physical target variables) as inputs (PDF p. 5, orig. p. 881).

Conclusion

Reitmann & Jung (2024) conclude that they have built a scalable VR-based teleoperation prototype for robotic boats that runs on mobile hardware, communicates through ROS, and is usable as a monitoring base and, later, an interaction interface. The field tests validated the technical concept and converted it into a concrete agenda: solve bandwidth over range (with 5G), reduce motion and sea sickness (with camera stabilization), build the context-driven assistance handler, and formalize mode-switching through an AI-driven state machine. The takeaway for this knowledge base is an applied demonstration that head-mounted VR plus a 360° immersive view plus live robot sensor data can outperform conventional remote-control interfaces for surface vehicles, with the principal unresolved questions being usability and the distance–delay tradeoff, which the authors defer to a planned user study.

References

Chang, E., Kim, H.T. and Yoo, B. (2020) 'Virtual Reality Sickness: A Review of Causes and Measurements', International Journal of Human–Computer Interaction. To be validated.

Di, Z., Shiqi, L., Wenge, Z. and Mingming, W. (2011) 'A Haptic Interface for Virtual Reality Based Teleoperation System'. Berlin: Springer. doi: 10.1007/978-3-642-19853-3_55. To be validated.

Dunbabin, M. (2015) 'Autonomous Greenhouse Gas Sampling Using Multiple Robotic Boats', in Wettergreen, D. and Barfoot, T.D. (eds.) Field and Service Robotics (Springer Tracts in Advanced Robotics, Vol. 113). Springer, pp. 17–30. To be validated.

Dunbabin, M.D. and Grinham, A. (2010) 'Experimental evaluation of an Autonomous Surface Vehicle for water quality and greenhouse gas emission monitoring', in ICRA. IEEE, pp. 5268–5274. To be validated.

Dunbabin, M.D. and Grinham, A. (2017) 'Quantifying Spatiotemporal Greenhouse Gas Emissions Using Autonomous Surface Vehicles', Journal of Field Robotics, 34(1), pp. 151–169. doi: 10.1002/rob.21665. Validated 2026-06-01: DOI resolved via Crossref API; title, authors (Dunbabin, Grinham), year (2017), journal, volume, pages all confirmed.

Eichhorn, M., Ament, C., Jacobi, M., Pfützenreuter, T., Karimanzira, D., Bley, K., Boer, M. and Wehde, H. (2018) 'Modular AUV System with Integrated Real-Time Water Quality Analysis', Sensors, 18(6), 1837. doi: 10.3390/s18061837. To be validated (DOI from the citing paper's Crossref deposit, 2026-05-31; not independently resolved).

Eichhorn, M., Taubert, R., Ament, C., Jacobi, M. and Pfützenreuter, T. (2017) 'Modular AUV System for Sea Water Quality Monitoring and Management', CoRR, abs/1702.08094. To be validated.

Gnatzig, S., Chucholowski, F., Tang, T. and Lienkamp, M. (2013) 'A System Design for Teleoperated Road Vehicles', in International Conference on Informatics in Control, Automation and Robotics. To be validated.

Hedayati, H., Walker, M. and Szafir, D. (2018) 'Improving Collocated Robot Teleoperation with Augmented Reality', in Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (HRI '18). New York: ACM, pp. 78–86. doi: 10.1145/3171221.3171251. Validated 2026-06-01: DOI resolved via Crossref API; title, authors (Hedayati, Walker, Szafir), year (2018), venue (HRI '18), pages all confirmed.

Hosseini, A. and Lienkamp, M. (2016) 'Enhancing telepresence during the teleoperation of road vehicles using HMD-based mixed reality', in 2016 IEEE Intelligent Vehicles Symposium (IV). IEEE, pp. 1366–1373. doi: 10.1109/IVS.2016.7535568. Validated 2026-06-01: DOI resolved via IEEE Xplore; title, authors (Hosseini, Lienkamp), year (2016), venue, pages all confirmed.

Kheddar, A., Neo, E.S., Tadakuma, R. and Yokoi, K. (2007) 'Enhanced Teleoperation Through Virtual Reality Techniques', in Ferre, M., Buss, M., Aracil, R., Melchiorri, C. and Balaguer, C. (eds.) Advances in Telerobotics (Springer Tracts in Advanced Robotics, Vol. 31). Springer, pp. 139–159. To be validated.

Lager, M. and Topp, E.A. (2019) 'Remote Supervision of an Autonomous Surface Vehicle using Virtual Reality', IFAC-PapersOnLine (10th IFAC Symposium on Intelligent Autonomous Vehicles IAV 2019). To be validated.

Lv, X., Zhang, M., Cui, F. and Zhang, X. (2006) 'Teleoperation of Robot Based on Virtual Reality', in ICAT Workshops. IEEE Computer Society, pp. 400–403. doi: 10.1109/ICAT.2006.124. To be validated (DOI from the citing paper's Crossref deposit, 2026-05-31; not independently resolved).

Merwe, D.B., Maanen, L., Haar, F.B., Dijk, R.J.E., Hoeba, N. and Stap, N. (2019) 'Human-Robot Interaction During Virtual Reality Mediated Teleoperation: How Environment Information Affects Spatial Task Performance and Operator Situation Awareness'. Springer. doi: 10.1007/978-3-030-21565-1_11. To be validated.

Monferrer, A. and Bonyuet, D. (2002) 'Cooperative Robot Teleoperation through Virtual Reality Interfaces', in IV. IEEE Computer Society, pp. 243–248. doi: 10.1109/IV.2002.1028783. To be validated (DOI from the citing paper's Crossref deposit, 2026-05-31; not independently resolved).

Nava-Balanzar, L., Sánchez-Gaytán, J.L., Fonseca-Navarro, F., Salgado-Jiménez, T., García-Valdovinos, L.G., Rubio-Lopez, O., Gómez-Espinosa, A. and Ramirez-Martinez, A. (2017) 'Towards Teleoperation and Automatic Control Features of an Unmanned Surface Vessel-ROV System: Preliminary Results', in ICINCO (2), pp. 292–299. To be validated.

Navarro, F., Fdez, J., Garzón, M., Roldán, J.J. and Barrientos, A. (2018) 'Integrating 3D Reconstruction and Virtual Reality: A New Approach for Immersive Teleoperation'. Springer. doi: 10.1007/978-3-319-70836-2_50. To be validated.

Neumeier, S., Gay, N., Dannheim, C. and Facchi, C. (2018) 'On the way to autonomous vehicles teleoperated driving', in AmE 2018 – Automotive meets Electronics; 9th GMM-Symposium. VDE, pp. 1–6. To be validated.

Pose, S., Reitmann, S., Licht, G.J., Grab, T. and Fieback, T. (2023) 'AI-Prepared Autonomous Freshwater Monitoring and Sea Ground Detection by an Autonomous Surface Vehicle', Remote Sensing, 15(3), 860. doi: 10.3390/rs15030860. To be validated (DOI from the citing paper's Crossref deposit, 2026-05-31; not independently resolved).

Reitmann, S. and Jung, B. (2024) 'VR-based Assistance System for Semi-Autonomous Robotic Boats', in Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24 Companion), Boulder, CO, USA, 11–14 March. New York: ACM, 5 pages. doi: 10.1145/3610978.3640750. vrRoboticBoats2024

Ryu, J., Park, S. and Kim, G.J. (2023) 'Sickness Reduction in FPV Drone Control: Improved Effects of Reverse Optical Flow with Static Landmarks Only', in Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, pp. 1–2. doi: 10.1145/3611659.3617219. To be validated (DOI from the citing paper's Crossref deposit, 2026-05-31; not independently resolved).

Schimpe, A., Feiler, J., Hoffmann, S., Majstorović, D. and Diermeyer, F. (2022) 'Open Source Software for Teleoperated Driving', in 2022 International Conference on Connected Vehicle and Expo (ICCVE), pp. 1–6. doi: 10.1109/ICCVE52871.2022.9742859. To be validated.

Specht, C., Świtalski, E. and Specht, M. (2017) 'Application of an autonomous/unmanned survey vessel (ASV/USV) in bathymetric measurements', Polish Maritime Research. To be validated.

Walker, M.E., Hedayati, H. and Szafir, D. (2019) 'Robot Teleoperation with Augmented Reality Virtual Surrogates', in 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 202–210. doi: 10.1109/HRI.2019.8673306. Validated 2026-06-01: DOI resolved via IEEE Xplore; title, authors (Walker, Hedayati, Szafir), year (2019), venue (HRI 2019), all confirmed.

Watanabe, K. and Takahashi, M. (2020) 'Head-synced Drone Control for Reducing Virtual Reality Sickness', Journal of Intelligent & Robotic Systems, 97(3–4), pp. 733–744. doi: 10.1007/s10846-019-01054-6. To be validated.

Open Questions

  • What does the planned user study find on the distance–delay correlation and on usability, reliability, and connectivity in real-world conditions?
  • Does 5G integration resolve the bandwidth-over-range limit that constrained the WiFi prototype to roughly 100 m?
  • How effective is the IMU/gimbal camera stabilization at reducing the combined motion and sea sickness during sustained teleoperation?
  • How does the AI-driven state machine concretize the transition functions between autonomous, semi-autonomous, and manual modes in practice?