Interactive mixed reality widgets for precise dexterity of tool manipulation to enhance surgical procedures in Industry 4.0 realm

Complex medical procedures, particularly those involving surgery, demand exceptional precision in spatial 6DOF (Six Degrees of Freedom) tool manipulation, where even minor errors can result in irreversible damage or undesirable outcomes. In this context, precise tool alignment is crucial, whether in minimally invasive surgeries or delicate procedures such as dental implantology. Mixed Reality (MR) technology has emerged as a promising tool for addressing these challenges by providing surgeons with real-time, spatially accurate guidance through visual assets known as widgets. These MR widgets have the potential to support surgeons by superimposing helpful information directly onto the physical environment, enabling them to perform complex tasks with higher precision. However, despite the advancements in MR technology, current approaches remain predominantly static or quasi-static, leading to persistent errors due to a lack of adaptive and dynamic solutions. Additionally, the absence of standardized guidelines for widget design and the underemphasis of user interface (UI) considerations in MR systems exacerbate usability issues, leading to increased cognitive and physical task loads for surgeons and ultimately detracting from performance and outcomes. This thesis addresses these challenges through a systematic investigation that employs experimental methodologies, including user studies with surgeons and domain experts, to evaluate and optimize innovative MR-based solutions. A comprehensive literature review reveals significant gaps in existing MR systems, such as inadequate widget design, suboptimal usability, high cognitive task demands, and limited adaptability in real-time surgical contexts, particularly in dentistry. A key contribution of this work is advancing the understanding of user-centered UI design for precision tool-to-target guidance systems. This research highlights the importance of addressing persistent challenges such as visual clutter, occlusion, and inclusivity, ensuring that widgets cater to diverse user needs, including those with visual impairments or varying cognitive capabilities. Building on these insights, this thesis introduces interactive widgets that integrate principles of cognitive perception, particularly those derived from Gestalt theory. By applying Gestalt principles such as proximity, continuity, and figure-ground organization, the research focuses on optimizing visual design and incorporating real-time error feedback mechanisms that respond to user actions. The result is a set of interactive widgets that significantly enhance the positional and angular precision of the tools while managing cognitive load and task completion time effectively. The findings demonstrate that the proposed widgets outperform traditional, static designs in terms of precision, efficiency, and usability. These widgets improve tool manipulation accuracy, streamline cognitive processes involved in high-stakes procedures, reduce task completion times, and enhance user preference. Moreover, these designs’ modular and adaptable nature extends beyond medical applications, offering valuable solutions for industries requiring high precision and safety, such as manufacturing, maintenance, and assembly. Furthermore, this research presents a flexible, open-source evaluation framework for MR widget design, promoting standardized testing methodologies and fostering greater collaboration within the scientific community. This framework facilitates consistent development and assessment of MR systems, ensuring reliability and cross-domain applicability. Looking to the future, this research explores several directions for enhancing MR widget design, including integrating haptic and auditory feedback systems to increase interaction fidelity, developing adaptive and personalized user interfaces tailored to individual user needs, and establishing standardized design guidelines to encourage innovation and consistency across industries. The aim is to pave the way for safer, more efficient, and precise outcomes in MRassisted procedures and systems, providing a foundation for continued advancements in MR-based precision tools.