Technique for Visualizing Interactions with a Technical Device in an Xr Scene

Simple SummaryContent extracted from patent full text and abstract with AI.

This invention provides a computer-implemented method and system for visualizing human interactions with a technical device (such as a medical device) in an extended reality (XR) environment. The system combines virtual scenes (e.g., of a device and surrounding environment) with real-time optical sensor data (e.g., cameras tracking a user's actions in a room), fusing them into a unified dataset. This approach allows for highly realistic simulation, visualization, training, and analysis of how humans interact with complex technical devices before any physical hardware exists.

Use CasesContent extracted from patent full text and abstract with AI.

• Product development and usability engineering for technical devices, especially medical equipment, by enabling early-stage virtual prototyping and interactive testing with users before physical prototypes exist.
• Operator or clinician training in safe and realistic virtual environments using XR headsets, reducing the need for physical devices or patient risk.
• Real-time workflow analysis and process optimization by capturing and analyzing detailed user interaction data.
• Synthetic data generation for training machine learning models in applications such as scene understanding, pose recognition, and workflow automation, especially where collecting real-world data is difficult.
• Remote or multi-user collaboration in XR environments, enabling team-based design reviews, virtual usability labs, or distributed training scenarios across different locations.
• Development of autonomous or AI-enabled technical devices that need to interpret or react to human interactions (e.g., medical robots).

BenefitsContent extracted from patent full text and abstract with AI.

• Reduces time and cost in product development by enabling rapid iteration based on realistic user feedback within virtual environments.
• Enhances safety and user competence by allowing operators to practice in realistic XR settings before using real devices.
• Improves product quality by enabling early detection and correction of design flaws or workflow bottlenecks.
• Generates rich, realistic synthetic datasets for machine learning, reducing dependency on expensive or hard-to-acquire real-world data.
• Supports scalable, flexible training and collaboration, overcoming geographical or resource limitations.
• Accelerates deployment and improves user acceptance of new device concepts through immersive, interactive experiences.
• Facilitates workflow optimization and compliance, particularly in healthcare, by allowing detailed process analysis before real-world rollout.

Technical Classifications (CPCs)

Inventors & Applicants

Patent Abstract

In one aspect, the invention relates to a computer-implemented method for visualizing interactions in an extended reality, XR, scene. The method comprises the steps of receiving a first dataset representing an XR scene comprising a technical device; displaying the XR scene on an XR headset or a head-mounted display, HMD; providing a room (302) for a user (306) wearing the XR headset or HMD for interacting with the XR scene, wherein the room (302) comprises a set of optical sensors comprising at least one optical sensor at a fixed location relative to the room (302); detecting optical sensor data of the user (306) as a second dataset while the user (306) is interacting in the room (302) with the XR scene; and fusing the first dataset and the second dataset for generating a third dataset.