Surgical training and pioneering medical professionals to become better surgeons is just one area where mixed reality and other immersive technologies are being used every day that we just don’t realise
Raimundo Sierra, co-founder of VirtaMed, breaks down the benefits of the available immersive technologies and explains how they’ve combined to make mixed reality a compelling means for training surgeons
The promise of virtual reality is clear: we can explore a virtual world from the safety of a headset and then come back to reality still possessing the experience and knowledge acquired in the virtual environment. This is the goal in medical education too: going from the reality of surgery to a safe practice environment of simulated surgery, and then bringing the learning back into surgery. Today, we have a spectrum of technologies, from virtual reality to augmented and mixed reality, all extending sensory inputs between the real world and the digital.
Mixed reality combines physical and virtual worlds to create the ultimate in realistic and interactive experiences. It is the optimal combination of the amazing advances in photorealistic virtual reality graphics and immersive soundscapes with other sensory inputs, most often with tactile objects. High fidelity experiences require these physical objects to be calibrated with the virtual world, and if the object bends or contracts then the sync between physical and virtual worlds needs to be perfect.
Enterprise organisations are embracing new digital tools not only for their scale and efficiency but also for their data-generating capabilities and potential synergies with artificial intelligence (AI) and machine learning. The unique mix of real and virtual elements is really making a difference to life around us in more ways than we imagined possible. One particular area is how mixed reality surgical simulators are being used to train future surgeons across the world.
Technology in training
It’s rare for someone to go through life without having an operational procedure. However, many surgeries are routine procedures that are performed by experienced surgeons regularly—such as the removal of the gallbladder or repair of torn ligaments. The last 100 years have instilled minimally invasive surgery as a preferred option for many procedures, and the next 100 years look to integrate robotics and augmented reality into the operating theatre.
Becoming a surgeon is a lengthy process, generally requiring four years of undergraduate school, four years of medical school, and three to 10 years of residency and fellowship training. Even, after this, in order to maintain their medical position and to keep up to date with the latest therapeutics, their medical education is a continued and enduring part of their career.
Today, we see a stark contrast between institutions that have been able to continue using training technology such as simulators and those that have experienced a 50% reduction in their time as primary operating surgeon during the pandemic. Indeed it is expected that the deficit of cadaver training will run past next year, and fortunately, there is evidence that the use of mixed reality simulators is enabling surgeons to train quicker and with more precision during procedures. Today’s simulators are so realistic that surgeons are truly able to undertake immersive and impactful training.
Mixed reality in a medical world
The consensus is clear: never the first time on a patient. There must be a pathway from the lecture theatre to the operating theatre. Simulators that combine visual virtual reality components with haptics that sensorise tools are giving a clear advantage to effective and transferable medical skills training.
Original simulators were clunky mechanical machines with large bulky displays that bore no real resemblance to what a surgeon would see in the operating theatre. For many decades they were boxes with vegetables or butcher’s offcuts inside. In the 1980s, the idea of realistic simulation training was there, but computing power limitations meant that simulators were at best a toy—more like playing PAC-MAN than a surgery. The real change came when research institutions began to combine computer vision with haptic interfaces, around the turn of the millennium.
VirtaMed was one such company, a spin-off of the Swiss Federal Institute of Technology and a combination of the results of 16 PhD theses that began in 2001 and founded the company in 2007. This research enabled true design modelling (face value aesthetics) and integrated advanced technologies that had previously been used outside of the medical environment—the emergence of mixed reality simulation.
Haptic feedback and sensorisation of instruments
Another technology that has ported into the entertainment/gaming world is haptic feedback—applying forces, vibrations, or motions to the user through their hands. In gaming, haptic feedback often acts as an interaction signal, such as a buzz to confirm a selection or a shake to indicate danger. These concepts have to be elevated and refined when used in mixed reality to simulate tissue feedback from a patient through sensorised surgical instruments.
Active haptics refers to the use of motors and actuators to stimulate a sensation based on computer 3D models and can be particularly useful when simulating structures that are physically not present. On the other hand, passive haptics are the use of replacement tissues, such as silicon models, that can be poked and prodded as in real life.
Orthopaedic surgeons recently found that it’s the passive haptic feedback that offers surgeons a real boost when it comes to their training, and we must factor in their interactions with particularly hard and rigid bone structures. In contrast, active haptics have been effectively used to simulate laparoscopic (abdominal) surgery, where tissue feedback from internal organs is more subtle. The true value of a simulation developer is to master both techniques and apply a smart combination of haptic feedback as they strive to simulate reality.
Taking mixed reality simulators into the future of medical training
Covid appears to be a breakpoint in medical education, where those embracing simulation have been able to continue training and those without simulators have gone without training. We have also seen a plethora of new tools being rapidly developed and brought to market during the pandemic, mainly focused on socially distant and independent learning. In order to stay ahead, we must build on this rapid innovation and integrate it into value chains.
The benefit of starting with mixed reality is that the hardest part is already done: there are virtual 3D models that can exactly reflect interactions in the real world. Therefore, these models can be adapted as multi-modal software platforms, transforming experiences into a purely digital online world or creating augmented reality overlays to physical objects. Interswappable physical and technical elements of mixed reality simulators mean that additional hardware or software elements can be added and upgraded when available.
Device developers, engineers and designers are monitoring and understanding how technology elements from other sectors can be applied to medical technology. The use of ‘realities’ and associated technologies is making the medical device market one of the largest growing sectors and a huge opportunity for XR technology developers.
These companies also have an advantage with their data-driven view of the world. On the technology side, they have been fed by data for years, and some have already started to integrate AI and machine learning technology to understand surgery at a data level. Surgeons are training themselves, and also training data models to understand their actions and reactions.
Surgical training and pioneering medical professionals to become better surgeons is just one area where ‘reality’ technologies are being used every day that we just don’t realise. Whether you’re developing virtual, augmented or mixed reality hardware or software, the possibilities within enterprise, education, entertainment and more are vast. It’s a multi-billion dollar industry that will only continue to grow over the next decade.
At VirtaMed, we’re excited to lead the integration of mixed reality simulators into surgical training, and see an endless array of possibilities for creating meaningful educational solutions.
Don’t forget to read the latest issue of VRWorldTech Magazine, which focuses on education.