Communications of the ACM
Medical image modeling tools and applications
Hepatic Surgery Simulation
The goal of surgical simulation is to provide highly realistic training to increase the diffusion of innovative and less-invasive procedures while decreasing the surgeon's learning curve. Consider, for instance, the development and refinement of laparoscopic surgery. In this type of surgery, abdominal operations such as hepatic (liver) resections are accomplished through small incisions. The abdomen of the patient is inflated with gas to create open space inside and a video camera is inserted into the abdomen through one of the small incisions. The video image is magnified and transmitted to a high-resolution monitor, allowing the surgeon to see the abdominal anatomy with great clarity. The surgery is performed using special instruments introduced through the other incisions. The advantages of surgical simulation for the patient include less pain and less strain for the body, faster recovery, and reduced overall illness time. The surgeon, however, must acquire specific skills and develop advanced hand-eye coordination before attempting to perform such a procedure in a real-world setting.
Prior to digital surgical simulation, physicians could train themselves on dedicated mechanical systems known as endotrainers or on animals. The endotrainers allow the surgeon to use the same elongated instruments as the ones used in minimally invasive surgery, inserted through the openings of a transparent box. The surgeon can then interact with plastic objects describing various organs of interest. This is a very crude and static approximation of reality, which does not account for the dynamic nature of the tissues, the presence of vessels, the indirect vision of the surgery field, the potential bleeding of organs, and other such factors. Surgical training using animals is much more realistic, but raises a number of ethical concerns that severely limit its current and future diffusion. Moreover, using animals for surgical training remains a rather expensive approach to teaching and learning.
In comparison, digital simulation can offer a number of immeasurable advantages: the ability to learn from mistakes and failure, to practice alone or with a mentor, and much greater versatility, allowing the simulation of various scenarios ranging from standard pathologies to extremely rare cases. But surgical simulators have more to offer than serving as a support for training surgical residents to perform known minimally invasive procedures. Surgical simulators can potentially be used to rehearse a complex procedure with the patient-specific data and thus become a natural extension of preoperative planning. Simulators could also be utilized to certify the skills of surgeons in a quantitative and objective way.
Different Generations of Simulators. Surgery simulators can be classified into three categories, as shown in Figure 1a. First-generation simulators describe only the anatomy, in particular the geometry of the structures involved in a surgical intervention. In these simulators, the user can essentially navigate within a virtual representation of the patient, with a limited set of possible interactions. Several simulators of this type have been developed, in particular for virtual endoscopy (for colonoscopy or bronchoscopy procedures). Some of these simulations are commercially available, already linked to the processing console of some imaging systems, such as magnetic resonance imaging (MRI) and computed tomography (CT). These simulators are mainly used as complementary diagnosis tools, and also to assist with surgery planning, but are generally not well adapted to the simulation of surgical gestures.
Second-generation simulators not only include the geometric modeling of the body anatomy, but also the modeling of the physical properties of the living tissues. The introduction of biomechanical properties is essential to allow realistic interactions between surgical instruments and soft tissues, including deformations and cutting. No second-generation simulators are currently commercially available, although a number of prototypes have been developed and demonstrated by several research groups: examples include the simulation of gall bladder resection [6] and the simulation of gynecological surgery [12].
Third-generation simulators combine anatomical, physical, and physiological modeling, that is, the modeling of the functions of some organic systems such as the cardiovascular, respiratory, or digestive systems. There is an additional degree of complexity due to the coupled nature of physiological and physical properties. Some interesting models exist for the cardiopulmonary system [8], the vascular system [11], and the cardiac system [1].
Issues in Surgical Simulation
The components of a second-generation surgical simulator are summarized in Figure 1b. These components are related to the geometric and physical modeling of human tissues and to the design of a multimodal user interface.
The geometrical models of anatomical structures are, in most cases, extracted from 3D medical images (usually CT or MRI images) with some dedicated image-processing tools. We can make a distinction between building "standard" anatomical models or patient-specific models. The former is relevant in the context of teaching surgical gestures to medical residents whereas the latter is required when the simulation is used to rehearse a particular surgical procedure.
The physical modeling of a simulator includes the modeling of contacts between virtual instruments and soft tissues as well as the biomechanical deformation of soft tissues. When a collision between an anatomical structure and a surgical instrument is detected, the boundary constraints on soft tissue models are updated and, depending on the nature of the instrument, a piece of tissue is removed. The soft tissue is then deformed according to a given biomechanical behavior. If several mathematical models have been proposed in the literature [5], the choice of a biomechanical model must take into account two contradictory elements: the simulation realism and the computation cost for implementing this model.
The user interface of the surgical simulator is of prime importance since it is essential for effective training that surgeons act as if they were operating on an actual patient. The two main senses that should be reproduced are vision and touch. To produce convincing visual feedback, it is necessary to correctly model the textures of tissues, the specific lighting effects of an endoscope, and different special effects such as the occurrence of blood. Haptic, or touch-sensitive, feedback is also an important element for the tridimensional understanding of a surgical procedure.
A Simulator of Hepatic Surgery
Here, we describe some of the main phases during our development of a hepatic surgery simulator.1 Our primary motivators were the importance of liver pathologies and the inherent complexity of hepatic surgery. We also realized that modeling the interaction with the deformable organs of the abdomen (in particular of the liver itself) was a very challenging research problem we wanted to study thoroughly.
Two complementary tools were developed as part of this project. The first concerns the planning of the hepatic surgery; the second is related to the simulation of the surgical gestures and movements. The planning of hepatic surgery mainly involves the determination of the so-called Couinaud (or functional) segments, which correspond to main vascular territories of the liver. Once surgery planning is completed, surgery simulation attempts to reproduce the surgical gestures with enough realism to allow for efficient training.
Geometric Modeling. We designed a combination of image-processing techniques to extract the principal hepatic structures of interest from the clinical CT images taken before surgery (preoperative images): the hepatic parenchyma, the main vessels, the Couinaud segments, and the potential lesions. This extraction is performed in two stages: first, the hepatic parenchyma is delineated, then the vessels, and finally the lesions are detected. To extract the hepatic parenchyma, one uses a geometric model based on a simplex mesh that is deformed to fit the corresponding 3D edges of the CT image. The extraction of the vessels is based on the exploitation of mathematical morphology and digital topology techniques. Finally, the extraction of the eight Couinaud segments is based on the computation of the regions of influence of the first branches of the portal vein (similar to a generalized Voronoi diagram). Figure 2 shows examples of these various stages. This led to the first computerized hepatic surgery planning system, used at the Hospital of Strasbourg, which proved to have a significant impact on surgical training and practice at that facility.
Physical Modeling. Modeling the deformable nature of biomaterials is extremely computationally expensive, which makes this task the most critical one to reach real-time performances. This is the reason why, since 1995, we have been working on the definition of new algorithms allowing a realistic deformation of soft tissues, while preserving reasonable computing costs.
For the sake of realism, we chose to rely on two widely accepted frameworks: the theory of continuum mechanics (more precisely, the elasticity theory) and the finite element method. The elasticity theory has been the basis for estimating the deformation of deformable structures for more than a century, especially in civil engineering. In addition, the method of reference for transforming a problem of mechanics into a number-crunching problem has been the finite element method. But those approaches were known to require hours or even days of computation to solve a given problem. Our contribution has been to propose new algorithms for computing soft tissue deformation in a few milliseconds.
We initially decided to simplify the behavior of biological soft tissues by assuming linear elastic materials. Indeed, for small deformations (less than 10% of the original size), linear elasticity is considered a valid approximation for most tissues. Next, we decomposed the domain of computationin our case, a liverinto a set of tetrahedra, with the goal that each tetrahedron should be as regular as possible. The number and the shape of tetrahedra strongly influence both the accuracy of the computation and the time required to solve the problem.
To build a linear elastic model of the liver, we need to define two additional items: the position of the liver at rest and its material properties described by its Young Modulus and Poisson ratio. We assume the rest position corresponds to the position of the liver in the preoperative CT scan image. Estimating reliable mechanical properties of living tissues, even under the simple assumption of linear elasticity, is a very difficult task because of their different behavior between in-vivo (within the body) and ex-vivo (outside the body) experimental conditions [10]. Currently, we are relying on data obtained in-vivo but there is a large amount of variability in the parameter estimation, possibly due to the acquisition method: for instance the variation of the degree of perfusion of an organ between in-vivo and ex-vitro conditions dramatically modifies its mechanical properties.
We proposed two complementary algorithms for optimizing the computation of the deformations of a linear elastic material based on the finite element method. The first approach exploits the linearity of the model during a precomputation stage. Indeed, a direct consequence of the linear model is that the deformation under the application of several forces is obtained by the addition of the deformations produced by each force independently. Therefore, it is possible to compute in a preliminary stage a set of deformations obtained from elementary forces applied to each node and to store this information. During the simulation, it then suffices to compute a linear combination of these stored deformations to retrieve the actual deformations of the model. This is a very efficient method that typically requires a few milliseconds to deform a liver model similar to the one shown in Figure 3a; the method is precisely described in [3].
The main limitation of the first approach is that the precomputation strategy assumes the topology of the model mesh remains the same during the simulation: this obviously prevents the simulation of cutting or suturing gestures. Moreover, this method is also a quasi-static method, as it computes directly the equilibrium solution produced by the application of the instruments against the liver surface. It does not take into account the viscoelastic nature of the soft tissues. Because of these limitations, we proposed a second approach introducing new deformable models we call tensor-mass models, because they share a number of similarities with the spring-mass models commonly used in the computer graphics community. The main (and essential) advantage of these tensor-mass models stems from the fact they really implement a discrete approximation of a volumetric and continuous mechanical behavior, which generally is not the case with classical spring-mass models.
This second approach is more costly in terms of computing time as it requires integration of the differential equation of the motion at each iteration. Therefore, depending on the performances of the computer and the rigidity of the material, it is necessary to limit the number of nodes of the mesh in order to satisfy the constraints of a real-time implementation (typically a few thousand nodes on a modern PC). This is why we introduced the hybrid models, which combine the precomputed and the tensor-mass models. By restricting the regions where cutting is allowed, it is therefore possible to include in the simulator deformable models of a large size. In the hepatic surgery context, we decomposed our model into eight regions, corresponding to the detected Couinaud segments, and restricted the cutting regions to a number of 3D bands at the interfaces between these segments. These bands are described by tensor-mass models, while the rest of the mesh is described by precomputed models. Details on tensor-mass and hybrid models are available in [4].
Visual Rendering. In a collaborative project called AISIM, specific tools were developed by graphics experts to produce a realistic rendering of the liver during surgery simulation. In particular, the projection of the reflection of the circular light source used in laparoscopic surgery enhances the realism, and improves the perception of the 3D deformations of the liver under the action of the instruments. An original method of texture mapping was introduced to minimize texture distortions all around the model [9]. In addition, the texture reproduces the marks created by the application of certain instruments like an electrical scalpel, further increasing the realism of the simulation (see Figure 3b).
The user interface of the surgical simulator is of prime importance since it is essential for effective training that surgeons act as if they were operating on an actual patient.
Conclusion
The implementation of a surgical simulator is very challenging because it requires the integration of different technologies, including advanced user interfaces, computational structural mechanics, robotics, and computer graphics. In particular, haptic and visual feedback require a high update rate (more than 1000Hz for the haptic display of hard materials), which is not compatible with the heavy computational demands of soft tissue deformations. The current architecture of our simulator platform relies on a multiprocessor graphics workstation, where one processor is dedicated to the extrapolation of forces sent to force-feedback devices.
The simulation of surgical gestures raises many important scientific and technological issues. One major issue concerns the degree of knowledge of the physical behavior of human tissues. Currently, this knowledge is mainly qualitative and it is of paramount importance for many biomedical applications that more biomechanical studies are undertaken in the near future to create representative and accurate models of the human body.
References
1. Ayache, N. et al. Towards model-based estimation of the cardiac electromechanical activity from ECG signals and ultrasound images. In Functional Imaging and Modeling of the Heart (FIMH'01), Helsinki, 2001.
2. Cotin, S., Delingette, H., and Ayache, N. A hybrid elastic model allowing real-time cutting, deformations and force-feedback for surgery training and simulation. The Visual Computer 16, 8 (2000), 437452.
3. Cotin, S., Delingette, H., and Ayache, N. Real-time elastic deformations of soft tissues for surgery simulation. IEEE Transactions On Visualization and Computer Graphics 5, 1 (Jan.Mar. 1999), 6273.
4. Delingette, H. and Ayache, N. Soft tissue modeling for surgery simulation. In N. Ayache, Ed. Computational Models for the Human Body: Handbook of Numerical Analysis. Elsevier, 2004.
5. Fung, Y.C. BiomechanicsMechanical Properties of Living Tissues. Springer-Verlag, 2d ed., 1993.
6. Kuhnapfel, U., Akmak, H., and Maa, H. Endoscopic surgery training using virtual reality and deformable tissue simulation. In Computers and Graphics 24 (2000), 671682.
7. Marescaux, J. et al. Virtual reality applied to hepatic surgery simulation: The next revolution. Annals of Surgery 228, 5 (Nov. 1998), 627634.
8. Metaxas, D., Kaye, J., and Primiano, F. A 3D virtual environment for modeling mechanical cardiopulmonary interactions. Medical Image Analysis (Media) 3, 5 (1997), 126.
9. Neyret, F. and Cani, M.P. Pattern-based texturing revisited. In SIGGRAPH 99 Conference Proceedings. Addison-Wesley, August 1999.
10. Picinbono, G., Delingette, H., and Ayache, N. Non-linear anisotropic elasticity for real-time surgery simulation. Graphical Models 65, 5 (Sept. 2003), 305321.
11. Quarteroni, A., Tuveri, M., and Veneziani, A. Computational vascular fluid dynamics: Problems, models, and methods. Computing and Visualization in Science 2 (2000), 163197.
12. Szekely, G., Baijka, M., and Brechbuhler, C. Virtual reality based simulation for endoscopic gynecology. In Proceedings of Medicine Meets Virtual Reality (MMVR'99), (San Francisco, 1999), 351357.
Footnotes
1Our hepatic surgery simulator was developed by our research team Epidaure during the past seven years at INRIA, the French National Institute for Research in Computer Science and Control, in collaboration with IRCAD (Institut de Recherche Contre le Cancer de l'Appareil Digestif) in Strasbourg, France. Additional details are available in [2, 3, 7].
Figures
Figure 1a. The different generations of surgical simulators.
Figure 1b. The different technological components of a second-generation simulator.
Figure 2. Extraction of the hepatic parenchyma (parts a and b), the vascular trees and hepatic lesions (parts c and d) from a CT scan image.
Figure 3a. Liver deformation using a linear combination of precomputed elementary deformations.
Figure 3b. Sequence of a simulated liver resection that includes the clipping and cutting of the portal vein.
©2005 ACM 0001-0782/05/0200 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2005 ACM, Inc.
No entries found