VISUAL COMPUTING FOR MEDICINE VISUAL COMPUTING FOR MEDICINE THEORY, ALGORITHMS, AND APPLICATIONS SECOND EDITION BERNHARD PREIM CHARL BOTHA AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Morgan Kaufmann is an imprint of Elsevier Acquiring Editor: Meg Dunkerley Editorial Project Manager: Heather Scherer Project Manager: Priya Kumaraguruparan Designer: Mark Rogers Morgan Kaufmann is an imprint of Elsevier 225 Wyman Street, Waltham, MA, 02451, USA © 2014 Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods or professional practices, may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information or methods described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. Library of Congress Cataloging-in-Publication Data Application submitted British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. ISBN: 978-0-12-415873-3 For information on all MK publications visit our website at www.mkp.com Printed in the United States of America 14 15 16 17 10 9 8 7 6 5 4 3 2 1 Acknowledgments This book was only possible with substantial support from a number of people. First, we want to thank Meg Dunkerley, Heather Scherer, Laura Lewin, and Lauren Mattos from Elsevier for the kind and inten- sive cooperation starting from the draft of a book proposal to the final stages of writing. Timo Ropinski provided the chapter on advanced volume rendering, making his long-term experience in that area avail- able to the readers. The CAS chapter is based on the significant foundations laid by Thomas Kroes, and is considered to be joint work with him. Petra Schumann, Petra Specht, and Steffi Quade did proof-reading, improved images, and helped with the generation of indices. We are grateful to all members, past and present, of the (Medical) Visualization group of the Delft University of Technology, who have formed an effective and especially pleasant plat- form for stimulating research on a number of the topics discussed in this book. We are grateful to the whole visualization group in Magdeburg, in particular Alexandra Baer, Steven Birr, Rocco Gasteiger, Sylvia Glaßer, Kerstin Kellermann, Paul Klemm, Christoph Kubisch, Arno Krüger, Kai Lawonn, Jeanette and Tobias Mönch, Konrad Mühler, Steffen Oeltze, Zein Salah, and Christian Tietjen. Jana and Lars Dornheim as well as Ivo Rössling carried out most of the developmental work described in the chapter on ENT sur- gery. Simon Adler (Fraunhofer IFF Magdeburg) contributed to surgery simulation. A number of Master students did extraordinary work that is partially reflected in this book: Roland Pfisterer, Daniel Proksch, and Christoph Russ. The book is largely based on their research; they helped to focus and streamline the discussion of their research results. A number of people from other institutions helped to improve the book primar- ily by carefully commenting on individual chapters: Christian Rössl and Holger Theisel from the Visual Computing Group (Univ. of Magdeburg), Sebastian Schäfer and Klaus D. Toennies (Image Processing group, Univ. of Magdeburg), Raimund Dachselt (User Interface group, Univ. of Magdeburg), Oliver Speck and Daniel Stucht (Biomedical Magnetic Resonance group, Univ. of Magdeburg), Axel Böse and Georg Rose (Medical Technology group, Univ. of Magdeburg), Philipp Berg, Gabor Janiga and Dominique Thevenin (Fluid Simulation group, Univ. of Magdeburg), Oliver Großer (Radiology group, University of Magdeburg), Werner Korb (University of Applied Sciences, Leipzig), Tobias Isenberg (INRIA-Saclay) and Stefan Schlechtweg (Univ. of Applied Sciences, Koethen), Volker Diehl, Volker Dicken, Christian Hansen, Anja Hennemuth, Jan Klein, and Felix Ritter from Fraunhofer MEVIS Bremen, Ragnar Bade, Tobias Boskamp, and Olaf Konrad from MeVis Medical Solutions, Julian Ang and Alf Ritter from Brainlab, Roy van Pelt and Anna Vilanova (TU Eindhoven), Ralph Brecheisen (Arcus Solutions), Christian Dick (TU Munich), Heinz Handels (Univ. of Luebeck), Stefan Weber (Univ. of Bern), Claes Lindström, (University Linköping), Nigel W. John (Bangor University). Finally, we want to acknowledge our long-term medical collaborators that provided the motivation for the research described in this book: Andreas Böhm, Andreas Dietz, Stefan Müller, and Gero Strauß (University hospital), Christoph Arens, Oliver Beuing, Jörg Franke, Martin Skalej, Christian and Ulrich xvii xviii Acknowledgments Vorwerk (University hospital Magdeburg), Karl Oldhafer (Asklepios hospital Hamburg). We would also like to thank all our colleagues in The Netherlands, both medical and technical, from the Image Processing Division (LKEB), and the departments of Orthopedics, Radiology, Anatomy, and Surgery of the Leiden University Medical Center, and from the departments of Ophthalmology and of Neuroscience and Anatomy of the Erasmus Medical Center in Rotterdam, for the fruitful collaboration over the years. A special and tender thanks to Uta Preim for providing feedback on all medical issues discussed in this book and for her complementary research, in particular on perfusion imaging. Foreword to the Second Edition Visual Computing for Medicine is an excellent textbook for students, researchers, and practitioners in the field of medical visualization. It is an authoritative resource for medical experts and technical personnel as well. The book is the sequel to the highly successful first edition which immediately established itself as the reference work in this rather new and vibrant research topic in medical informatics. Dirk Bartz as one of the co-authors of the first edition unfortunately and untimely passed away in 2010. This left a sorely felt void in our community and prevented him from collaborating on the second edition. This second edition provides a substantially updated, restructured, and extended view on the current state of the field. In a recent talk Donald Knuth, the preeminent computer scientist, was asked about relevant open research directions in computer science. After some reflections, he said that medical visualization was one of the important topics in this respect. This was good to hear, though he quickly (and regrettably) added that according to his opinion not many problems have been solved yet. I take the liberty to slightly disagree and put this textbook forward as compelling and written evidence to the contrary. And what an evidence it is! On over one-thousand pages the authors survey the intensive and rapid developments in our area. The relatively short period between first and second edition and the considerable amount of added material in extent and volume are very clear indications of the fast-paced evolution of visualization in medicine. The book is concerned with diagnosis, treatment, and therapy planning with a focus on the currently most prevalent 2D and 3D imaging modalities. It thoroughly discusses the elaborate pipeline from data acquisition, analysis, and interpretation to advanced volume visualization and exploration techniques. Human computer interaction in the context of medical visualization has been covered in detail and encompasses significant topics like volume interaction, labeling, and measurement. Important applica- tion areas and advanced visualization techniques for blood vessels, virtual endoscopy, ENT surgery plan- ning, perfusion, and diffusion data are extensively dealt with. The book is very well structured, where the 22 chapters are grouped into five focal themes. The authors primarily organize the book according to techniques as most of these are applicable to a variety of medical tasks. Some of the material has been combined into completely new chapters like the one on projection-based medical visualization techniques. Hints at further readings at the end of each chap- ter point the interested reader to additional useful material not discussed within the chapter. Various advanced topics, which are of interest to the software engineer but are maybe too detailed for the gen- eral audience, are included in clearly marked break-out sections. The substantial reference list is another eloquent testimony of the breadth and depth of the topic. The authors are highly recognized experts in the field of medical visualization. They have achieved the impressive feat of comprehensively covering a dynamic and rapidly emerging subject. The book provides informative, broad, and didactically well-organized information for specialists from diverse areas of expertise. The book will be the standard guide to medical visualization for years to come. Dr. Eduard Gröller Vienna University of Technology xix Preface to the Second Edition This second edition of “Visualization in Medicine” reflects the dynamic development of medical imag- ing, algorithmic processing and applications in medical research and clinical use after 2006. After the tragic passing of Dirk Bartz in March 2010, Charl Botha stepped in to prepare this new edition. In addition to careful rewriting of all chapters, we added a number of completely new chapters and reor- ganized and updated others significantly. Advances in imaging technology, e.g., hybrid devices, ultra high field MRI, intraoperative imaging, and the trend towards interventional procedures, are reflected in various parts of the book. Since more and more advanced applications, e.g., in processing the complex multi-modal data of cardiac or neuroradiological MRI, have entered the stage of routine clinical use, human-computer interaction is becoming increasingly important. A comprehensive chapter was added to introduce HCI concepts with applications in medicine, incorporating recent interaction styles and technology. Also the chapter related to clinical practice was strongly extended by discussing also nuclear medicine, radiation treatment and medical team meetings in addition to the classical diagnostic settings. Another essential trend is the combination of biomedical simulations with advanced visual explora- tion. As a consequence, we prepared a chapter that introduces basic techniques, such as the generation of simulation grids from medical imaging data and flow visualization, to explore the results. We study a number of specific applications, such as the simulation of blood flow to better predict the success of treatment options. While the first edition of this book was focused on visual exploration, we have added discussions of data analysis techniques and their integration in what is widely called “visual analytics”. This relates, e.g., to cluster analysis and dimension reduction. We discuss these techniques in relation to high- dimensional data, such as perfusion data and diffusion tensor imaging data. They are, however, also relevant for volume classification, the basic process of assigning transfer functions to medical volume data. Computer-assisted surgery (CAS), one of the most essential applications for medical visualization technology, has matured in the last decade. We use experiences gained in the design and evaluation of such systems to prepare a general introductory chapter on CAS, followed by chapters treating selected application areas, such as orthopedics. Intraoperative imaging and intraoperative guidance have grown in importance in the last years. The chapter devoted to this topic was significantly extended, e.g., with techniques developed for soft-tissue surgery. Even the chapters discussing basic medical visualization techniques, such as surface and direct volume rendering, deserved a careful revision. Among others, GPU-based techniques play a more prominent role now. GPU-based rendering enables a huge step in improving image quality without compromising performance. We discuss how these improvements are employed, e.g., in virtual endoscopy—another chapter that could be improved by taking advantage of many new and refined techniques. xxi xxii Preface to the Second edition The increasing size and complexity of medical image data motivated the development of visualiza- tion techniques that radically differ from the classical surface and volume rendering techniques. To convey the complex information of medical flow data along with the relevant anatomy, for example, benefits from illustrative techniques that render the anatomy sparsely. Thus, illustrative rendering plays a more prominent role in this second edition discussing the extraction of various features from medical volume data and related meshes as a basis for rendering. A second radically new class of visualization techniques are map-based techniques. While some isolated techniques, such as stretched curved planar reformations of vascular structures, have been introduced more than a decade ago, we can now discuss this topic in a more general fashion in a separate chapter. DTI was rather new when the first edition was prepared. It is meanwhile a mature technique that is discussed in a wider scope as one out of several techniques to understand brain connectivity. Medical education in anatomy, interventional radiology and surgery remains an important use case of visual computing. One comprehensive chapter is dedicated to such applications with a focus on recent trends, such as web-based training platforms, and (automatic) skills assessment. Companion Website Visit this book’s companion website for this work: http://medvisbook.com/ Author Biography PROF. DR.-ING. BERNHARD PREIM was born in 1969 in Magdeburg, Germany. He received the diploma in computer sci- ence in 1994 (minor in mathematics) and a Ph.D. in 1998 from the Otto-von-Guericke University of Magdeburg (Ph.D. thesis “Interactive Illustrations and Animations for the Exploration of Spatial Relations”). In 1999 he finished work on a German textbook on Human Computer Interaction which appeared at Springer. He then moved to Bremen where he joined the staff of MeVis (Center for Medical Diagnosis and Visualization Systems, Bremen). In close collaboration with radiologists and surgeons he directed the work on “computer-aided planning in liver surgery” focusing on virtual resection, automatic resection proposals, visualization of vascular structures, and the integration of measurements in 3D visualizations. This work was largely influenced by Prof. Heinz- Otto Peitgen, the founder and director of MEVIS. In June 2002 Bernhard Preim received the post-doctoral lecture qualification for computer science from the University of Bremen. Since Mars 2003 he is full professor for “Visualization” at the computer science department at the Otto-von-Guericke-University of Magdeburg, heading a research group which is focussed on medical visualiza- tion and applications in surgical education and surgery planning. The focus of this research is illustrative medical visualization, visual exploration of blood flow, virtual endoscopy, and in particular surgery in the ear, nose, throat region. These developments are summarized in a textbook Visualization in Medicine (Co-author Dirk Bartz). His continuous interest in HCI lead to another textbook “Interaktive Systeme” (Co-author: R. Dachselt) (Springer, 2010). His regular teaching activities include “Medical Visualization”, “Computer-Assisted Diagnosis and Treatment” as well as the introductory courses on “Visualization” and “Interactive Systems”. Bernhard Preim founded the working group Medical Visualization in the German Society for Computer Science in 2003 and acted as speaker until 2012. He is also a long-term member of CURAC, the German society for computer-assisted surgery, where he became board member in 2007, and vicepresident in 2009. He was Co-Chair and Co-Organizer of the first and second Eurographics Workshop on Visual Computing in Biology and Medicine (VCBM, together with Charl Botha) and is now member of the steer- ing committee of that workshop. He is the chair of the scientific advisory board of ICCAS (International Competence Center on Computer-Assisted Surgery, since 2010), member of the advisory boards of Fraunhofer Heinrich-Hertz-Institute, Berlin and the Institute for Innovative Surgical Training Technologies (ISTT), Leipzig. He is also regularly a Visiting Professor at the University of Bremen where he closely collaborates with Fraunhofer MEVIS. At the University of Magdeburg, Bernhard Preim is member of the Board (since 2008). Bernhard Preim is married with the radiologist Uta Preim (Medical Doctor), born Hahn and has two children. DR. CHARL P. BOTHA graduated from the University of Stellenbosch, South Africa, in 1997 with a degree in electronics engi- neering, followed by an M.Sc. in digital signal processing, in 1999, and finally a Ph.D. in medical visualization from the Delft University of Technology (TU Delft) in the Netherlands, under the supervision of Frits Post, one of the pioneers of scientific visuali- zation in Europe. After completing his Ph.D., he was appointed (2006) and soon after tenured (2007) as an assistant professor of Visualization at the TU Delft, where he started and headed the medical visualization lab. He also had an appointment at LKEB, the medical image processing section of the Department of Radiology at the Leiden University Medical Center (LUMC), in order to cultivate and expand the fruitful research collaboration between the technical university and the academic hospital. His research focused on surgical planning and guidance, and visual analysis for medical research. He has published on, among other topics, anatomical modeling, virtual colonoscopy, shoulder replacement, and diffusion tensor imaging. Together with Bernhard Preim he initiated the Eurographics Workshop series on Visual Computing for Biology and Medicine, acted as co-chair in 2008 and 2010, and served as editor together with Prof. Preim of the Computers and Graphics special issue on VCBM. Prior to his Ph.D. he worked in industry designing embedded image processing systems and algorithms for two different companies. Shortly after the Ph.D., he co-founded Treparel Information Solutions, a company specializing in data mining, and he acts as science advisor to Clinical Graphics, a spin-off company founded by an ex-Ph.D. student to commercialize surgical planning research results. He recently also decided to make the move back into industry full-time, where he has started a company that focuses on bringing computer science, imaging, and visualization research into real-world practice. He remains actively involved with the medical visualization community through the MedVis.org website and its related resources. Charl is married to Stella Botha-Scheepers, MD, Ph.D., a rheumatologist and internist, with whom he has two children. xxiii Chapter 01 Introduction Visualizationreferstotheuseofcomputergraphicstechniquestocreateinteractivevisualrepresentationsof data,with the goal of amplifying human cognition.When visualization is applied to medical data,it is calledvisualizationinmedicine,ormedicalvisualizationforshort. Most medical data has an inherent spatial embedding. For this reason,medical visualization is seen asaspecialareaofscientificvisualization.Thestartofscientificvisualizationasaresearchfieldisconsidered by many to be the publication of the 1987 report of the NSF onVisualization in Scientific Computing [McCormick et al., 1987]. However, literature reveals instances of medical visualization, following the definitionwehavesetinthefirstparagraph,asfarbackasthe1960s. Inanearlyradiotherapyplanningexample,patientcontourswereacquiredfromlinedrawingswith a mechanical digitizer, and then shown on an oscilloscope display combined with calculated isodose distributions,using a computer especially designed for this purpose [Cox et al.,1966,Holmes,1970]. Already then this system was put into clinical use. Sunguroff and Greenberg [1978] demonstrated the extraction and visualization of smooth 3D surfaces from CT data. By the end of the 1970s, McShan et al. [1979] had demonstrated the use of 3D graphics for radiotherapy planning. In the early 1980s, 3Dvisualizationwasbeingusedclinicallyforthecomputer-basedpreoperativeplanningofcraniofacial surgery[Vannieretal.,1983b]. Ontheonehand,thelongtraditionofscientiststhatillustratetheirworkbycarefullycraftedgraphics laid the foundation for both scientific and medical visualization.Anatomical illustration, starting with daVinci’swork,isaprominentexample.Ontheotherhand,medicalvisualizationisbasedoncomputer graphics that provide algorithms for the efficient rendering of data,with additional influences coming fromtheworldofimageprocessingandmedicalimageanalysis. 1.1 VISUALIZATION IN MEDICINE AS A SPECIALTY OF SCIENTIFIC VISUALIZATION Scientificvisualizationdealsprimarilywiththevisualization,exploration,andanalysisofdatasetsarising frommeasurementsorsimulationofrealworldphenomena.Theinvestigationofairflowaroundplanes andcarsisawell-knownexample.Theunderlyingdatasetsofscientificvisualizationsareoftenverylarge, whichmakesitnecessarytoconsidertheefficiencyandhencethetimeandspacecomplexityofalgorithms. Importantgoalsandresearchscenariosofscientificvisualizationare: (cid:129) toexploredata(undirectedsearchwithoutaspecifichypothesis), (cid:129) totestahypothesisbasedonmeasurementsorsimulationsandtheirvisualization,and (cid:129) thepresentationofresults. There are many relevant examples in medical visualization that address these general visualization goals.Whether or not a patient is suffering from a certain disease is a hypothesis to be tested through clinical investigations and medical imaging. If a physician cannot sufficiently assess a disease based on the symptoms described by the patient and by clinical examinations,radiological image data might be VisualComputingforMedicine,SecondEdition.http://dx.doi.org/10.1016/B978-0-12-415873-3.00001-8 1 ©2014ElsevierInc.Allrightsreserved.