Multi-modal vision techniques for image-guided surgery

During minimally-invasive surgery, endoscopes and other surgical tools enter the body of the patient through small openings, allowing the operations to be performed with less post-operative pain and faster recovery for the patient, as well as less wound complications. Although great advantages are achieved from the patient’s point-of-view, several difficulties have still to be handled by the surgeons. First, the limited field of view of the endoscope makes target-area localizations and related identifications complex. Second, surgeons should still match mentally the medical images for the surgical planning with the current patient anatomy. These difficulties can be addressed by using computer vision technologies for guiding surgical procedures. More specifically, this thesis aims at the following three points, the first two for neurosurgery, and the third for perfusion assessment during surgery. The challenges for this dissertation are partitioned in two primary aspects. The first point aims at the fusion of medical images on the endoscopic view, in order to develop an augmented reality system. The second point is based on hyperspectral imaging (HSI), which is compared with diffuse reflectance spectroscopy (DRS), to improve brain tissue classification for neurosurgery. The third point exploits the PPG imaging (iPPG) technique and its potential use is evaluated for peripheral arterial disease (PAD) and organ perfusion assessment during surgery. To address the first point, a new neurosurgical application is implemented on the already existing Philips Augmented Reality (AR) surgical navigation system, designed for spinal surgery. This navigation system incorporates an optical tracking system (OTS) with four video cameras embedded in the flat detector of the motorized C-arm. A hand-eye camera calibration algorithm for the fusion of medical images on the endoscopic view is implemented and integrated into the AR system. This technology is validated for endo