Search

Manually segmenting masses from native mammograms is a very time-consuming and error-prone task. Therefore, an integrated computer-aided diagnosis (CAD) system is required to assist radiologists for automatic and precise breast mass delineation. In this work, we present a two-stage multi-scale pipeline that provides accurate mass delineations from high-resolution full mammograms. First, we propose an extended deep detector integrating a multi-scale fusion strategy for automated mass localization. Second, a convolutional encoder-decoder network using nested and dense skip connections is used to fine-delineate candidate masses. Experiments on public DDSM-CBIS and INbreast datasets reveals strong robustness against the diversity of size, shape and appearance of masses, with an average Dice of 80.44% on INbreast. This shows promising accuracy as an automated full-image mass segmentation system, towards better interaction-free CAD.

Supervised deep learning on medical imaging requires massive manual annotations, which are expertise-needed and time-consuming to perform. Active learning aims at reducing annotation efforts by adaptively selecting the most informative samples for labeling. We propose in this paper a novel deep active learning approach for dual-view mammogram analysis, especially for breast mass segmentation and detection, where the necessity of labeling is estimated by exploiting the consistency of predictions arising from craniocaudal (CC) and mediolateral-oblique (MLO) views. Intuitively, if mass segmentation or detection is robustly performed, prediction results achieved on CC and MLO views should be consistent. Exploiting the inter-view consistency is hence a good way to guide the sampling mechanism which iteratively selects the next image pairs to be labeled by an oracle. Experiments on public DDSM-CBIS and INbreast datasets demonstrate that comparable performance with respect to fully-supervised models can be reached using only 6.83% (9.56%) of labeled data for segmentation (detection). This suggests that combining dual-view mammogram analysis and active learning can strongly contribute to the development of computer-aided diagnosis systems.

Mammography is the primary imaging modality used for early detection and diagnosis of breast cancer. X-ray mammogram analysis mainly refers to the localization of suspicious regions of interest followed by segmentation, towards further lesion classification into benign versus malignant. Among diverse types of breast abnormalities, masses are the most important clinical findings of breast carcinomas. However, manually segmenting breast masses from native mammograms is time-consuming and error-prone. Therefore, an integrated computer-aided diagnosis system is required to assist clinicians for automatic and precise breast mass delineation. In this work, we present a two-stage multi-scale pipeline that provides accurate mass contours from high-resolution full mammograms. First, we propose an extended deep detector integrating a multi-scale fusion strategy for automated mass localization. Second, a convolutional encoder-decoder network using nested and dense skip connections is employed to fine-delineate candidate masses. Unlike most previous studies based on segmentation from regions, our framework handles mass segmentation from native full mammograms without any user intervention. Trained on INbreast and DDSM-CBIS public datasets, the pipeline achieves an overall average Dice of 80.44% on INbreast test images, outperforming state-of-the-art. Our system shows promising accuracy as an automatic full-image mass segmentation system. Extensive experiments reveals robustness against the diversity of size, shape and appearance of breast masses, towards better interaction-free computer-aided diagnosis.

International audience; Breast cancer screening benefits from the visual analysis of multiple views of routine mammograms. As for clinical practice, computer-aided diagnosis (CAD) systems could be enhanced by integrating multi-view information. In this work, we propose a new multi-tasking framework that combines craniocaudal (CC) and mediolateral-oblique (MLO) mammograms for automatic breast mass detection. Rather than addressing mass recognition only, we exploit multi-tasking properties of deep networks to jointly learn mass matching and classification, towards better detection performance. Specifically, we propose a unified Siamese network that combines patch-level mass/non-mass classification and dual-view mass matching to take full advantage of multi-view information. This model is exploited in a full image detection pipeline based on You-Only-Look-Once (YOLO) region proposals. We carry out exhaustive experiments to highlight the contribution of dual-view matching for both patch-level classification and examination-level detection scenarios. Results demonstrate that mass matching highly improves the full-pipeline detection performance by outperforming conventional single-task schemes with 94.78% as Area Under the Curve (AUC) score and a classification accuracy of 0.8791. Interestingly, mass classification also improves the performance of mass matching, which proves the complementarity of both tasks. Our method further guides clinicians by providing accurate dual-view mass correspondences, which suggests that it could act as a relevant second opinion for mammogram interpretation and breast cancer diagnosis.

Longitudinal medical image analysis is crucial for identifying the unobvious emergence and evolution of early lesions, towards earlier and better patient-specific pathology management. However, traditional computer-aided diagnosis (CAD) systems for diabetic retinopathy (DR) rarely make use of longitudinal information to improve DR analysis. In this work, we present a deep information fusion framework that exploits two consecutive longitudinal studies for the assessment of early DR severity changes. In particular, three fusion schemes are investigated: (1) early fusion of inputs, (2) intermediate fusion of feature vectors incorporating Spatial Transformer Networks (STN) and (3) late fusion of feature vectors. Exhaustive experiments compared with respect to no-fusion baselines validate that incorporating prior DR studies can improve the referable DR severity classification performance through the late fusion scheme whose AUC reaches 0.9296. Advantages and limitations of the different fusion methods are discussed in depth. We also propose different pre-training strategies which are employed to bring considerable performance gains for DR severity grade change detection purposes.

International audience; In clinical practice, radiologists use multiple views of routine mammograms for breast cancer screening. Similarly, computer-aided diagnosis (CAD) systems could be enhanced by integrating information arising from pairs of views. In this work, we present a new multi-tasking framework that combines craniocaudal (CC) and mediolateral-oblique (MLO) mammograms. We exploit multi-tasking properties of deep networks to jointly learn mass matching and classification, towards better detection performance. A combined Siamese model that includes patch-level mass classification and dual-view mass matching is used to take full advantage of multi-view information. This network is exploited in a full image detection pipeline based on You-Only-Look-Once (YOLO) region proposals. Experiments highlight the benefits of dual-view analysis for both patch-level classification and examination-level detection scenarios. Our pipeline outperforms conventional single-task deep models with 94.78% as Area Under the Curve (AUC) score and a classification accuracy of 0.8791. Additionally to these gains, our method further guides clinicians by providing accurate multi-view mass correspondences. This suggests that it could act as a relevant automatic second opinion for mammogram interpretation and breast cancer diagnosis.

The stance and swing phases of the gait cycle are defined by foot strike (FS) and foot off (FO). Accurate determination of these events is thus an essential component of 3D motion recordings processing. Several methods have been developed for the automatic detection of these events (based on the heuristics of 3D marker position, velocity and acceleration), however the results may be inaccurate due to the high variability that is intrinsic to pathological gait. For this reason, gait events are still commonly determined manually, which is a tedious process. Here we propose a new application (DeepEvent) of a long short term memory recurrent neural network for the automatic detection of gait events. The 3D position and velocity of the markers on the heel, toe and lateral malleolus were used by the network to determine FS and FO. The method was developed from 10526 FS and 9375 FO from 226 children. DeepEvent predicted FS within 5.5 ms and FO within 10.7 ms of the gold standard (automatic determination using force platform data) and was more accurate than common heuristic marker trajectory-based methods proposed in the literature and another deep learning method. A sensitivity analysis showed that DeepEvent mainly used the toe and heel markers (z-axis (longitudinal) position and velocity) at the beginning and end of gait cycle to predict FS, and the toe marker (x-axis (anterior/posterior) velocity and z-axis position and velocity) at around 60% of the gait cycle to predict FO.

In recent years, Artificial Intelligence (AI) has proven its relevance for medical decision support. However, the “black-box” nature of successful AI algorithms still holds back their wide-spread deployment. In this paper, we describe an eXplanatory Artificial Intelligence (XAI) that reaches the same level of performance as black-box AI, for the task of classifying Diabetic Retinopathy (DR) severity using Color Fundus Photography (CFP). This algorithm, called ExplAIn, learns to segment and categorize lesions in images; the final image-level classification directly derives from these multivariate lesion segmentations. The novelty of this explanatory framework is that it is trained from end to end, with image supervision only, just like black-box AI algorithms: the concepts of lesions and lesion categories emerge by themselves. For improved lesion localization, foreground/background separation is trained through self-supervision, in such a way that occluding foreground pixels transforms the input image into a healthy-looking image. The advantage of such an architecture is that automatic diagnoses can be explained simply by an image and/or a few sentences. ExplAIn is evaluated at the image level and at the pixel level on various CFP image datasets. We expect this new framework, which jointly offers high classification performance and explainability, to facilitate AI deployment.

Video feedback provides a wealth of information about surgical procedures and is the main sensory cue for surgeons. Scene understanding is crucial to computer assisted interventions (CAI) and to post-operative analysis of the surgical procedure. A fundamental building block of such capabilities is the identification and localization of surgical instruments and anatomical structures through semantic segmentation. Deep learning has advanced semantic segmentation techniques in the recent years but is inherently reliant on the availability of labelled datasets for model training. This paper introduces a dataset for semantic segmentation of cataract surgery videos complementing the publicly available CATARACTS challenge dataset. In addition, we benchmark the performance of several state-of-the-art deep learning models for semantic segmentation on the presented dataset. The dataset is publicly available at https://cataracts-semantic-segmentation2020.grand-challenge.org/.

The automatic analysis of the surgical process, from videos recorded during surgeries, could be very useful to surgeons, both for training and for acquiring new techniques. The training process could be optimized by automatically providing some targeted recommendations or warnings, similar to the expert surgeon’s guidance. In this paper, we propose to reuse videos recorded and stored during cataract surgeries to perform the analysis. The proposed system allows to automatically recognize, in real time, what the surgeon is doing: what surgical phase or, more precisely, what surgical step he or she is performing. This recognition relies on the inference of a multilevel statistical model which uses 1) the conditional relations between levels of description (steps and phases) and 2) the temporal relations among steps and among phases. The model accepts two types of inputs: 1) the presence of surgical tools, manually provided by the surgeons, or 2) motion in videos, automatically analyzed through the Content Based Video retrieval (CBVR) paradigm. Different data-driven statistical models are evaluated in this paper. For this project, a dataset of 30 cataract surgery videos was collected at Brest University hospital. The system was evaluated in terms of area under the ROC curve. Promising results were obtained using either the presence of surgical tools (A z = 0.983) or motion analysis (A z = 0.759). The generality of the method allows to adapt it to other kinds of surgeries. The proposed solution could be used in a computer assisted surgery tool to support surgeons during the surgery.

This paper addresses breast mass segmentation from high-resolution mammograms. To cope with strong class imbalance, huge diversity of size, shape, texture and contour as well as limited receptive field, mass segmentation is achieved through a multi-scale cascade of deep convolutional encoder-decoders without any pre-detection scheme. Multi-scale information is integrated using auto-context to make long-range spatial context arising from lower scale impact training at higher resolution. The pipeline is trained end-to-end to benefit from simultaneous segmentation refinement performed at each level. It incorporates transfer learning and fine tuning from DDSM-CBIS to INbreast datasets to further improve mass delineations. The comprehensive evaluation provided for high-resolution INbreast images highlights promising model generalizability against standard encoder-decoder strategies.

In the last decades, large datasets of fundus photographs have been collected in diabetic retinopathy (DR) screening networks. Through deep learning, these datasets were used to train automatic detectors for DR and a few other frequent pathologies, with the goal to automate screening. One challenge limits the adoption of such systems so far: automatic detectors ignore rare conditions that ophthalmologists currently detect, such as papilledema or anterior ischemic optic neuropathy. The reason is that standard deep learning requires too many examples of these conditions. However, this limitation can be addressed with few-shot learning, a machine learning paradigm where a classifier has to generalize to a new category not seen in training, given only a few examples of this category. This paper presents a new few-shot learning framework that extends convolutional neural networks (CNNs), trained for frequent conditions, with an unsupervised probabilistic model for rare condition detection. It is based on the observation that CNNs often perceive photographs containing the same anomalies as similar, even though these CNNs were trained to detect unrelated conditions. This observation was based on the t-SNE visualization tool, which we decided to incorporate in our probabilistic model. Experiments on a dataset of 164,660 screening examinations from the OPHDIAT screening network show that 37 conditions, out of 41, can be detected with an area under the ROC curve (AUC) greater than 0.8 (average AUC: 0.938). In particular, this framework significantly outperforms other frameworks for detecting rare conditions, including multitask learning, transfer learning and Siamese networks, another few-shot learning solution. We expect these richer predictions to trigger the adoption of automated eye pathology screening, which will revolutionize clinical practice in ophthalmology.

International audience; Finding correspondences between structural entities decomposing images is of high interest for computer vision applications. In particular, we analyze how to accurately track superpixels - visual primitives generated by aggregating adjacent pixels sharing similar characteristics - over extended time periods relying on unsupervised learning and temporal integration. A two-step video processing pipeline dedicated to long-term superpixel tracking is proposed. First, unsupervised learning-based superpixel matching provides correspondences between consecutive and distant frames using new context-rich features extended from greyscale to multi-channel and forward-backward consistency constraints. Resulting elementary matches are then combined along multi-step paths running through the whole sequence with various inter-frame distances. This produces a large set of candidate long-term superpixel pairings upon which majority voting is performed. Video object tracking experiments demonstrate the accuracy of our elementary estimator against state-of-the-art methods and proves the ability of multi-step integration to provide accurate long-term superpixel matches compared to usual direct and sequential integration.

Diabetic Retinopathy (DR) is the most common cause of avoidable vision loss, predominantly affecting the working-age population across the globe. Screening for DR, coupled with timely consultation and treatment, is a globally trusted policy to avoid vision loss. However, implementation of DR screening programs is challenging due to the scarcity of medical professionals able to screen a growing global diabetic population at risk for DR. Computer-aided disease diagnosis in retinal image analysis could provide a sustainable approach for such large-scale screening effort. The recent scientific advances in computing capacity and machine learning approaches provide an avenue for biomedical scientists to reach this goal. Aiming to advance the state-of-the-art in automatic DR diagnosis, a grand challenge on "Diabetic Retinopathy - Segmentation and Grading" was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI - 2018). In this paper, we report the set-up and results of this challenge that is primarily based on Indian Diabetic Retinopathy Image Dataset (IDRiD). There were three principal sub-challenges: lesion segmentation, disease severity grading, and localization of retinal landmarks and segmentation. These multiple tasks in this challenge allow to test the generalizability of algorithms, and this is what makes it different from existing ones. It received a positive response from the scientific community with 148 submissions from 495 registrations effectively entered in this challenge. This paper outlines the challenge, its organization, the dataset used, evaluation methods and results of top-performing participating solutions. The top-performing approaches utilized a blend of clinical information, data augmentation, and an ensemble of models. These findings have the potential to enable new developments in retinal image analysis and image-based DR screening in particular.

International audience; Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparo-scopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected,-H. Conze et al. / Medical Image Analysis 52 (2019) 24-41 25 the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.

The world faces difficulties in terms of eye care, including treatment, quality of prevention, vision rehabilitation services, and scarcity of trained eye care experts. Early detection and diagnosis of ocular pathologies would enable forestall of visual impairment. One challenge that limits the adoption of computer-aided diagnosis tool by ophthalmologists is the number of sight-threatening rare pathologies, such as central retinal artery occlusion or anterior ischemic optic neuropathy, and others are usually ignored. In the past two decades, many publicly available datasets of color fundus images have been collected with a primary focus on diabetic retinopathy, glaucoma, age-related macular degeneration and few other frequent pathologies. To enable development of methods for automatic ocular disease classification of frequent diseases along with the rare pathologies, we have created a new Retinal Fundus Multi-disease Image Dataset (RFMiD). It consists of 3200 fundus images captured using three different fundus cameras with 46 conditions annotated through adjudicated consensus of two senior retinal experts. To the best of our knowledge, our dataset, RFMiD, is the only publicly available dataset that constitutes such a wide variety of diseases that appear in routine clinical settings. This dataset will enable the development of generalizable models for retinal screening.

This paper investigates the automatic monitoring of tool usage during a surgery, with potential applications in report generation, surgical training and real-time decision support. Two surgeries are considered: cataract surgery, the most common surgical procedure, and cholecystectomy, one of the most common digestive surgeries. Tool usage is monitored in videos recorded either through a microscope (cataract surgery) or an endoscope (cholecystectomy). Following state-of-the-art video analysis solutions, each frame of the video is analyzed by convolutional neural networks (CNNs) whose outputs are fed to recurrent neural networks (RNNs) in order to take temporal relationships between events into account. Novelty lies in the way those CNNs and RNNs are trained. Computational complexity prevents the end-to-end training of "CNN+RNN" systems. Therefore, CNNs are usually trained first, independently from the RNNs. This approach is clearly suboptimal for surgical tool analysis: many tools are very similar to one another, but they can generally be differentiated based on past events. CNNs should be trained to extract the most useful visual features in combination with the temporal context. A novel boosting strategy is proposed to achieve this goal: the CNN and RNN parts of the system are simultaneously enriched by progressively adding weak classifiers (either CNNs or RNNs) trained to improve the overall classification accuracy. Experiments were performed in a dataset of 50 cataract surgery videos and a dataset of 80 cholecystectomy videos. Very good classification performance are achieved in both datasets: tool usage could be labeled with an average area under the ROC curve of $A_z = 0.9961$ and $A_z = 0.9939$, respectively, in offline mode (using past, present and future information), and $A_z = 0.9957$ and $A_z = 0.9936$, respectively, in online mode (using past and present information only)., Accepted for publication in Medical Image Analysis

Purpose A retrospective pilot study is conducted to demonstrate the utility of a novel support vector machine learning (SVML) algorithm in a small three-dimensional (3D) sample yielding sparse optical coherence tomography (spOCT) data for the automatic monitoring of neovascular (wet) age-related macular degeneration (wAMD). Methods From the anti-vascular endothelial growth factor injection database, 588 consecutive pairs of OCT volumes (57.624 B-scans) were selected in 70 randomly chosen wAMD patients treated with ranibizumab. The SVML algorithm was applied to 183 OCT volume pairs (17.934 B-scans) in 30 patients. Four independent, diagnosis-blinded retina specialists indicated whether wAMD activity was present between 100 pairs of consecutive OCT volumes (9800 B-scans) in the remaining 40 patients for comparison with the SVML algorithm and a non-complex baseline algorithm using only retinal thickness. The SVML algorithm was assessed using inter-observer variability and receiver operating characteristic (ROC) analyses. Results The retina specialists showed an average Cohen's κ of 0.57 ± 0.13 (minimum: 0.41, maximum: 0.83). The average κ between the proposed algorithm and the retina specialists was 0.62 ± 0.05 and 0.43 ± 0.14 between the baseline algorithm and the retina specialists. Using each of the four retina specialists as the reference, the proposed method showed a superior area under the ROC curve of 0.91 ± 0.03 compared to the ROC 0.81 ± 0.05 shown by the baseline algorithm. Conclusion The SVML algorithm was as effective as the retina specialists were in detecting activity in wAMD. Support vector machine learning (SVML) may be a useful monitoring tool in wAMD suited for small samples that yield sparse OCT data possibly derived from self-measuring OCT-robots.

International audience; In recent years, endoscopy has been increasingly used for ocular surgeries whenever viewing with a microscope is altered by occlusion or opacity. However, ocular endoscopy suffers from several limitations, including reduced field of view and limited resolution, which may compromise its usability. Hopefully, image processing techniques such as mosaicking and super-resolution could help in alleviating these problems, by artificially enlarging the field and increasing resolution. All these techniques rely on the ability to estimate the optical flow between consecutive frames, which is particularly challenging for those images. This paper investigates the use of the state-of-the-art FlowNet algorithm for motion estimation in ocular endoscopy videos. Because FlowNet is strongly supervised, an artificial dataset of consecutive image pairs with ground truth optical flow is generated using eye fundus photographs from Kaggle's Diabetic Retinopathy Detection dataset. A FlowNet model, initialized on the public Flying Chairs dataset, is fine-tuned on these images. Initial experiments show that, unlike any other optical flow estimation method, this model can successfully capture motion between ocular endoscopy image pairs.

International audience; In this paper, we analyze how to accurately track superpixels over extended time periods for computer vision applications. A two-step video processing pipeline dedicated to long-term superpixel tracking is proposed based on unsupervised learning and temporal integration. First, unsupervised learning-based matching provides superpixel correspondences between consecutive and distant frames using context-rich features extended from greyscale to multi-channel. Resulting elementary matches are then combined along multi-step paths running through the whole sequence with various inter-frame distances. This produces a large set of candidate long-term superpixel pairings upon which majority voting is performed. Video object tracking experiments demonstrate the efficiency of this pipeline against state-of-the-art methods.

International audience; This paper presents an innovative way to reach accurate semi-dense registration between images based on robust matching of structural entities. The proposed approach relies on a decomposition of images into visual primitives called supervoxels generated by aggregating adjacent voxels sharing similar characteristics. Two new categories of features are estimated at the supervoxel extent: mid-level spectral features relying on a spectral method applied on supervoxel graphs to capture the non-linear modes of intensity displacements, and mid-level context-rich features describing the broadened spatial context on the resulting spectral representations. Accurate supervoxel pairings are established by nearest neighbor search on these newly designed features. The effectiveness of the approach is demonstrated against state-of-the-art methods for semi-dense longitudinal registration of abdominal CT images, relying on liver label propagation and consistency assessment.

In this paper, we propose an homomorphic proxy re-encryption scheme (HPRE) that allows different users to share data they outsourced homomorphically encrypted using their respective public keys with the possibility by next to process such data remotely. Its originality stands on a solution we propose so as to compute the difference of data encrypted with Damgard-Jurik cryptosystem. It takes also advantage of a secure combined linear congruential generator that we implemented in the Damgard-Jurik encrypted domain. Basically, in our HPRE scheme, the two users, the delegator and the delegate, ask the cloud server to generate an encrypted noise based on a secret key, both users previously agreed on. Based on our solution to compute the difference in Damgard-Jurik encrypted domain, the cloud computes in clear the differences in-between the encrypted noise and the encrypted data of the delegator, obtaining thus blinded data. In order the delegate gets access to the data, the cloud just has to encrypt these differences using the delegate's public key and then removes the noise. This solution doesn't need extra communication between the cloud and the delegator. Our HPRE was implemented in the case of the sharing of uncompressed images stored in the cloud showing good time computation performance, it is unidirectional and collusion-resistant. Nevertheless, it is not limited to images and can be used with any kinds of data.

This paper introduces a new algorithm for recognizing surgical tasks in real-time in a video stream. The goal is to communicate information to the surgeon in due time during a video-monitored surgery. The proposed algorithm is applied to cataract surgery, which is the most common eye surgery. To compensate for eye motion and zoom level variations, cataract surgery videos are first normalized. Then, the motion content of short video subsequences is characterized with spatiotemporal polynomials: a multiscale motion characterization based on adaptive spatiotemporal polynomials is presented. The proposed solution is particularly suited to characterize deformable moving objects with fuzzy borders, which are typically found in surgical videos. Given a target surgical task, the system is trained to identify which spatiotemporal polynomials are usually extracted from videos when and only when this task is being performed. These key spatiotemporal polynomials are then searched in new videos to recognize the target surgical task. For improved performances, the system jointly adapts the spatiotemporal polynomial basis and identifies the key spatiotemporal polynomials using the multiple-instance learning paradigm. The proposed system runs in real-time and outperforms the previous solution from our group, both for surgical task recognition ( $A_z = 0.851$ on average, as opposed to $A_z = 0.794$ previously) and for the joint segmentation and recognition of surgical tasks ( $A_z = 0.856$ on average, as opposed to $A_z = 0.832$ previously).

In recent years, several algorithms were proposed to monitor a surgery through the automatic analysis of endoscope or microscope videos. This paper aims at improving existing solutions for the automated analysis of cataract surgeries, the most common ophthalmic surgery, which are performed under a microscope. Through the analysis of a video recording the surgical tray, it is possible to know which tools are put on or taken from the surgical tray, and therefore which ones are likely being used by the surgeon. Combining these observations with observations from the microscope video should enhance the overall performance of the system. Our contribution is twofold: first, datasets of artificial surgery videos are generated in order to train the convolutional neural networks (CNN) and, second, two classification methods are evaluated to detect the presence of tools in videos. Also, we assess the impact of the manner of building the artificial datasets on the tool recognition performance. By design, the proposed artificial datasets highly reduce the need for fully annotated real datasets and should also produce better performance. Experiments show that one of the proposed classification methods was able to detect most of the targeted tools well.

The automatic detection of surgical tools in surgery videos is a promising solution for surgical workflow analysis. It paves the way to various applications, including surgical workflow optimization, surgical skill evaluation and real-time warning generation. A solution based on convolutional neural networks (CNNs) is proposed in this paper. Unlike existing solutions, the proposed CNN does not analyze images independently. it analyzes sequences of consecutive images. Features extracted from each image by the CNN are fused inside the network using the optical flow. For improved performance, this multi-image fusion strategy is also applied while training the CNN. The proposed framework was evaluated in a dataset of 30 cataract surgery videos (6 hours of videos). Ten tool categories were defined by surgeons. The proposed system was able to detect each of these categories with a high area under the ROC curve (0.953 ≤ A z ≤ 0.987). The proposed detector, based on multi-image fusion, was significantly more sensitive and specific than a similar system analyzing images independently (p = 2.98 × 10−6 and p = 2.07 × 10−3, respectively).

In this paper, we propose a new secure content based image retrieval (SCBIR) system adapted to the cloud framework. This solution allows a physician to retrieve images of similar content within an outsourced and encrypted image database, without decrypting them. Contrarily to actual CBIR approaches in the encrypted domain, the originality of the proposed scheme stands on the fact that the features extracted from the encrypted images are themselves encrypted. This is achieved by means of homomorphic encryption and two non-colluding servers, we however both consider as honest but curious. In that way an end to end secure CBIR process is ensured. Experimental results carried out on a diabetic retinopathy database encrypted with the Paillier cryptosystem indicate that our SCBIR achieves retrieval performance as good as if images were processed in their non-encrypted form.

With the increased prevalence of retinal pathologies, automating the detection and progression measurement of these pathologies is becoming more and more relevant. Color fundus photography is the leading modality for assessing retinal pathologies. Because eye fundus cameras have a limited field of view, multiple photographs are taken from each retina during an eye fundus examination. However, operators usually don't indicate which photographs are from the left retina and which ones are from the right retina. This paper presents a novel algorithm that automatically assigns each photograph to one retina and builds a composite image (or “mosaic”) per retina, which is expected to push the performance of automated diagnosis forward. The algorithm starts by jointly forming two mosaics, one per retina, using a novel graph theoretic approach. Then, in order to determine which mosaic corresponds to the left retina and which one corresponds to the right retina, two retinal landmarks are detected robustly in each mosaic: the main vessel arch surrounding the macula and the optic disc. The laterality of each mosaic derives from their relative location. Experiments on 2790 manually annotated images validate the very good performance of the proposed framework even for highly pathological images.

Mammography remains the most effective tool for the early detection of breast cancer, as well as the systems of computer-aided detection/diagnosis (CAD) is typically used as a second opinion by the radiologists. So, the main goal of our method is to introduce a new approach for automatic detecting the suspicious lesions in mammograms (regions of interest) for early diagnosis of breast cancer. This study has two phases: The first one is the preprocessing step and the second one is the detection of Regions of Interest (ROIs). Our method has tested with the well-known Mammography Image Analysis Society (MIAS) database and we’ve used Free-Receiver Operating Characteristics (FROC) to measure methods performance. The obtained experimental results show that our algorithm’s performance has sensitivity of 94.75 % at 0.54 false positive per image.

Deep learning is quickly becoming the leading methodology for medical image analysis. Given a large medical archive, where each image is associated with a diagnosis, efficient pathology detectors or classifiers can be trained with virtually no expert knowledge about the target pathologies. However, deep learning algorithms, including the popular ConvNets, are black boxes: little is known about the local patterns analyzed by ConvNets to make a decision at the image level. A solution is proposed in this paper to create heatmaps showing which pixels in images play a role in the image-level predictions. In other words, a ConvNet trained for image-level classification can be used to detect lesions as well. A generalization of the backpropagation method is proposed in order to train ConvNets that produce high-quality heatmaps. The proposed solution is applied to diabetic retinopathy (DR) screening in a dataset of almost 90,000 fundus photographs from the 2015 Kaggle Diabetic Retinopathy competition and a private dataset of almost 110,000 photographs (e-ophtha). For the task of detecting referable DR, very good detection performance was achieved: $A_z = 0.954$ in Kaggle's dataset and $A_z = 0.949$ in e-ophtha. Performance was also evaluated at the image level and at the lesion level in the DiaretDB1 dataset, where four types of lesions are manually segmented: microaneurysms, hemorrhages, exudates and cotton-wool spots. The proposed detector outperforms recent algorithms trained to detect those lesions specifically, as well as competing heatmap generation algorithms for ConvNets. This detector is part of the Messidor system for mobile eye pathology screening. Because it does not rely on expert knowledge or manual segmentation for detecting relevant patterns, the proposed solution is a promising image mining tool, which has the potential to discover new biomarkers in images., Accepted for publication in Medical Image Analysis

A complete prototype for the automatic detection of normal examinations on a teleophthalmology network for diabetic retinopathy screening is presented. The system combines pathological pattern mining methods, with specific lesion detection methods, to extract information from the images. This information, plus patient and other contextual data, is used by a classifier to compute an abnormality risk. Such a system should reduce the burden on readers on teleophthalmology networks.

PURPOSE We assessed the suitability of a low-cost, handheld, nonmydriatic retinograph, namely the Horus DEC 200, for diabetic retinopathy (DR) diagnosis. Two factors were considered: ease of image acquisition and image quality. METHODS One operator acquired fundus photographs from 54 patients using the Horus and AFC-330, a more expensive, nonportable retinograph. Satisfaction surveys were filled out by patients. Then, two retinologists subjectively assessed image quality and graded DR severity in one eye of each patient. Objective image quality indices also were computed. RESULTS During image acquisitions, patients had difficulty locating the fixation target inside the Horus: by default, 53.7% of them had to fixate external points with the contralateral eye, as opposed to none of them using the AFC-330 (P < 0.0001). This issue impacted the duration of image acquisitions. Images obtained by the Horus were of significantly lower quality according to the experts (P = 0.0002 and P = 0.0004) and to the objective criterion (P < 0.0001). As a result, up to 20.4% of eyes were inadequate for interpretation, as opposed to 9.3% using the AFC-330. However, no significant difference was found in terms of DR severity according to both experts (P = 0.557 and P = 0.156). CONCLUSIONS The Horus can be used to screen DR, but at the cost of longer examination times and higher proportions of patients referred to an ophthalmologist due to inadequate image quality. TRANSLATIONAL RELEVANCE The Horus is adequate to screen DR, for instance in primary care centers or in mobile imaging units.

This paper describes a computer-aided detection and diagnosis system for breast cancer, the most common form of cancer among women, using mammography. The system relies on the Multiple-Instance Learning (MIL) paradigm, which has proven useful for medical decision support in previous works from our team. In the proposed framework, breasts are first partitioned adaptively into regions. Then, features derived from the detection of lesions (masses and microcalcifications) as well as textural features, are extracted from each region and combined in order to classify mammography examinations as “normal” or “abnormal”. Whenever an abnormal examination record is detected, the regions that induced that automated diagnosis can be highlighted. Two strategies are evaluated to define this anomaly detector. In a first scenario, manual segmentations of lesions are used to train an SVM that assigns an anomaly index to each region; local anomaly indices are then combined into a global anomaly index. In a second scenario, the local and global anomaly detectors are trained simultaneously, without manual segmentations, using various MIL algorithms (DD, APR, mi-SVM, MI-SVM and MILBoost). Experiments on the DDSM dataset show that the second approach, which is only weakly-supervised, surprisingly outperforms the first approach, even though it is strongly-supervised. This suggests that anomaly detectors can be advantageously trained on large medical image archives, without the need for manual segmentation.

Adaptive wavelet-based image characterizations have been proposed in previous works for content-based image retrieval (CBIR) applications. In these applications, the same wavelet basis was used to characterize each query image: This wavelet basis was tuned to maximize the retrieval performance in a training data set. We take it one step further in this paper: A different wavelet basis is used to characterize each query image. A regression function, which is tuned to maximize the retrieval performance in the training data set, is used to estimate the best wavelet filter, i.e., in terms of expected retrieval performance, for each query image. A simple image characterization, which is based on the standardized moments of the wavelet coefficient distributions, is presented. An algorithm is proposed to compute this image characterization almost instantly for every possible separable or nonseparable wavelet filter. Therefore, using a different wavelet basis for each query image does not considerably increase computation times. On the other hand, significant retrieval performance increases were obtained in a medical image data set, a texture data set, a face recognition data set, and an object picture data set. This additional flexibility in wavelet adaptation paves the way to relevance feedback on image characterization itself and not simply on the way image characterizations are combined.

Automated detection of lesions in retinal images is a crucial step towards efficient early detection, or screening, of large at-risk populations. In particular, the detection of microaneurysms, usually the first sign of diabetic retinopathy (DR), and the detection of drusen, the hallmark of age-related macular degeneration (AMD), are of primary importance. In spite of substantial progress made, detection algorithms still produce 1) false positives - target lesions are mixed up with other normal or abnormal structures in the eye, and 2) false negatives - the large variability in the appearance of the lesions causes a subset of these target lesions to be missed. We propose a general framework for detecting and characterizing target lesions almost instantaneously. This framework relies on a feature space automatically derived from a set of reference image samples representing target lesions, including atypical target lesions, and those eye structures that are similar looking but are not target lesions. The reference image samples are obtained either from an expert- or a data-driven approach. Factor analysis is used to derive the filters generating this feature space from reference samples. Previously unseen image samples are then classified in this feature space. We tested this approach by training it to detect microaneurysms. On a set of images from 2739 patients including 67 with referable DR, DR detection area under the receiver-operating characteristic curve (AUC) was comparable (AUC=0.927) to our previously published red lesion detection algorithm (AUC=0.929). We also tested the approach on the detection of AMD, by training it to differentiate drusen from Stargardt's disease lesions, and achieved an AUC=0.850 on a set of 300 manually detected drusen and 300 manually detected flecks. The entire image processing sequence takes less than a second on a standard PC compared to minutes in our previous approach, allowing instantaneous detection. Free-response receiver-operating characteristic analysis showed the superiority of this approach over a framework where false positives and the atypical lesions are not explicitly modeled. A greater performance was achieved by the expert-driven approach for DR detection, where the designer had sound expert knowledge. However, for both problems, a comparable performance was obtained for both expert- and data-driven approaches. This indicates that annotation of a limited number of lesions suffices for building a detection system for any type of lesion in retinal images, if no expert-knowledge is available. We are studying whether the optimal filter framework also generalizes to the detection of any structure in other domains.

Purpose To compare the performance of automated diabetic retinopathy (DR) detection, using the algorithm that won the 2009 Retinopathy Online Challenge Competition in 2009, the Challenge2009, against that of the one currently used in EyeCheck, a large computer-aided early DR detection project. Design Evaluation of diagnostic test or technology. Participants Fundus photographic sets, consisting of 2 fundus images from each eye, were evaluated from 16 670 patient visits of 16 670 people with diabetes who had not previously been diagnosed with DR. Methods The fundus photographic set from each visit was analyzed by a single retinal expert; 793 of the 16 670 sets were classified as containing more than minimal DR (threshold for referral). The outcomes of the 2 algorithmic detectors were applied separately to the dataset and were compared by standard statistical measures. Main Outcome Measures The area under the receiver operating characteristic curve (AUC), a measure of the sensitivity and specificity of DR detection. Results Agreement was high, and examination results indicating more than minimal DR were detected with an AUC of 0.839 by the EyeCheck algorithm and an AUC of 0.821 for the Challenge2009 algorithm, a statistically nonsignificant difference (z-score, 1.91). If either of the algorithms detected DR in combination, the AUC for detection was 0.86, the same as the theoretically expected maximum. At 90% sensitivity, the specificity of the EyeCheck algorithm was 47.7% and that of the Challenge2009 algorithm was 43.6%. Conclusions Diabetic retinopathy detection algorithms seem to be maturing, and further improvements in detection performance cannot be differentiated from best clinical practices, because the performance of competitive algorithm development now has reached the human intrareader variability limit. Additional validation studies on larger, well-defined, but more diverse populations of patients with diabetes are needed urgently, anticipating cost-effective early detection of DR in millions of people with diabetes to triage those patients who need further care at a time when they have early rather than advanced DR. Financial Disclosure(s) Proprietary or commercial disclosure may be found after the references.

Optical coherence tomography (OCT) is becoming one of the most important modalities for the noninvasive assessment of retinal eye diseases. As the number of acquired OCT volumes increases, automating the OCT image analysis is becoming increasingly relevant. In this paper, a method for automated characterization of the normal macular appearance in spectral domain OCT (SD-OCT) volumes is reported together with a general approach for local retinal abnormality detection. Ten intraretinal layers are first automatically segmented and the 3-D image dataset flattened to remove motion-based artifacts. From the flattened OCT data, 23 features are extracted in each layer locally to characterize texture and thickness properties across the macula. The normal ranges of layer-specific feature variations have been derived from 13 SD-OCT volumes depicting normal retinas. Abnormalities are then detected by classifying the local differences between the normal appearance and the retinal measures in question. This approach was applied to determine footprints of fluid-filled regions--SEADs (Symptomatic Exudate-Associated Derangements)--in 78 SD-OCT volumes from 23 repeatedly imaged patients with choroidal neovascularization (CNV), intra-, and sub-retinal fluid and pigment epithelial detachment. The automated SEAD footprint detection method was validated against an independent standard obtained using an interactive 3-D SEAD segmentation approach. An area under the receiver-operating characteristic curve of 0.961 +/- 0.012 was obtained for the classification of vertical, cross-layer, macular columns. A study performed on 12 pairs of OCT volumes obtained from the same eye on the same day shows that the repeatability of the automated method is comparable to that of the human experts. This work demonstrates that useful 3-D textural information can be extracted from SD-OCT scans and--together with an anatomical atlas of normal retinas--can be used for clinically important applications.

International audience; We propose in this article a Content-Based Image Retrieval method for diagnosis aid in medical fields. In the proposed system, images are indexed in a generic fashion, without extracting domain-specific features: a signature is built for each image from its wavelet transform. These image signatures characterize the distribution of wavelet coefficients in each subband of the decomposition. A distance measure is then defined to compare two image signatures and thus retrieve the most similar images in a database when a query image is submitted by a physician. To retrieve relevant images from a medical database, the signatures and the distance measure must be related to the medical interpretation of images. As a consequence, we introduce several degrees of freedom in the system so that it can be tuned to any pathology and image modality. In particular, we propose to adapt the wavelet basis, within the lifting scheme framework, and to use a custom decomposition scheme. Weights are also introduced between subbands. All these parameters are tuned by an optimization procedure, using the medical grading of each image in the database to define a performance measure. The system is assessed on two medical image databases: one for diabetic retinopathy follow up and one for screening mammography, as well as a general purpose database. Results are promising: a mean precision of 56.50%, 70.91% and 96.10% is achieved for these three databases, when five images are returned by the system.

11 pages; International audience; In this paper, we propose an automatic method to detect microaneurysms in retina photographs. Microaneurysms are the most frequent and usually the first lesions to appear as a consequence of diabetic retinopathy. So, their detection is necessary for both screening the pathology and follow up (progression measurement). Automating this task, which is currently performed manually, would bring more objectivity and reproducibility. We propose to detect them by locally matching a lesion template in subbands of wavelet transformed images. To improve the method performance, we have searched for the best adapted wavelet within the lifting scheme framework. The optimization process is based on a genetic algorithm followed by Powell's direction set descent. Results are evaluated on 120 retinal images analyzed by an expert and the optimal wavelet is compared to different conventional mother wavelets. These images are of three different modalities: there are color photographs, green filtered photographs, and angiographs. Depending on the imaging modality, microaneurysms were detected with a sensitivity of respectively 89.62%, 90.24%, and 93.74% and a positive predictive value of respectively 89.50%, 89.75%, and 91.67%, which is better than previously published methods.

International audience; In this paper, we propose a secure implementation of a content-based image retrieval (CBIR) method that makes possible diagnosis aid systems to work in externalized environment and with outsourced data as in cloud computing. This one works with homomorphic encrypted images from which it extracts wavelet based image features next used for subsequent image comparison. By doing so, our system allows a physician to retrieve the most similar images to a query image in an outsourced database while preserving data confidentiality. Our Secure CBIR is the first one that proposes to work with global image features extracted from encrypted images and does not induce extra communications in-between the client and the server. Experimental results show it achieves retrieval performance as good as if images were processed non-encrypted.

This paper describes an experimental computer-aided detection and diagnosis system for breast cancer, the most common form of cancer among women, using mammography. The system relies on the Multiple-Instance Learning (MIL) paradigm, which has proven useful for medical decision support in previous works from our team. In the proposed framework, the breasts are first partitioned adaptively into regions. Then, either textural features, or features derived from the detection of masses and microcalcifications, are extracted from each region. Finally, feature vectors extracted from each region are combined using an MIL algorithm (Citation k-NN or mi-Graph), in order to recognize "normal" mammography examinations or to categorize examinations as "normal", "benign" or "cancer". An accuracy of 91.1% (respectively 62.1%) was achieved for normality recognition (respectively three-class categorization) in a subset of 720 mammograms from the DDSM dataset. The paper also discusses future improvements, that will make the most of the MIL paradigm, in order to improve "benign" versus "cancer" discrimination in particular.

With the increased prevalence of retinal pathologies, automating the detection of these pathologies is becoming more and more relevant. In the past few years, many algorithms have been developed for the automated detection of a specific pathology, typically diabetic retinopathy, using eye fundus photography. No matter how good these algorithms are, we believe many clinicians would not use automatic detection tools focusing on a single pathology and ignoring any other pathology present in the patient’s retinas. To solve this issue, an algorithm for characterizing the appearance of abnormal retinas, as well as the appearance of the normal ones, is presented. This algorithm does not focus on individual images: it considers examination records consisting of multiple photographs of each retina, together with contextual information about the patient. Specifically, it relies on data mining in order to learn diagnosis rules from characterizations of fundus examination records. The main novelty is that the content of examination records (images and context) is characterized at multiple levels of spatial and lexical granularity: 1) spatial flexibility is ensured by an adaptive decomposition of composite retinal images into a cascade of regions, 2) lexical granularity is ensured by an adaptive decomposition of the feature space into a cascade of visual words. This multigranular representation allows for great flexibility in automatically characterizing normality and abnormality: it is possible to generate diagnosis rules whose precision and generalization ability can be traded off depending on data availability. A variation on usual data mining algorithms, originally designed to mine static data, is proposed so that contextual and visual data at adaptive granularity levels can be mined. This framework was evaluated in e-ophtha, a dataset of 25,702 examination records from the OPHDIAT screening network, as well as in the publicly-available Messidor dataset. It was successfully applied to the detection of patients that should be referred to an ophthalmologist and also to the specific detection of several pathologies.

International audience; Huge amounts of surgical data are recorded during video-monitored surgery. Content-based video retrieval systems intent to reuse those data for computer-aided surgery. In this paper, we focus on real-time recognition of cataract surgery steps: the goal is to retrieve from a database surgery videos that were recorded during the same surgery step. The proposed system relies on motion features for video character- ization. Motion features are usually impacted by eye motion or zoom level variations, which are not necessarily relevant for surgery step recognition. Those problems certainly limit the performance of the retrieval system. We therefore propose to refine motion feature extraction by applying pre-processing steps based on a novel pupil center and scale tracking method. Those pre-processing steps are evaluated for two different motion features. In this paper, a similarity measure adapted from Piciarelli's video surveillance system is evaluated for the first time in a surgery dataset. This similarity measure provides good results and for both motion features, the proposed pre- processing steps improved the retrieval performance of the system significantly.

During a medical examination, clinicians build a health record containing all available information about their patient. A promising way to support their decisions is to retrieve similar patient records from a medical archive. Confronted to similar cases, clinicians may confirm or revise their decisions by analogy reasoning. In order to retrieve patient records, two challenges need to be addressed. First, how to characterize complex elements in patient records (images, videos, etc.)? Second, how to combine heterogeneous elements in these records (demographic and clinical data, images, videos, etc.) in order to define clinically-relevant similarity metrics? After a short review of content-based image, video or health record retrieval techniques, this chapter presents the solutions we have developed for two applications in ophthalmology: computer-aided retinal diagnosis and computer-aided eye surgery. Medical archives are a great asset to develop the medical decision supports of tomorrow. Thanks to major advances in information retrieval, network data storage (cloud), with related topics such as security, virtually any medical decision problem can benefit from information stored in medical archives.