Technologies for Augmented Reality Breast Surgery

Institution: Stanford University
Investigator(s): Bruce  Daniel , M.D. -
Award Cycle: 2016 (Cycle 22) Grant #: 22IB-0006 Award: $235,100
Award Type: IDEA
Research Priorities
Detection, Prognosis and Treatment>Innovative Treatment Modalities: search for a cure



Initial Award Abstract (2016)

Non-technical overview of the research topic and relevance to breast cancer: Augmented Reality is a novel technology in which computer-generated virtual objects appear projected onto real objects in the world around us. Although primarily being developed for gaming and industrial design applications, Augmented Reality has great promise to improve medical care. One of the main challenges for breast cancer surgeons is to remove early stage breast cancers without leaving any residual tumor in the breast. MRI can reveal breast cancer accurately. The goal of this research is to develop Augmented Reality methods to accurately display MRI on the patient’s actual breast, in 3 dimensions, interactively right before surgery, to help surgeon plan a more successful operation.

The question(s) or central hypotheses of the research: The central hypothesis of this research is that fusion of MRI with the actual breast will enable surgeons to avoid leaving residual non-palpable tumor behind during lumpectomy, and thus reduce the number of repeat surgeries prompted by incomplete resection, which currently occurs in about 25% of cases. The specific question considered in this proposal is: how accurately can previously acquired MRI data be aligned and depicted on the breast using augmented reality..

The general methodology: We propose two areas of research: technology development, and clinical testing. The objective of the technology development is to create an accurate, robust system for projecting the location, extent, and appearance of breast cancer and its surrounding tissues onto the patientís actual breast. This will entail developing new methods to make MRI pictures of the breast when the patient is lying ďsupineĒ, on her back. It will also require methods to align those images to views of the actual breast being presented by the Augmented Reality device. The objective of the Clinical Testing is to see how accurately the system is showing the location and extent of breast cancer in the breast. Since this is the first test of the system in women with breast cancer, it will be done in women who donít need it, specifically women with tumors that can be easily felt and removed by the surgeon without any images. This will pave the way for future tests in larger groups of cancer patients to see if it can reduce the number of repeat surgeries that are done because of concern for tumor left behind in the breast. .

Innovative elements of the project and potential impact: Supine MRI would improve the use of MRI for surgical planning and provide a more comfortable option for women getting breast MRIís The Augmented Reality display of MRI on the actual breast is entirely novel If it is successful in enabling more surgeries that donít leave residual tumor, then it could eliminate thousands of repeat surgeries in the US annually.




Progress Report 1 (2017)

Each year, approximately 300,000 women are diagnosed with breat cancer in the United States. Most choose lumpectomy as their initial surgery. Unfortunately, about one quarter of lumpectomy patients, about 40,000 women, must undergo repeat surgery because tumor was found near the edge of the removed specimen, raising concern for residual tumor in the breast. This happens because the surgeon cannot see or feel the full extent of their patient's tumor during surgery. To tackle this problem, we are creating an augmented reality system that enables the surgeon to see the tumro right inside the breast, using images from previously acquired MRI scans. Providing the surgeon with reliable information about the shape and the size of the tumor will enable thm to resect it more precisely, and minimize repeat surgeries.

Our project has three aims. The first was to implement technology to acquire high resolution MRI images of the patient's breast while she lies on her back. The second aim was to develop augmented reality to display these images within the patient's breast. Our third aim is to measure accuracy of the system in 10 patients with breast cancers. Overall, we have accomplished the first two aim, and enrolled four patients for aim 3.

To achieve these aims, we faced several hurdles. These include 1) respiratory motion while scanning of the patient on her back, 2) inability of the HoloLens to detect the location of the breast, 3) distortions in the HoloLens display system for virtual objects at arns-length , and 4) data appear to float above the breast. Our major accomplishments address these including good-quality breath-held supine contrast-enhanced MRI scanning., semi-automated holographic alignment using computer vision recognition of skin fiducials, task-based user calibration, and implementation of a full-featured AR system that uses a transparent skin grid, mesh rendering of segmeted tumors, and cut-away renderings to convincingly display tumors inside the breast, which the surgeon controls using gestures, voice and menus.

Going forward, we will complete our pilot study of 10 subjects. We have secured pilot funding to develop non-rigid deformation models. We have NIH grants in review to develop more techmology and study non-palpable tumors as well. Ultimatley, we envision a clinical trial to validate that AR-based surgical planning can reduce repeat surgery in a substantial fraction of the approximately 170,000 women who undergo lumpectomy each year in the US.