Technologies for Augmented Reality Breast Surgery

Institution: Stanford University
Investigator(s): Bruce  Daniel , M.D. -
Award Cycle: 2016 (Cycle 22) Grant #: 22IB-0006 Award: $235,100
Award Type: IDEA
Research Priorities
Detection, Prognosis and Treatment>Innovative Treatment Modalities: search for a cure



Initial Award Abstract (2016)

Non-technical overview of the research topic and relevance to breast cancer: Augmented Reality is a novel technology in which computer-generated virtual objects appear projected onto real objects in the world around us. Although primarily being developed for gaming and industrial design applications, Augmented Reality has great promise to improve medical care. One of the main challenges for breast cancer surgeons is to remove early stage breast cancers without leaving any residual tumor in the breast. MRI can reveal breast cancer accurately. The goal of this research is to develop Augmented Reality methods to accurately display MRI on the patient’s actual breast, in 3 dimensions, interactively right before surgery, to help surgeon plan a more successful operation.

The question(s) or central hypotheses of the research: The central hypothesis of this research is that fusion of MRI with the actual breast will enable surgeons to avoid leaving residual non-palpable tumor behind during lumpectomy, and thus reduce the number of repeat surgeries prompted by incomplete resection, which currently occurs in about 25% of cases. The specific question considered in this proposal is: how accurately can previously acquired MRI data be aligned and depicted on the breast using augmented reality..

The general methodology: We propose two areas of research: technology development, and clinical testing. The objective of the technology development is to create an accurate, robust system for projecting the location, extent, and appearance of breast cancer and its surrounding tissues onto the patient’s actual breast. This will entail developing new methods to make MRI pictures of the breast when the patient is lying “supine”, on her back. It will also require methods to align those images to views of the actual breast being presented by the Augmented Reality device. The objective of the Clinical Testing is to see how accurately the system is showing the location and extent of breast cancer in the breast. Since this is the first test of the system in women with breast cancer, it will be done in women who don’t need it, specifically women with tumors that can be easily felt and removed by the surgeon without any images. This will pave the way for future tests in larger groups of cancer patients to see if it can reduce the number of repeat surgeries that are done because of concern for tumor left behind in the breast. .

Innovative elements of the project and potential impact: Supine MRI would improve the use of MRI for surgical planning and provide a more comfortable option for women getting breast MRI’s The Augmented Reality display of MRI on the actual breast is entirely novel If it is successful in enabling more surgeries that don’t leave residual tumor, then it could eliminate thousands of repeat surgeries in the US annually.




Final Report (2018)

This project pursued technologies to assist surgeons planning breast lumpectomy. Lumpectomy is the most common cancer surgery, and yet in a significant number of cases initial lumpectomy is not completely successful. Both over-excision, in which excessive normal tissue is removed, and under-excision, in which tumor is left behind, occur too frequently because it is difficult for surgeons to precisely know the extent of tumor in the breast. Our proposed approach to this problem is to project highly accurate pre-operative 3D breast MRI data onto the patient at the start of surgery. The project had 3 aims. The first aim, to develop high quality supine breast MRI which closely approximates the “surgical” position, was accomplished. The second aim, to develop an APP for the Microsoft HoloLens to display images on the body, was accomplished. The third aim, to test the system in 10 women with palpable tumors was nearly accomplished, and remains the major success of this project. The system has been tested for 8 breast cancers in 7 women (one had bilateral cancer) and accrual is continuing to complete the pilot study even after the end of the funding period. Augmented reality improved the ability of surgeons to estimate the size of tumors in the breast. Two hurdles remain. One is technical. The precise alignment of the virtual MRI data to the actual breast remains imperfect. Potential sources of variation include anisotropy in the accuracy of computer vision-based alignment, and possible bulk deformation of breast tissue by gravity, etc. The other hurdle is subject accrual. Women with palpable tumors are uncommon, and some are not interested in participating in research-related activities that do not directly advance their care. Nevertheless, we still believe completing the pilot study of 10 patients will happen in the next few months. Moving forward, we anticipate that tracking the actual tumor location during surgery (rather than extrapolating from fiducials around the breast) will be the most robust solution. Thanks to the preliminary data provided by the CBCRP IDEA award, we were successful in obtaining a new “Stanford Women’s Cancer Center” 1-year grant which we will use to study tumor tracking technologies as part of a new collaboration with Professor John Pauly and his lab in electrical engineering. We are also using this data to apply for NIH RO1 level funding of this aspect of the project. The activities supported by the CBCRP did not overlap with other areas of funded research in our laboratory.