Q&A: Quantified Surgery

Interview with Larry Smarr, Sonia Ramamoorthy and Jurgen Schulze


Larry Smarr

Larry Smarr, PhD (patient)
Harry E. Gruber Professor
Computer Science and Engineering
UC San Diego Jacobs School of Engineering
Founding director,
California Institute for Telecommunications
and Information (Calit2)


Sonia Ramamoorthy

Sonia Ramamoorthy, MD (surgeon)
Professor of Surgery
Chief, Section of Colon and Rectal Surgery
Vice –Chair Surgical Quality, NSQIP
Disease Team Leader, Colorectal Cancer
UC San Diego School of Medicine

Jurgen P. Schulze

Jurgen P. Schulze, PhD (computer scientist)
Associate Research Scientist
Qualcomm Institute, UC San Diego
Associate adjunct professor
Department of Computer Science, UC San Diego


QUESTION: What is quantified surgery?

Smarr: We're living through a digital transformation in medicine, and that means we can make use of more data about our bodies than we ever could before, including 3D versions of imaging technologies. The question is, how can we use that data to minimize the risk of surgery, to have better outcomes? That was the experiment that we set out to try to work on here.

QUESTION: What exactly was the experiment?

Smarr: We had been working at Calit2 for a number of years on taking sets of MRI or CT scan slices and turning them into “transparent Larry,” a three-dimensional model of me. In Calit2’s virtual reality facility, I've given many (virtual) tours through my body. But, until a date was set for my surgery and it sunk in that I was going to get my colon resected, it hadn't occurred to me that we could use (all of that imaging data) as a pre-surgical planning tool.

QUESTION: What happens now when most patients come to you needing surgery?

Ramamoorthy: Currently, when a patient presents to us in our clinic, they come in with a series of complaints and a question as to whether surgery will make things better. We rely on three main sources for information.

You talk to the patient, get a sense of what and where their symptoms are, if they have pain here or there, how long they've had symptoms, what makes it better or worse, etc.

Then there is the physical exam. Can we palpate something? Do we feel something in the suspect area? Is there other evidence of disease on exam? Are there other findings that might impact surgery for example body habitus or previous incisions?

The final piece of information are the diagnostic studies, laboratory studies like blood tests and X-rays that help us see within and understand what's going on internally.

For the most part, we get a lot of information from these three sets of information. Certainly our imaging studies help us understand in two dimensions what's happening inside a patient's abdomen, for example. Putting it all together we know where the pathology will be and where we have to go—and most of the time, what we have to do.

We then apply our standard surgical principles, experience and knowledge to approaching the problem. Do we really have a very personalized, detailed, three dimensional understanding of what's going on with a patient’s anatomy? No. It would be really nice to have that known before we get into the operating room.

QUESTION: So how exactly did things progress?

Ramamoorthy: Larry asked me, “How do you plan the surgery? Where are you going to go in there? What are you going to do?” And I explained to him what we do currently. And he said, “Oh my God! You know, I think I can help you. I think we can increase the amount of information you have going into this operation. I think I can help you visualize what's going on in my body to help improve and optimize the operative experience for both of us.”

As a surgeon, you bring a lot of experience and knowledge to the table. You walk into the OR and you say, “There's nothing that I can't handle.” Which it true. But, wouldn't you rather be prepared for what you're going to encounter in the moment? I mean, who goes into a football game without having studied the opponent's typical patterns and made a strategy for what they're going to do? We do that now, but if you don't know what you will encounter or are going to encounter, how can you plan properly for how you are going to approach it?

Smarr: I told Sonia, “I've got a transparent digital version of myself. We could be planning this surgery using that 3D visualization before you go inside of me." And she said, "Wow. I've never done that in 20 years. Let's go do it."

So we brought her over to Calit2’s Qualcomm Institute, literally just a few days before the operation, for pre-surgical planning. And she was able to essentially have a 3D Google Map of the inside of me, to see what the difficult points would be and thereby plan out the five hours of the surgery without having to discover the particularities of my insides, which are different in everyone, once she was inside me. She was much more able to plan the surgery, reduce the risk, have me under anesthesia for less time, all very good outcomes.

QUESTION: In some ways, wasn’t this idea of “quantified surgery” a natural extension of Larry’s earlier concept of “quantified self?” Hasn’t he been working toward this for a while, tapping into resources like Jurgen and Calit2?

Schulze: When Larry came to me with his first MRI scan a few years ago, I'd already had a lot of experience visualizing medical data, and the way that I approach this topic is that I take the MRI slices that we get from the MRI scanner and stack them up into a 3D model so that you can see the three-dimensional structure of what's in the scan.

So, when I first got the MRI that he brought me, I did exactly that, took the slices and looked at that part of his body, with the colon and everything around it, in 3D in virtual reality. And, it looked really good. It was very clear where each organ was and how everything was arranged, and from that point on, it turned into our go-to tool in order to visualize his MRI and CT scans along the way.

I took the original scan and the 3D model and I segmented out the organs that mattered for the surgery, which were the colon and the spleen, the bladder and the ureters, and the major blood vessels. It can take a long time to segment out these organs because you have to go through, slice by slice, and mark where the organs are. I combined all these cross-sections into a 3D model that was able to display on a monitor, and you were able to see reasonably well where all these organs were on a regular monitor in the operating room.

Smarr: Once I was able to see inside my own (virtual) body, I realized that we could utilize another technology which is changing very rapidly: 3D printing. I could actually get my staff to create from the MRI data a realistic 3D version of the end of the descending colon, which is connected to the sigmoid colon that was going to be surgically removed, although at the time, I didn't think surgery was in my future.

So we were able to take the normal MRI or CT scan slices, convert them into 3D graphics and then take the part that was diseased, the sigmoid colon, print it out at life scale. I could see that there was an unusual kink in it and lots of diverticula.

Once I realized that I was actually going to go through surgery, I thought to myself, “Well, as long as I'm going to have this done, I'm blessed to have all this talented staff, all kinds of computers and visualization that the ordinary person doesn't have. Why don't I use my surgery as a translational pilot project to see if using the 3D visualization actually does make a positive difference for the surgeon?”

It turned out to be a good experiment because Sonia was able to show that these new digital techniques were quite helpful in the surgery. Now, the beautiful thing with digital is that the technology is exponentially decreasing in cost over time, just like your computer or your smartphone get faster and faster for the same money, so we can imagine this becoming routine in a few years.

QUESTION: Still, in some ways this doesn’t seem like a huge technological leap. We already see pretty detailed imaging of, say, the brain or heart. What’s different here?

Ramamoorthy: Imaging technology has really progressed in other fields, like cardiovascular or brain surgery. That’s because, in part, we’re looking at very fixed organs with a very homogeneous texture to them. There is blood running through vessels, but vessels aren't moving around. And you're looking at a single structure. In contrast, in the abdomen, you have a liver, kidneys, the bowels. The small and large intestines are constantly moving through peristalsis, the bowels are hollow organs with variable amounts of liquid, solid and air, and the whole abdominal cavity moves with each breath.

All of those things need to be registered in imaging so that you can actually get a good sense of what's going on, and then reconstruct it three dimensionally. Which sounds easy enough, but it's very complicated. And what we need in surgery is that three dimensional view, but knowing that, well, these organs might move. Some of these organs might be moved a little bit by the time I get in there and operate. They may even get distorted a little bit of what I'm doing operatively. So how can I plan my surgical moves with purpose and efficiency? What's my no-fly zone? What do I want to avoid getting into and how do I get around any problems ie.fly around the storm to get to my destination without heading straight into it.

What we really wanted to understand was just the three dimensionality of that structure (Smarr’s intestinal kink). It was located in the area that I call the high rent district. A lot of important structures are there that we did not want to harm in any way.

But with the amount of inflammation he had, it's like swimming in muddy water. So you want to try to clear things up as much as you can to see what you need to see, and that's what the 3D imaging did for me. I was able to look above and below and realize there were a few other areas that would be challenging. Could we get more information about those? How much of a resection would I have to do? What would be the appropriate cut point on the top end? What would be the cut point on the bottom end? That's an important consideration when you're talking about patient's organs.

If we had done this case a year ago, we would simply have relied on CT scans in two dimensions, plus our lab tests, our physical exam, our history of what was going on. We would have put a scope in to take a look and get a sense of the architecture, then put in our robotic instruments as we do for everybody. A standard approach.

I think we would have encountered some problems and we would have been “Oh, this is here and that’s going on there. We didn't necessarily anticipate that.” Do we manage these kinds of things? Absolutely. But does it make the process longer? Does it require a little bit more dissection? Maybe a little bit more trauma to the tissue? Perhaps. That's what would have happened a year ago. I think Larry would have been fine, but it would have been a different operation.

QUESTION: What was it like during the procedure being able to look at both the real and virtual patient at the same time, but maybe see different things?

Ramamoorthy: (During the procedure), there were times I would look at my 3D model and tell my team to adjust the angle of the model to mimic what I'm seeing intra-op so that, as I keep moving through an area (of the abdomen) they see what I see. It was incredibly helpful.

A whole other aspect was teaching the residents and the nurses in the room. I could say, “You see this, guys? You understand what we are doing here? Do you see this plane? This is where you want to be. This is how we want to approach this.”

It's all the stuff that we already do in surgery, but it was integrated in a way that I hadn’t experienced before. Because Jurgen had taken me through the “virtual Larry” I had already visualized the process of his surgery. My brain was already working on it. So, when I got in there, which is different than what we do today, when I encountered things I had had time to think about them before.

One of the unique things we had in that operating room that day was our chief of gastroenterology, Dr. (William) Sandborn, and our chief of minimally invasive surgery, Dr. (Santiago) Horgan, who were in there helping us with some of integrated information. But in reality, we wouldn't necessarily need them physically in the operating room. There's a way for us in this day and age to bring them into the operating room remotely and virtually to get them to see what we are seeing, to get their input.

QUESTION: Jurgen, you were in the operating room that day, assisting with the imaging. What was the experience like for you?

Schulze: As a computer scientist, you're used to dealing with data, and you're looking at it in a way that's very abstract. When it comes to this work, the big difference to me was that it wasn't just data, but it was data that mattered for the surgery of Larry Smarr, who runs the center that I work for. And he was on the operating table. And my data would matter for his surgery. So, that was a whole different way of approaching the importance of the work that I did—and also the potential consequences.

QUESTION: A big challenge, though, is practicality. Larry had access to facilities, resources and expertise at UC San Diego Health and on campus that aren’t broadly available. What are the prospects of that happening, of more typical patients being able to follow his example?

Ramamoorthy: It will take some time. I think one of the things we realized is that integration of all the software into the operating room still needs more work, which is what we are going to be doing here with our engineering department. We really want to try to streamline it so that (3D modeling for surgeries) can be done very easily. We need to train our OR staff into how to bring all of this information together.

Then there's a whole other side that’s about brain integration. You're seeing three or four different views of the same thing: a 3D model, the actual patient in the operating room from the outside, a robotic view from the inside and an endolumenal picture (inside the bowel itself). Just processing all that in your head simultaneously as a surgeon so you can decide your next move is a skill set that requires training. We have to get all of the technology into the OR so that we can start getting use to processing and integrating that information so it can impact our operative plan in real time.

Schulze: We're hoping that we can use this technique for a lot more surgeries, really making it commonplace in the operating room. What it takes to make that happen is that we're going to have to make the process easier and faster. That's possible, though, because no part in the process requires a person to sit there and do all the work. A lot of it can be done by the computer if the computer just knows how to do it. There's also machine learning these days that's been very successfully applied in many disciplines related to what we're doing in this project. So, the hope is that we can combine all these technologies in order to make it possible for a surgeon to use this technique on an everyday basis—ideally with every surgery where it makes sense—and make it simple enough to use for surgeons so they can just push a button, basically, and bring it up in the operating room to help with their surgery.

Smarr: The great thing about our economy these days is that the consumer world is driving the computing world, which means we can now use these new technologies in specialized areas like surgery or radiology. If it were left to surgery or radiology alone, they would not have the money to drive this technological development. So that's one of the things that I specialize in, in understanding what changes in technology, driven by the mass market, are now going to be applicable to specialized areas.

I see this as more and more technology is being brought into the operating room to reduce risk, to shorten the operating time, and to have a better outcome for the patient. Now that we know this is valuable in surgery, what we're going to do is simplify the software, make a much better user interface, and then train up a number of our surgeons in the use of it, and get another 10 patients to refine the procedure, then 100 patients and so on…

QUESTION: Finally, how’s the patient doing?

Ramamoorthy: He's doing great. Larry did really well post-operatively. He was up and moving around very quickly. He wears a fitness monitor so we know he was on the move. He was out of the hospital and back to his quality of life in no time flat.

Smarr: Like many people that have an abdominal problem, before the surgery I was getting worse and worse. It was ruining my life. Now, I feel like a new man. I feel like I'm 20 years younger. I love my new colon! It works great, and I just think this is a fabulous outcome, and I'm so happy we could do it at UC San Diego.