Norsk
Photo: Hanne Kristine Fjellheim, SopraSteria
Microsoft launched the HoloLens glasses in 2016. These are far from ordinary glasses, in that they are in fact a wireless computer that enables the user to see an ‘extended reality’. This could be a hologram of the liver shown in the right position on the patient’s body, or another organ that appears to hover in the air, and that can be turned and rotated or taken to pieces and observed from the inside out.
These glasses currently play a major, but not the main, role in nine projects in Inven2’s portfolio. Some of the projects involve developing better tools for surgeons who operate cancer patients on a daily basis or who perform complicated operations, from cardiovascular to hip joint surgery. A number of skilled professionals play the main roles in these projects.
One of these is Ola Wiig, an orthopaedist at the Oslo University Hospital Rikshospitalet. He explains how the HoloLens technology can give the surgeons a kind of virtual X-ray vision:
“If we manage to do this, it will be the greatest innovation in the field of medicine since X-radiation was discovered back in 1895. And Norway is at the international forefront in terms of developing the use of this technology,” says Wiig.
Among other things, the HoloLens technology makes it possible for surgeons to see three-dimensional holograms of the individual organs of the patient they are going to operate. Before complicated operations, a number of surgeons can plan the operation together by looking at the same hologram, thereby achieving an entirely different shared spatial understanding of what they are going to do, and where to place the incision, than by looking at two-dimensional CT images.
But how did we get here?
From endoscopic surgery to holograms
Let’s journey back in time, to 1996. The Intervention Centre at Rikshospitalet in Oslo is established and will contribute to the development of new treatments and treatment strategies. Surgeon Bjørn Edwin is another of the main players here. He needs better operating tools for removing cancerous tumours in the liver. Since we only have one liver, it is important not to remove too much healthy tissue around the tumour, in case it becomes necessary to operate again to remove a new tumour. At the same time, it is important to remove enough so that no cancer cells remain that can form a new tumour.
At this time, endoscopic surgery was in its initial stages and still very experimental. How were the surgeons supposed to see what they were doing inside the body without opening it up, and be sure that they did not cut vital blood vessels that would lead to major blood loss from the patient?
At this time, Ole Jakob Elle, another of the main players, started at the Intervention Centre. He has a background in robotics and navigation and, together with Edwin and other colleagues, develops increasingly better models of the body’s organs with particular focus on the liver.
“The purpose has always been to create an anatomical map of the individual patient’s insides that is as complete as possible, but it must be completely precise or things could go terribly wrong,” explains Elle. He is head of the Medical Robotics, Visualisation and Navigation group at the Intervention Centre.
In 2004–2008, the Intervention Centre in cooperation with other parties is granted the prestigious EU project ARISER, which stands for ‘Augmented Reality in Surgery’. In the project, the scientists experiment with several different head-mounted screens that show 3D models of the liver, but they are not good enough.
At this point, the work stagnates for a period until it is given a boost through several rounds of funding from the South-Eastern Norway Regional Health Authority and more EU projects. One of these is HiPerNav, or ‘High Performance soft-tissue Navigation’, which is an ongoing EU project coordinated by the Intervention Centre.
“Just before the HoloLens glasses became a reality, we had already developed a 3D model for use in liver surgery,” explains Elle.
When his developers become aware of the glasses, they are unable to get hold of them, as they were not yet on the market. The time of writing is now 2015. In 2016, Sopra Steria contacts them. As a close partner of Microsoft, they have now gained access to the new HoloLens glasses and wish to cooperate with the Intervention Centre. A new chapter starts and things start moving quickly.
International innovation award
Sopra Steria are technology consultants and among the first in Europe to be given the HoloLens glasses to use.
“We contacted the Innovation Centre after Norway Health Tech tipped us off that they had been working on the technology for some time. Just 48 hours after this, we were under way with the first project on liver cancer,” says John Berland, Innovation Director of Sopra Steria Scandinavia.
The project progresses very smoothly. The developers from Sopra Steria as good as move into the Intervention Centre and work closely with the clinicians and technologists there. Shortly afterwards, they present the liver project at a global conference in the USA. Here, the interdisciplinary team comprising members from both the private and public sector receive an innovation award, major international attention and praise from Microsoft’s chief medical officer from the stage.
In 2018, just two years afterwards, Inven2 has as many as nine ongoing projects that use HoloLens in different ways in medical treatment. A number of these projects collaborate under the umbrella ‘HoloCare’.
HoloCare is the world’s first innovation centre for the use of “Mixed Reality”, which is a combination of the virtual and real world, in the health field. It was established at the Intervention Centre.
Can see children’s hearts before surgery
Meet another of the main players in the development of HoloLens use: Henrik Brun. Henrik Brun is a paediatric cardiologist who has specialised in planning operations of children with serious heart conditions.
“We have worked with 3D modelling for a long time. Children who need heart surgery often have such serious heart conditions that deciding where to start is a complex matter. When we use HoloLens in the surgery planning process, we can almost go into the heart together with the surgeons who are going to operate, and point out, explain and reach a proper common understanding of what we need to do to optimise the heart’s functioning,” says Brun.
He says that the start-up phase in the use of HoloLens went very smoothly, since they were already familiar with 3D modelling and 3D printing.
“The use of holograms has revolutionised the use of 3D in the planning phase. The next step for us is to use this during surgery and not just for planning, so it’s going to be exciting,” says Brun.
He has presented the technology at several international heart congresses where he has received much attention.
“Many are amazed when they see what we are doing, even experts in the fields of imaging, i.e. advanced and digital use of images in medicine, so this shows that Norway really is at the forefront here,” says Brun.
He is clear that the technology will make cardiovascular surgery better and safer for the simple reason that holograms are closer to reality than an image you see on a screen or on paper.
“The holograms leave less to the imagination and linguistic reproduction of pathology. Everyone involved sees the same thing and can be sure that there is a common understanding of what the organic heart disease looks like and what should be done,” says Brun.
Must utilise the health industry
John Berland of Sopra Steria believes that Norway must commercially exploit the advantage we have from the HoloLens technology.
“We have an advantage now over the rest of the world, and we should be smart and utilise it both commercially and in terms of research. This has been a team effort among interdisciplinary professional groups since the start and it has enormous potential,” says Berland.
Elin Melby is responsible for technology projects at Inven2. She is the project manager for all commercialisation projects that involve HoloLens at Oslo University Hospital and Akershus University Hospital, and is in complete agreement with Berland in relation to commercial exploitation.
“We are currently looking into the possibility of starting a company that encompasses all applications developed by the researchers and clinics relating to HoloLens. There is great potential here. It is exciting that Norway is so far ahead in an area that can revolutionise surgical practice,” says Melby. She is closely cooperating on this work with Bjarne Tvete, who is a business developer at Inven2.
The HoloLens projects have received significant funding from the South-Eastern Norway Regional Health Authority and Oslo University Hospital, which has played a vital role in developing the use of the technology. The projects are also expanding from Norway and applying for international funding and EU partners.
Ole Jakob Elle is delighted to see how vastly the use of HoloLens has grown in just a few years, and supports Berland and Melby in their view that Norway now has a unique advantage.
“The scope of use we and others are now investigating is big. HoloLens is being tested for use for everything from wound healing at Sunnaas Hospital and colon cancer at Ahus to the treatment of people with severe mental health conditions at Ullevål Hospital. As of now, I do not know of any other technology that provides this type of visualisation and depth vision, and it will be exciting to be a part of the leap from use in the planning phase to use during surgery,” says Elle.
Revolutionising surgery
Back to Ola Wiig. Wiig is an orthopaedist at Rikshospitalet University Hospital and primarily works with children’s joints and hips. He is working intensely on developing HoloLens for use on this group of patients.
“The HoloLens technology is particularly well-suited to skeletal surgery and I believe it will be a simple matter to develop it for this purpose,” he says.
He believes he will be able to start using HoloLens in hip operations already in a year’s time, but they are dependent on funding to develop a number of software applications that are necessary for optimal use.
He has previously used 3D printing of patients’ skeletons, but constantly repeats that HoloLens will revolutionise the way in which surgeons work and will lead to major ripple effects.
“They will make fewer mistakes, the results will be better and the length of patients’ hospital stays can be significantly reduced. For the surgeons, it’s like having virtual X-ray vision, which will be the greatest innovation in the field of medicine since X-radiation was discovered back in 1895. This is big. There’s no doubt about that. And Norway is at the international forefront in terms of developing the use of this technology,” says Wiig.
Read more about Holocare: https://holocare.org/