Touching Neurons with Augmented Reality (Part I)

Back in the late 17th century, Antony Van Leeuwenhoek became the first man to design and use a microscope. It was handheld and had only one single lens, but it laid the groundwork for the future of science. He went on to better the design, developing better lenses than his contemporaries who were working to achieve the same goals, and reached a magnification of 270x, which was unheard of for the time. With that sort of magnification, he was able to see bacteria, yeast, blood cells, and lots of tiny organisms swimming around in just a drop of water. He discovered a whole new world full of living things smaller than anything anyone at the time could have imagined, and without this discovery, the healthcare field would not look anything like it does today. 

You don’t have to have a vast medical knowledge to know just how impactful the microscope was. From microbiology to microsurgery, it brought a new understanding to the human body and the things that make it sick. Today, technologies such as Confocal Laser Scanning Microscopy and Scanning Electron Microscopy offer a whole new spectrum of possibilities, far beyond even what Mr. Leeuwenhoek could have even dreamed. These two technologies are far too complex to talk about in just one article, which is why there will be a part two to this story, and even with those, we will only be able to scratch the surface of these imaging systems–specifically, we’ll be focusing on how augmented reality can have a place in them.

Confocal Laser Scanning Microscopy (CLSM) is an optical system that  uses a spatial pinhole, which is placed at the confocal plane of the lens, to eliminate out-of-focus light. It provides a non-invasive optical sectioning of intact tissue in vivo–something that isn’t possible with traditional microscopes. The images are high resolution 3D reconstructions of the tissue, created by compiling sets of images obtained at different angles and depths. It relies on light to work, however, which ultimately limits its usefulness in studying more complex structures. Scanning Electron Microscopy (SEM) is a very useful tool in microbiology and several other fields, providing greater power and depth than the CLSM by producing images of the sample by scanning it with a focused beam of electrons. However, the problem with SEM is that it cannot be used in-vivo, which its application in the healthcare field, including both the practice of medicine and the training of med students.

Taking all this into account, we could assert that CLSM is better suited for visualization of general structures in vivo, and SEM is superior when a higher magnification is needed, such as in microbiology and cell study. However, augmented reality could very well expand upon the best qualities of both these technologies, and this is what a study performed by Riddle, Wasser, and McCarthy as published by the The Journal of Biocommunication explored.

How is this possible? First, the researchers obtained a basic 3D rendering from CT scan and MRI data, known  as the Zstack, in which images were taken at multiple different levels of focus, or at different “cuts,” which can be “stacked” to achieve depth from which they could create a 3D build from the 2D images. Once the Zstack images were created, several graphic engines (like UNTIY) were used to create that 3D build of the object–in this study, the object was a CLSM image of a neuron. That 3D image can then be translated into the Google Glass device, or even just a tablet, so that the user can examine and study it with microscopic fidelity, adjusting it at will. This is a great first step into microscopic augmented reality, and you can watch the process in the video below.

We at ARinMED are truly excited about this technology. It brings so many possibilities into the field of medicine and medical research: so many, in fact, that we’re not done talking about it. Stay tuned for a second part to this article, where we’ll discuss the rest of this amazing tech and how it might change the research landscape forever. Let us know what you think in the comments section below!