An experiment – with mixed success – in combining HoloLens augmented reality with touchscreen interaction.
Microsoft’s HoloLens is probably my favorite new technology device of the last decade – the first time I tried it out, it felt truly magical. It’s still the only augmented reality device I’ve used that feels seamless and “just works.”
That said, it has some limitations, one of which is a lack of intuitive interaction. The user’s instinct is to tap the holographic objects to navigate, but that’s not quite how HoloLens works. With this device, you “gaze” at an object to highlight it, then tap your finger to select it. There’s a learning curve to this, which makes it challenging to deploy the devices in exhibits and public environments.
I decided to explore the possibilities of using HoloLens with touch displays – not only would this provide familiar, tactile interaction, but it would work nicely in museums and brand environments where a physical component could provide a visual anchor for an augmented-reality experience. I set out to build a simple prototype that would give me just enough structure to experiment with the combined technology platform and to understand its strengths and limitations.
My primary challenge was to align HoloLens’ virtual content precisely with a real-world touch display. I initially tested Vuforia, probably the most common development toolkit for augmented reality. Vuforia support is built into Unity, the platform I’m using for HoloLens development, so it seemed like an easy choice.
After resolving some challenges around software version compatibility, I was able to compile Vuforia’s sample app and run it on my HoloLens. Vuforia applications use a device camera to look for printed tracking tags, which the app uses to interpolate the camera’s point of view and render 3D content on top of the scene. For my purposes, I wanted the HoloLens to track digital tag images, rather than printed ones, directly off of a small tabletop touch display.
It worked – more or less. In order to initially recognize the tracking images, I had to hold the device fairly perpendicular to the touchscreen. Once the hologram appeared, it was more flexible, but there was visible shakiness (0:25 in the video above) and drifting (0:02). Worse still, if the tracking image went out of view, the hologram flickered off. There are composite approaches that can use Vuforia to initialize positioning and switch to the HoloLens’ own tracking thereafter, but I decided to try a different approach….
For my second test, I built a custom application that would allow manual manipulation of a holographic grid, allowing me to align it precisely with a matching grid on the touch display. To keep things simple, I used voice control in lieu of navigation buttons (that may not sound simple, but it’s remarkably easy to implement for HoloLens). Through voice, I could choose whether to scale, rotate, or move the object, then use a simple hand gesture to perform the transformation.
Once I got this up and running on the device, it was pretty easy to use. Unfortunately, while the HoloLens content was smooth and stable, it still drifted. To use Microsoft’s own terminology, it was “swimming.” If I kept my head steady, the hologram was stable, but as I moved slightly, the image shifted – only a few centimeters or so, but enough to make it impossible to consistently align the grids.
I tried out some of Microsoft’s default applications to see if I would get the same results. Using the built-in Holograms app, I selected a virtual dog and placed it atop the touch display. As expected, the effect is smooth, but doesn’t stand up to scrutiny: as the video shows (at 0:28:00), the dog’s position shifts relative to the grid lines. Had the image been floating in space, without the display underneath it, I probably wouldn’t have noticed this.
I did some more research; others online faced the same issue, so I was fairly certain I wasn’t doing anything wrong. Ultimately, a combination of factors are working against me:
- The shininess of the touch display surface makes it less than ideal for HoloLens’ visual tracking.
- HoloLens is expressly meant for viewing content from a distance of 1.5 meters or greater (all of my content fell within one meter).
- HoloLens smooths all of its virtual content to minimize jitter, privileging elegance over accuracy.
- HoloLens’s tracking technology simply doesn’t support this level of precision.
It would be silly to complain about these limitations – it’s pretty miraculous that HoloLens works as well as it does in the first place. That said, it’s a two-year-old device using rapidly developing technologies, so it’s entirely possible that its next generation will overcome all this issues. But rather than wait a year or two until the next HoloLens arrives, I decided to see if I could work around its current limitations.
To finish this stage of the prototype, I built a simple application which animates a circle to follow the user’s touch point across the grid. Using a simple network connection, these touchpoints are sent wirelessly to the HoloLens application, which replicates the motion using a 3D cube.
Except for the “swimming,” it’s not bad. The cube is never quite perfectly aligned to the grid below it, but it’s consistent, and the coarseness of the grid snapping helps with perceived accuracy.
I’m tempted to try a few more things… using printed Vuforia tags in lieu of virtual ones, or adding non-reflective materials around the display edge to try to improve tracking. But neither of these workarounds would successfully scale up to a larger display.
Instead, as a next step, I’ll build out the prototype to incorporate some use cases for exhibit experiences, and see whether I can build interfaces that will work successfully in spite of the precision issue. Stay tuned….