At Robots and Pencils’ head office, you’ll notice something a little different when you come in. Our digital signage may seem “ordinary” but when you walk past it, you may feel as though you’re being watched…
“Mona Lisa” is a subtly fun way to welcome visitors to our office. It uses new technology to take what you’d normally expect, digital signage in a tech office, and turn it into something a little different. For me, Mona Lisa was an experiment that started with a lot of different approaches and it took some time to narrow them down to find what worked best for the situation I was working with.
MAKING IT
I’ll start off with the scenario: We wanted to install a display in our office entrance to greet visitors when they arrived for their meetings. We wanted it to look like a piece of art hanging on the wall, but with a twist: the eyes would follow your movement throughtout the room. You’ve probably seen similar effect before but without any fancy technology. We had a few different options available in order to pull this off on a television screen and they can be categorized two ways: image analysis and depth mapping.
The image analysis method uses algorithms to make a good guess as to where a face is in a picture. You’ve probably noticed that your iPhone will use this to find faces in a photo and focus on them. This method is simpler because it only requires a regular camera, but the algorithms only work well when the lighting and contrast of the image is ideal. The depth mapping method is best known by the Xbox Kinect hardware. Here a special camera tracks how far away millions of points are in the room, and builds a sort of topographic map based on that. This method can be more accurate than plain image analysis because it can see in a third dimension but requires special hardware at an extra cost.
IMAGE ANALYSIS
There are two different ways that we could implement image analysis. There’s the popular OpenCV software that is used often for these sorts of projects, but we tried the built-in frameworks that Apple makes available on OS X and iOS. This was really the easiest way of the ones I tried. Given some guesses at average face size it was even possible to get the approximate distance from the face to the camera.
I’ll note that although OS X often has some more advanced features, iOS has definitely gotten the love here. It has some APIs that OS X doesn’t have that would have been helpful, but thankfully it wasn’t that hard to port from one to the other.
The drawbacks of image analysis became instantly apparent as soon as I started testing it away from my desk. Anytime the room was backlit (meaning light from a fixture or window was shining towards the camera lens) it proved much less likely to find a face, and this wouldn’t work at all in the well-lit entrance to our office.
This forced me to try out the second method.
DEPTH MAPPING
We ordered a Kinect camera and I got started figuring out how to get it hooked up to the Mac. There’s no official SDK for the Mac (only for Windows), however lots of smart people have stepped in to fill this void with OpenNI and CocoaOpenNI. It took some fiddling to get everything up and running, but once I started getting information into my app it was smooth sailing.
The best part of depth mapping was that it works so much better in over-lit or backlit rooms. It also provides more accurate information about the distance from the camera than the calculations I could make with image analysis.
THE FACE
You might be wondering why this project was called Mona Lisa to begin with. We originally used an image of Da Vinci’s Mona Lisa, but after having some demos up on the wall for a while we realized that the art theme didn’t fit with our plans to display more information.
I started out just moving the two eye images in front of the larger image to simulate the tracking, but it wasn’t quite convincing enough. I decided to try it out with the SceneKit framework on OS X in order to render 3D eyes that would rotate instead! The effect actually turned out to be very subtle but helped add to the realism, and I think it’s an important detail to make it more convincing.
THE WELCOME
Ultimately Mona Lisa is about welcoming our guests, so it can also show the visitors’ names, meeting times and room when they arrive. It’s a little touch. In the future we’re looking to tie the visitor information to our office meeting calendar and other ways to showcase the technology we know so well.
- Brandon Evans, Robot.
