Razorfish + Emerging Experiences

We recently had the opportunity to debut the RockstAR experience at SXSW – check out video of the experience in action. We like to think of it as the classic photo booth taken to the next level with augmented reality, multi-touch and social integration. Let’s go behind-the-scenes and take a look at both the software and hardware that brings this experience to life.

First, let’s talk software. The application was built on the recently announced Razorfish Vision Framework. The framework provides a platform to power augmented reality, gestural and other vision-based experiences. For the RockstAR experience, we are analyzing each frame coming from an infrared camera to determine if faces are found in the crowd. Once a face is detected, it is assigned a unique ID and tracked. Once we receive a lock on the face, we can pass position and size information to the experience where we can augment animations and graphics on top of the color camera feed. This technology has a variety of uses. For instance, face tracking can be used to track impressions on static or interactive digital experiences in the retail environment. Here is a screenshot taken from the debug-mode of the experience which shows the face tracking engine at work using the infrared camera.

In addition to the vision-based technology, the experience was fully multi-touch enabled – users can gesture on a virtual joystick to swap out bands and snap pictures.

Because the classic photo booth experience is a social activity, we took it to the next level with twitter and Flickr integration. As pictures were snapped, we’d immediately make them available online. A QR code was rendered with each picture to quickly allow users to navigate to the RockstAR photo on their mobile device. Once the experience is extended to mobile, users can email the pictures to their friends, set it as wallpaper, re-tweet it to their twitter followers, etc.

Let’s move on to hardware. Unfortunately, you can’t purchase infrared AR-ready cameras at your local Walmart… at least not until Project Natal comes out later this year. Therefore, we needed to build a dual-camera system that would support the face tracking in infrared and the color video feed for display on the screen. We decided to go with 2 commercial-grade Firefly MV cameras with custom lenses.

One of the cameras we modified to see only infrared light by replacing the IR-blocking filter with a IR band-pass filter. This allows only a narrow range of infrared light to reach the camera CCD.

We also purchased and tested a variety of infrared illuminators. These are used to illuminate the environment with invisible infrared light allowing the infrared camera to accurately track faces in low-light conditions.

Sparks were flying as we fused the color and infrared cameras together — just another day at the office.

We created a portable rig for the camera and infrared illuminators. Adjustable camera mounts and industrial strength velcro provide flexibility and portability across a variety of installations.

We used a presentation remote clicker as an alternative way to drive the experience. We primarily used it as a remote camera trigger which allowed us to quickly snap pictures of unsuspecting people from a distance.

The experience was powered by a 55″ multi-touch screen and a CPU provided by DFI Technologies. We’ve been working with DFI to build PCs that will power the next-generation of interactive experiences. These PCs are small form factor and can be mounted behind the multi-touch screen.

Last but not least, we bring you the pink rug. We can’t reveal too much information about this technology… we need to keep some things secret. Just know that it is critical to the overall experience.

We’ve been very busy since the Windows Phone 7 Series experience launch in Spain, and there’s plenty more excitement over the next couple of weeks. We will be announcing the Razorfish Vision Framework at this year’s SxSW Interactive Conference. We have created a prototype application that is a mixture of Augmented Reality and Multi-touch called RockstAR and will be showcasing it at the Razorfish hosted cocktail party on Saturday the 13th at the Paradise bar. Come by and augment your reality!

Also …

Below is a quick list of the conferences we’re at in March. Please let us know if you are also attending and we’ll make sure you get a chance to check out our experiences.

• GDC 2010 (March 9-13, SanFran) – Lesley just returned from San Francisco, CA where we helped Microsoft showcase the WP7S experience in their Game Developers Conference booth.
• SxSW 2010 Interactive (March 12-15, Austin) – We’re hosting a panel on the 12th, please come interact with us as we discuss “Touch + The Holy Grail of Delight” at 2pm http://my.sxsw.com/events/event/6124. We are also launching a fun new AR experience for the Razorfish cocktail party. Finally we will be at the Microsoft Speakeasy event showcasing the WP7S experience.
• MIX10 (March 15-17, Vegas) – @stevedawson and @hulljon will be attending the conference. The WP7S experience will also be making an appearance … with updated content!
• CTIA (March 22-25, Vegas) – WP7S experience will be out in force at this international wireless conference.

A quick thanks to all the peeps that helped out with RockstAR:

• Steve – @stevedawson
• Luke – @lhamilton
• Alex – @alexn
• Dung – @dungtienle
• Sara – @saragiessen
• Heather – @heathergately
• Jonathan – @hulljon

As part of our rapid prototyping process we will sometimes shoot quick little videos to share with our clients – its easier than hauling equipment all over the country . This video was shot on one of our 55″ multi-touch displays for the Windows Phone team just before heading over to Mobile World Congress in Barcelona.

The Microsoft Surface technology offers such great possibilities. Consider how we solved one of man’s greatest challenges – how to watch multiple NCAA March Madness games simultaneously. These are the wonders of what you can create while working at the office late. Notice how the interface allows us to easily switch between live feeds as the games are in progress. This is just another example of how Microsoft Surface technology can deliver an innovative and engaging customer experience.

A childhood favorite – Table Toss – is transformed using Microsoft Surface technology. Like the old fashioned game, Surface Table Toss involves players taking turns at tossing a bean bag towards a target. However, in this version, the target is a bulls-eye created on a very expensive table (Note: No Surface tables were injured in the making of this game). Scoring is determined by the distance of each bean bag from the center bulls-eye. Each bean bag contains a unique Microsoft Surface tag to associate the score to each player. The combination of a baseball theme and photo-realistic particle effects creates a one-of-a-kind experience on Surface. While this game is fun for the whole family, we do not recommend throwing other items at the table, even if you lose.

Ever wonder what your co-worker three cubes over did last weekend? Our friends at Amnesia Razorfish in Australia made it easy to find out with a Microsoft Surface staff directory application. The application uses Surface tags printed on all Amnesia Razorfish business cards to show extra information about employees such as blog posts, tweets and Flickr photos. Recommended for business purposes only, not for spying on your coworkers.