First, let’s talk software. The application was built on the recently announced Razorfish Vision Framework. The framework provides a platform to power augmented reality, gestural and other vision-based experiences. For the RockstAR experience, we are analyzing each frame coming from an infrared camera to determine if faces are found in the crowd. Once a face is detected, it is assigned a unique ID and tracked. Once we receive a lock on the face, we can pass position and size information to the experience where we can augment animations and graphics on top of the color camera feed. This technology has a variety of uses. For instance, face tracking can be used to track impressions on static or interactive digital experiences in the retail environment. Here is a screenshot taken from the debug-mode of the experience which shows the face tracking engine at work using the infrared camera.

In addition to the vision-based technology, the experience was fully multi-touch enabled – users can gesture on a virtual joystick to swap out bands and snap pictures.

Because the classic photo booth experience is a social activity, we took it to the next level with twitter and Flickr integration. As pictures were snapped, we’d immediately make them available online. A QR code was rendered with each picture to quickly allow users to navigate to the RockstAR photo on their mobile device. Once the experience is extended to mobile, users can email the pictures to their friends, set it as wallpaper, re-tweet it to their twitter followers, etc.

Let’s move on to hardware. Unfortunately, you can’t purchase infrared AR-ready cameras at your local Walmart… at least not until Project Natal comes out later this year. Therefore, we needed to build a dual-camera system that would support the face tracking in infrared and the color video feed for display on the screen. We decided to go with 2 commercial-grade Firefly MV cameras with custom lenses.

One of the cameras we modified to see only infrared light by replacing the IR-blocking filter with a IR band-pass filter. This allows only a narrow range of infrared light to reach the camera CCD.

We also purchased and tested a variety of infrared illuminators. These are used to illuminate the environment with invisible infrared light allowing the infrared camera to accurately track faces in low-light conditions.

Sparks were flying as we fused the color and infrared cameras together — just another day at the office.

We created a portable rig for the camera and infrared illuminators. Adjustable camera mounts and industrial strength velcro provide flexibility and portability across a variety of installations.

We used a presentation remote clicker as an alternative way to drive the experience. We primarily used it as a remote camera trigger which allowed us to quickly snap pictures of unsuspecting people from a distance.

The experience was powered by a 55″ multi-touch screen and a CPU provided by DFI Technologies. We’ve been working with DFI to build PCs that will power the next-generation of interactive experiences. These PCs are small form factor and can be mounted behind the multi-touch screen.

Last but not least, we bring you the pink rug. We can’t reveal too much information about this technology… we need to keep some things secret. Just know that it is critical to the overall experience.

We recently created an experience named RockstAR which features augmented reality and multi-touch technology. It is the classic photo booth experience taken to the next level with interactive technology, social integration (currently the experience posts to twitter, twitpic and flickr), good ole fashioned Rock n Roll and a little 80s video game nostalgia. We also can’t leave out the pink rug – one of the most important parts of any experience.

The application is the first demonstration of the Razorfish Vision Framework (RVF) and it is integrated with our Razorfish Touch Framework (RTF). The experience was featured at several SXSW Interactive Conference 2010 events including the Razorfish and Microsoft parties.

Stay tuned as we’ll be posting a behind-the-scenes tech walkthrough in the next week.

At the 2008 Razorfish Client Summit, the Seattle office created a social experience that allowed attendees to place their badge on Surface to enter a raffle for a free XBOX and bid on guitar signed by the “fifth Beetle” George Martin. In addition, attendees could form connections between eachother using the Surface table – as connections were made, behind-the-scenes we would e-mail each attendee the other person’s contact information. The client summit experience was extended for the Razorfish Atlanta Open House. Badges were tagged and guests congregated around the table to exchange information and connect with people that shared similar interests.

Recently our friends in the Neue Digitale / Razorfish in Germany created a social application for the Microsoft Xtopia conference named “Event Connect” that allowed attendees to access their Facebook account on Surface. Users could exchange photos of the event and send friend requests to eachother within the experience. This same experience was recently re-skinned by the Austin office and featured at the 2009 Razorfish SXSW party.

The Amnesia Razorfish office in Australia printed Surface tags on their business cards . When placed on Surface, these cards can access a variety of social feeds, including blog posts, Twitter updates and photos from Flickr. Check out video of the application here.

Ever wonder what your co-worker three cubes over did last weekend? Our friends at Amnesia Razorfish in Australia made it easy to find out with a Microsoft Surface staff directory application. The application uses Surface tags printed on all Amnesia Razorfish business cards to show extra information about employees such as blog posts, tweets and Flickr photos. Recommended for business purposes only, not for spying on your coworkers.