Fresh out of R&D from the Razorfish Emerging Experiences team is a product code-named “5D”. 5D started out as an idea to re-invent personal shopping. Our goal was to create a retail experience platform for both consumers and sales associates that enables multi-channel sales through immersive and connected digital devices in retail environments. And the only way to do it is to seamlessly integrate five key components – devices, content, experiences, analytics and CRM with a touch of digital magic!

The team announced 5D at the 2012 NRF Convention & Expo in New York City in partnership with NEC and Microsoft. Leveraging Windows Embedded, Microsoft Surface, MS Tag, Windows Phone and Kinect for Windows we created a prototype around a fictitious brand “Razorfashion” that demonstrates how various touch points along the customer journey can attract consumers into the store, drive product engagement and arm store associates with more contextualized digital tools.

You can read the full press release here

When it comes to travel, people care more about where they’re going than how they’re getting there. Delta Airlines understands this and asked Razorfish’s Emerging Experiences team to create an engaging experience for the WIRED Holiday Store in NYC. We wanted to tap into users’ imagination and sense of playfulness so that they walk away from the experience thinking about what kind of destinations they want to go to next and, of course, Delta.

In four (4) weeks we concepted, designed, developed and launched The Untravel Idea – a new, personal way for leisure travelers to encounter destinations. We wanted users to touch the future of travel.

The experience gives people an open-ended, creative experience that puts the user in control. First users choose the type mood they are looking for on their next getaway. From there users can select from a wide variety of relevant words that match their mood, and when put together, show them a range of destination possibilities. The result is a beautiful montage of photographic imagery that will transport the user’s imagination.

To extend the experience beyond the store, users are prompted to use their mobile device to snap a pic of a QR tag associated with each destination that allows them to explore additional destination info, video and travel packages.

The event was a complete success. People couldn’t wait to see where Delta would take them next. The Takeaway – Delta is not just an airline; they’re giving me new ways to discover travel destinations.

This is the future of travel … and it’s just the beginning.

Since we were running the experience in the Microsoft booth, we decided to add some new characters – the most popular of which being Steve Ballmer:

We used the experience as a way to engage with conference attendees and demonstrate an innovative use of Microsoft Tag technology. As conference attendees had their RockstAR snapshot taken, we’d ask them to download the tag reader application to their mobile device. Afterwards, they could take a snapshot of the Microsoft Tag and retrieve their photo. We took over 300 photos at the event.

The RockstAR experience is another example of how you can use tag technology to extend an interactive in-store experience to a customers’ mobile device. Wishlists, shopping carts, mobile content delivery, product ratings & reviews and wayfinding are some of the examples of how tag technology can be used to change the way people shop in retail.

Check out our pictures from the event.

First, let’s talk software. The application was built on the recently announced Razorfish Vision Framework. The framework provides a platform to power augmented reality, gestural and other vision-based experiences. For the RockstAR experience, we are analyzing each frame coming from an infrared camera to determine if faces are found in the crowd. Once a face is detected, it is assigned a unique ID and tracked. Once we receive a lock on the face, we can pass position and size information to the experience where we can augment animations and graphics on top of the color camera feed. This technology has a variety of uses. For instance, face tracking can be used to track impressions on static or interactive digital experiences in the retail environment. Here is a screenshot taken from the debug-mode of the experience which shows the face tracking engine at work using the infrared camera.

In addition to the vision-based technology, the experience was fully multi-touch enabled – users can gesture on a virtual joystick to swap out bands and snap pictures.

Because the classic photo booth experience is a social activity, we took it to the next level with twitter and Flickr integration. As pictures were snapped, we’d immediately make them available online. A QR code was rendered with each picture to quickly allow users to navigate to the RockstAR photo on their mobile device. Once the experience is extended to mobile, users can email the pictures to their friends, set it as wallpaper, re-tweet it to their twitter followers, etc.

Let’s move on to hardware. Unfortunately, you can’t purchase infrared AR-ready cameras at your local Walmart… at least not until Project Natal comes out later this year. Therefore, we needed to build a dual-camera system that would support the face tracking in infrared and the color video feed for display on the screen. We decided to go with 2 commercial-grade Firefly MV cameras with custom lenses.

One of the cameras we modified to see only infrared light by replacing the IR-blocking filter with a IR band-pass filter. This allows only a narrow range of infrared light to reach the camera CCD.

We also purchased and tested a variety of infrared illuminators. These are used to illuminate the environment with invisible infrared light allowing the infrared camera to accurately track faces in low-light conditions.

Sparks were flying as we fused the color and infrared cameras together — just another day at the office.

We created a portable rig for the camera and infrared illuminators. Adjustable camera mounts and industrial strength velcro provide flexibility and portability across a variety of installations.

We used a presentation remote clicker as an alternative way to drive the experience. We primarily used it as a remote camera trigger which allowed us to quickly snap pictures of unsuspecting people from a distance.

The experience was powered by a 55″ multi-touch screen and a CPU provided by DFI Technologies. We’ve been working with DFI to build PCs that will power the next-generation of interactive experiences. These PCs are small form factor and can be mounted behind the multi-touch screen.

Last but not least, we bring you the pink rug. We can’t reveal too much information about this technology… we need to keep some things secret. Just know that it is critical to the overall experience.

Customers are being faced with increasingly complex buying decisions, especially when it comes to technology and services. As a result, increased pressure is being placed on store associates to provide knowledgeable service to customers. Our Emerging Experiences team used this opportunity to develop a solution to demonstrate how an immersive interactive experience can assist customers and store associates with complex buying decisions in a retail setting.

The Razorfone solution is focused around providing an experience where customers can learn about wireless products. Phone and netbook features and associated wireless plans are presented in a rich interactive environment. Accessories are presented to the customer as a cross-sell opportunity. The use of 3D allows for customers to view their device in any color and from any angle, regardless of availability of the physical device within the retail store. The customer can validate the wireless coverage where they live, work and play through the use of the interactive coverage map using simple pan and zoom gestures. Customers can personalize their device after purchase with ringtones, games and applications through the use of QR technology, increasing sales and helping to eliminate buyers remorse.

As the complexity of technology and service offerings increases, training store associates on these products and services can be expensive and ineffective. In addition, the turnover rate in retail can have a negative effect on the number of knowledgable store associates. In an assisted-sales scenario, the store associates can focus on being brand ambassadors while the experience provides the information the customer needs to make informed decisions about the wireless products.

Watch the above video as we walk you through the Razorfone experience. We’d love to hear your feedback!