Since we were running the experience in the Microsoft booth, we decided to add some new characters – the most popular of which being Steve Ballmer:
We used the experience as a way to engage with conference attendees and demonstrate an innovative use of Microsoft Tag technology. As conference attendees had their RockstAR snapshot taken, we’d ask them to download the tag reader application to their mobile device. Afterwards, they could take a snapshot of the Microsoft Tag and retrieve their photo. We took over 300 photos at the event.
The RockstAR experience is another example of how you can use tag technology to extend an interactive in-store experience to a customers’ mobile device. Wishlists, shopping carts, mobile content delivery, product ratings & reviews and wayfinding are some of the examples of how tag technology can be used to change the way people shop in retail.
We recently had the opportunity to debut the RockstAR experience at SXSW – check out video of the experience in action. We like to think of it as the classic photo booth taken to the next level with augmented reality, multi-touch and social integration. Let’s go behind-the-scenes and take a look at both the software and hardware that brings this experience to life.
First, let’s talk software. The application was built on the recently announced Razorfish Vision Framework. The framework provides a platform to power augmented reality, gestural and other vision-based experiences. For the RockstAR experience, we are analyzing each frame coming from an infrared camera to determine if faces are found in the crowd. Once a face is detected, it is assigned a unique ID and tracked. Once we receive a lock on the face, we can pass position and size information to the experience where we can augment animations and graphics on top of the color camera feed. This technology has a variety of uses. For instance, face tracking can be used to track impressions on static or interactive digital experiences in the retail environment. Here is a screenshot taken from the debug-mode of the experience which shows the face tracking engine at work using the infrared camera.
In addition to the vision-based technology, the experience was fully multi-touch enabled – users can gesture on a virtual joystick to swap out bands and snap pictures.
Because the classic photo booth experience is a social activity, we took it to the next level with twitter and Flickr integration. As pictures were snapped, we’d immediately make them available online. A QR code was rendered with each picture to quickly allow users to navigate to the RockstAR photo on their mobile device. Once the experience is extended to mobile, users can email the pictures to their friends, set it as wallpaper, re-tweet it to their twitter followers, etc.
Let’s move on to hardware. Unfortunately, you can’t purchase infrared AR-ready cameras at your local Walmart… at least not until Project Natal comes out later this year. Therefore, we needed to build a dual-camera system that would support the face tracking in infrared and the color video feed for display on the screen. We decided to go with 2 commercial-grade Firefly MV cameras with custom lenses.
One of the cameras we modified to see only infrared light by replacing the IR-blocking filter with a IR band-pass filter. This allows only a narrow range of infrared light to reach the camera CCD.
We also purchased and tested a variety of infrared illuminators. These are used to illuminate the environment with invisible infrared light allowing the infrared camera to accurately track faces in low-light conditions.
Sparks were flying as we fused the color and infrared cameras together — just another day at the office.
We created a portable rig for the camera and infrared illuminators. Adjustable camera mounts and industrial strength velcro provide flexibility and portability across a variety of installations.
We used a presentation remote clicker as an alternative way to drive the experience. We primarily used it as a remote camera trigger which allowed us to quickly snap pictures of unsuspecting people from a distance.
The experience was powered by a 55″ multi-touch screen and a CPU provided by DFI Technologies. We’ve been working with DFI to build PCs that will power the next-generation of interactive experiences. These PCs are small form factor and can be mounted behind the multi-touch screen.
Last but not least, we bring you the pink rug. We can’t reveal too much information about this technology… we need to keep some things secret. Just know that it is critical to the overall experience.
Before we left for the evening, we recorded a quick walkthrough of the Windows Phone booth and EMC (Executive Meeting Center) locations where we have touch experiences deployed to support the Windows Phone 7 Series launch event.
Members of the press and blogging community have been recording video of the experience throughout the conference. These videos have begun appearing online – here are a couple of the videos we’ve found:
After a long night of celebrating the successful launch of Windows Phone 7 Series in Barcelona, we are back at the Windows Phone booth at Mobile World Congress. The crowds are still huge and the experiences are running great. Each experience is collecting touch and interaction information in the background – we are going to begin processing this information to determine how many sessions we are seeing, average session time, the most popular areas of the experience, etc. We will use this information as a guide to optimize the experience for the next event.
The Windows Phone team is showing live projected demonstrations of the device in the theatre area – these demonstrations are attracting huge crowds.
Members of the press camped out at the Windows Phone press lounge located across the plaza from Mobile World Congress. Because of the huge turnout for the announcement, much of the press watched the launch event live from the downstairs press lounge. After the show, we launched 6 experiences at this location allowing members the press to touch and interact with Windows Mobile 7 Series for the first time.
Members of the press who weren’t able to watch the event in the theatre or the press lounge huddled around screens outside in the reception area. We went live with 2 experiences at this location.
Conference attendees watching the event live at the Windows Phone booth at Mobile World Congress. We had an additional 2 experiences running at this location.
Cameras were out as the interface was unveiled for the first time. The phone interface design was kept a secret up until launch day. Preventing pictures and other leaks of information from making it to the press turned out to be a huge undertaking. The Windows Phone team went to great lengths to prevent leaks – in fact, many of the Microsoft employees working on the team never had the opportunity to see the interface until launch day. We based our experience off of some hands-on time in Redmond and videos of the experience. Our team was able to reverse-engineer the design, animation and interaction of the user interface. Accuracy was extremely important and we had to ensure the design and motion in our experience was a perfect re-creation of the experience on the actual device. We built the experience on top of the Razorfish Touch Framework. Using the framework allowed us to rapidly develop the application from scratch in under 4 weeks.
The product launch was a huge success and the Windows Phone team has been celebrating in Barcelona. The reaction from the press and blog community has been overwhelmingly positive. The conference is far from over but so far we are off to a great start!
The Razorfish EE team has been kicking the tires on Windows 7 for almost a year prior to product launch. With great success we’ve been running the Razorfish Touch Framework (RTF) on Windows 7 across a number of multi-touch displays. Recently our Neue Digitale office in Germany had the opportunity to port the Audi Configurator Surface experience to Windows 7 for the Windows 7 launch event. Utilizing the RTF, the team was able to bring the Microsoft Surface experience to Windows 7 in under a week. This makes the Audi Configurator, not only the first Windows 7 multi-touch application on the German market, but also a tool which is hardware independent and can be installed on a variety of multi-touch enabled devices.
We also had the opportunity to test drive with an experimental multi-touch hardware device named “dreaMTouch” manufactured by Elektrosil Systeme der Elektronik GmbH for the Windows 7 launch. We recently worked with the device for the DMEXCO (Digital Marketing Exposition Conference) in Germany where we showcased the Razorfashion application and our other multi-touch experiences. The device supports an incredible 32 simultaneous touches on a 46″ screen. The device itself only measures a few inches thick. The dreaMTouch is now available for purchase.
We’ve been busy traveling the globe demonstrating our touch experiences. The reaction from clients and the industry has been great – we’ve met some wonderful people along the way. Recently we were at the Microsoft Vendor Fair showing our Surface and vertical multi-touch experiences. With the Windows 7 release, the excitement within Microsoft is understandable. Much like our other clients, we’re working with Microsoft in realizing the potential of touch related experiences from a sales and marketing perspective. To see evidence of this look no further than the Microsoft retail store.
Razorfashion goes international. We jumped at a chance to support our Neue Digitale team in Germany at DMEXCO (Digital Marketing Exposition Conference). To support the effort, we’ve been working with a company in Germany on an experimental multi-touch device named dreaMTouch. Once we figured out how to get the pre-release device through customs, we were able to integrate the hardware with the Microsoft-based Razorfish Touch Framework in under a day. Since integrating with the hardware was so easy, we decided to feature the Razorfashion application during the conference as well.
Check out the above technology demonstration of the multi-touch Virtual Earth control and the Razorfish Touch Framework. This video demonstrates a weather experience and the cross-platform capabilities of the framework on Microsoft Surface or other touch-enabled devices.
At the 2009 Razorfish Client Summit, the Emerging Experiences group announced the availability of the Razorfish Touch Framework. This framework allows for rapid development of rich and engaging touch-based experiences. These experiences can be deployed across a variety of hardware platforms allowing for re-use of code and content.
We have been using the framework on a number of confidential client projects and internal prototypes since early 2009. For the client summit, we were given the opportunity to create a touch experience, we named “Razorfashion,” using the framework and targeted the experience towards the fashion industry. This application is an easily customizable and deployable fashion retail solution which also serves to demonstrate the capabilities of the framework to the public. Be sure to check out the video of the Razorfashion experience. Check back soon for videos and posts about our other cross-platform, multi-touch experiences.
The Razorfish Touch Framework runs on Windows XP and Vista – there is no need to wait for the availability of native multi-touch support in Windows 7. The framework is built on Microsoft Windows Presentation Foundation (WPF) and provides a core set of touch-enabled controls to jump-start touch-based application development. The complexities of interfacing with specific touch hardware platforms is handled by the framework, allowing the designers and developers to focus on creating rich, engaging next-generation touch experiences.
