Our team was asked to help launch the Microsoft Windows Phone 7 project at Mobile World Congress 2010. The project was a whirlwind experience – starting with 5 weeks of design/development and 11 days of deployment and support that spanned 2 continents and countless late nights. It was all worth it when Steve Ballmer made the introduction and we were all a part of history as the next generation in mobile experiences was announced to the 50k MWC attendees and a larger worldwide audience.  The people lucky enough to be in attendance couldn’t wait to get their hands on the experience we built.

We setup 10 touchscreens in 2 locations and the experiences were in constant use. Microsoft has since taken the touchscreens to countless other events including MIX10, SXSW 2010, CES, CTIA and many many more.

In addition to the touchscreen experiences, we also worked with our Seattle team to produce a microsite experience that would allow those not in attendance to get a taste of the phone.

Since we were running the experience in the Microsoft booth, we decided to add some new characters – the most popular of which being Steve Ballmer:

We used the experience as a way to engage with conference attendees and demonstrate an innovative use of Microsoft Tag technology. As conference attendees had their RockstAR snapshot taken, we’d ask them to download the tag reader application to their mobile device. Afterwards, they could take a snapshot of the Microsoft Tag and retrieve their photo. We took over 300 photos at the event.

The RockstAR experience is another example of how you can use tag technology to extend an interactive in-store experience to a customers’ mobile device. Wishlists, shopping carts, mobile content delivery, product ratings & reviews and wayfinding are some of the examples of how tag technology can be used to change the way people shop in retail.

Check out our pictures from the event.

First, let’s talk software. The application was built on the recently announced Razorfish Vision Framework. The framework provides a platform to power augmented reality, gestural and other vision-based experiences. For the RockstAR experience, we are analyzing each frame coming from an infrared camera to determine if faces are found in the crowd. Once a face is detected, it is assigned a unique ID and tracked. Once we receive a lock on the face, we can pass position and size information to the experience where we can augment animations and graphics on top of the color camera feed. This technology has a variety of uses. For instance, face tracking can be used to track impressions on static or interactive digital experiences in the retail environment. Here is a screenshot taken from the debug-mode of the experience which shows the face tracking engine at work using the infrared camera.

In addition to the vision-based technology, the experience was fully multi-touch enabled – users can gesture on a virtual joystick to swap out bands and snap pictures.

Because the classic photo booth experience is a social activity, we took it to the next level with twitter and Flickr integration. As pictures were snapped, we’d immediately make them available online. A QR code was rendered with each picture to quickly allow users to navigate to the RockstAR photo on their mobile device. Once the experience is extended to mobile, users can email the pictures to their friends, set it as wallpaper, re-tweet it to their twitter followers, etc.

Let’s move on to hardware. Unfortunately, you can’t purchase infrared AR-ready cameras at your local Walmart… at least not until Project Natal comes out later this year. Therefore, we needed to build a dual-camera system that would support the face tracking in infrared and the color video feed for display on the screen. We decided to go with 2 commercial-grade Firefly MV cameras with custom lenses.

One of the cameras we modified to see only infrared light by replacing the IR-blocking filter with a IR band-pass filter. This allows only a narrow range of infrared light to reach the camera CCD.

We also purchased and tested a variety of infrared illuminators. These are used to illuminate the environment with invisible infrared light allowing the infrared camera to accurately track faces in low-light conditions.

Sparks were flying as we fused the color and infrared cameras together — just another day at the office.

We created a portable rig for the camera and infrared illuminators. Adjustable camera mounts and industrial strength velcro provide flexibility and portability across a variety of installations.

We used a presentation remote clicker as an alternative way to drive the experience. We primarily used it as a remote camera trigger which allowed us to quickly snap pictures of unsuspecting people from a distance.

The experience was powered by a 55″ multi-touch screen and a CPU provided by DFI Technologies. We’ve been working with DFI to build PCs that will power the next-generation of interactive experiences. These PCs are small form factor and can be mounted behind the multi-touch screen.

Last but not least, we bring you the pink rug. We can’t reveal too much information about this technology… we need to keep some things secret. Just know that it is critical to the overall experience.

We’ve been very busy since the Windows Phone 7 Series experience launch in Spain, and there’s plenty more excitement over the next couple of weeks. We will be announcing the Razorfish Vision Framework at this year’s SxSW Interactive Conference. We have created a prototype application that is a mixture of Augmented Reality and Multi-touch called RockstAR and will be showcasing it at the Razorfish hosted cocktail party on Saturday the 13th at the Paradise bar. Come by and augment your reality!

Also …

Below is a quick list of the conferences we’re at in March. Please let us know if you are also attending and we’ll make sure you get a chance to check out our experiences.

• GDC 2010 (March 9-13, SanFran) – Lesley just returned from San Francisco, CA where we helped Microsoft showcase the WP7S experience in their Game Developers Conference booth.
• SxSW 2010 Interactive (March 12-15, Austin) – We’re hosting a panel on the 12th, please come interact with us as we discuss “Touch + The Holy Grail of Delight” at 2pm http://my.sxsw.com/events/event/6124. We are also launching a fun new AR experience for the Razorfish cocktail party. Finally we will be at the Microsoft Speakeasy event showcasing the WP7S experience.
• MIX10 (March 15-17, Vegas) – @stevedawson and @hulljon will be attending the conference. The WP7S experience will also be making an appearance … with updated content!
• CTIA (March 22-25, Vegas) – WP7S experience will be out in force at this international wireless conference.

A quick thanks to all the peeps that helped out with RockstAR:

• Steve – @stevedawson
• Luke – @lhamilton
• Alex – @alexn
• Dung – @dungtienle
• Sara – @saragiessen
• Heather – @heathergately
• Jonathan – @hulljon

Neue Digitale / Razorfish developed the first realtime 3d configurator for Microsoft Surface which was showcased at this year’s IAA – the world’s leading automotive fair.

Developed in collaboration with Realtime Technology AG, the application allows multiple users to configure an Audi A4 simultaneously by changing the car’s paint finishes, rims and by selecting and coloring style package components. The configured A4 is experienced in an immersive 3d environment, in which users can navigate seamlessly by zooming and panning using a multitouch-enabled interface. Special perspectives allow a more detailed view for specific components, for example for configuring rims or taking a closer look at interior details.

To control the actual car configuration process users can place multilingual tokens on the table. All available options appear around the token and can be selected by touching or rotating the physical token.

The final car configuration is projected on an external High Definition display, attracting and engaging other visitors.

Check out the above technology demonstration of the multi-touch Virtual Earth control and the Razorfish Touch Framework. This video demonstrates a weather experience and the cross-platform capabilities of the framework on Microsoft Surface or other touch-enabled devices.

We have been using the framework on a number of confidential client projects and internal prototypes since early 2009. For the client summit, we were given the opportunity to create a touch experience, we named “Razorfashion,” using the framework and targeted the experience towards the fashion industry. This application is an easily customizable and deployable fashion retail solution which also serves to demonstrate the capabilities of the framework to the public. Be sure to check out the video of the Razorfashion experience. Check back soon for videos and posts about our other cross-platform, multi-touch experiences.

The Razorfish Touch Framework runs on Windows XP and Vista – there is no need to wait for the availability of native multi-touch support in Windows 7. The framework is built on Microsoft Windows Presentation Foundation (WPF) and provides a core set of touch-enabled controls to jump-start touch-based application development. The complexities of interfacing with specific touch hardware platforms is handled by the framework, allowing the designers and developers to focus on creating rich, engaging next-generation touch experiences.

The following platforms are currently supported:

• Microsoft Surface
• NextWindow
• N-trig
• Single-touch (Elo, single-touch capacitive, mouse & HID-based hardware)
• TUIO-based custom multi-touch hardware (touch wall, table top)
• Multi-point “touchless” platforms

The cross-platform support of the framework is an industry first. Stay tuned for future videos and posts showing the framework in action.

model photo credit: *katanaz-stock

DaVinci is a Microsoft Surface application that blurs the lines between the physical and virtual world by combining object recognition, real-world physics simulation and gestural interface design.

One of the important design and technology considerations around building direct manipulation interfaces is to ensure virtual objects behave like real-world objects. The real world physics behavior of these objects is a type of visual affordance, which aids in delivering an intuitive user experience – users know what to expect when they manipulate objects in the virtual world when they behave like objects in the real world. The physics engine used in DaVinci is being applied to our other applications to create interfaces which have natural real world physics.

The DaVinci experience could easily be extended to teach Newton’s laws of motion, gravity, friction, orbital motion and other physics concepts in a classroom setting.

The Microsoft Surface technology offers such great possibilities. Consider how we solved one of man’s greatest challenges – how to watch multiple NCAA March Madness games simultaneously. These are the wonders of what you can create while working at the office late. Notice how the interface allows us to easily switch between live feeds as the games are in progress. This is just another example of how Microsoft Surface technology can deliver an innovative and engaging customer experience.

With an abundance of plan and phone choices, shopping for cell phone service can require a lot of time. However, this award-winning in-store solution streamlines the sales process by enabling customers to make informed decisions around wireless technologies. The solution combines Microsoft Surface technology with an innovative and engaging user interface. It is the first Surface application to be deployed and remains the largest deployment to date (50 tables in 12 stores).

Check out the video walkthrough of the AT&T Retail Surface Experience above. We provide an in-depth walkthrough of the experience and cover some of the environmental, user experience and technology considerations when developing applications for the Microsoft Surface platform.